TWI761331B - Sample serialization method and apparatus - Google Patents

Abstract

Embodiments of the present invention provide a method and device for serializing samples, which relate to the technical field of machine training. The method includes: acquiring each character string in the sample to be serialized; determining the management server corresponding to each character string according to the correspondence between each character string and each management server; sending the character string to the corresponding The management server is used for each management server to convert the received string into the corresponding serialized ID according to the mapping table maintained by it; wherein, the strings in the mapping table maintained by different management servers are different from each other; Receive the serialized ID corresponding to each string returned by each management server; convert the string in each sample data into the corresponding serialized ID according to the serialized ID corresponding to each received string. The invention reduces the query time of the serialized ID of the string, thereby reducing the time for serializing the sample and improving the serialization efficiency.

Description

Sample serialization method and apparatus

The present invention relates to the technical field of machine training, in particular to a sample serialization method and a sample serialization device.

In the Internet, a large amount of data can be generated based on the user's network behavior, and in order to study various behavioral habits of users, various models may be constructed, and in order to train these models, machine learning systems are generally used. In the machine learning system, because the strings of each dimension in the sample data may not be serialized IDs, such as digital IDs, they are named according to business requirements. Then, if the string of sample data is directly trained, the amount of calculation is relatively large and the resource consumption is high.

Therefore, in order to reduce the amount of computation, before training, it is necessary to convert all strings in the sample data into serialized IDs, such as digital IDs. For example, the format of a sample data is as follows: there are two columns: the first column is the label column, which records whether the user clicks or not. If the record is 1, it means the user clicked, and if the record is 0, it means the user did not click; is the feature column, which is all the features of the sample, separated by commas, for example: 1 user_id_123,age_1,sex_1,age_comb_city3

Then you need to convert all "user_id_123, age_1, sex_1, age_comb_city3" into digital IDs, that is, you need to establish the following mapping relationship: {string collection}->{number collection}

Then the above-mentioned "user_id_123, age_1, sex_1, age_comb_city3" converts the mapping relationship as: user_id_123->number X, age_1->number Y, sex_1->number Z, age_comb_city3->number F.

However, in the process of using the inventor, it is found that when there are many string set elements, the single-machine memory cannot be loaded, and the time to serialize the sample data is very long. For example, when there are 2 billion strings, each machine needs to load a complete The mapping table, the memory exceeds 40G, and the serialization time is also very long.

In view of the above problems, the embodiments of the present invention are proposed to provide a sample serialization method and a corresponding sample serialization device that overcome the above problems or at least partially solve the above problems.

In order to solve the above problems, the present invention discloses a sample serialization method, which includes: obtaining each character string in the sample to be serialized; management server; send the string to the corresponding management server for each management The server converts the received string into the corresponding serialized ID according to the mapping table maintained by it; the strings in the mapping tables maintained by different management servers are different from each other; Serialized ID of the string; according to the serialized ID corresponding to each received string, convert the string in each sample data into the corresponding serialized ID.

The present invention also discloses a sample serialization method, comprising: receiving a character string; the character string is sent by the serialization server according to the corresponding relationship between the character string and each management server; the character string is sent by the serialization server from Obtained from the sample data; according to the locally maintained mapping table, the received string is converted into a serialized ID; wherein, the strings in the mapping tables maintained by different management servers are different from each other; The serialization ID is returned to the corresponding serialization server, so that the serialization server can convert the strings in each sample data into corresponding serialization IDs according to the serialization IDs corresponding to the received strings.

The present invention also discloses a sample serialization device, comprising: a character string extraction module for obtaining each character string in the sample to be serialized; a management server determination module for matching each character string with each management server Correspondence between each string to determine the management server corresponding to each string; A string sending module is used to send the string to the corresponding management server, so that each management server can convert the received string into a corresponding serialized ID according to the mapping table maintained by the management server; wherein, The strings in the mapping tables maintained by different management servers are different from each other; the serialized ID receiving module is used to receive the serialized ID corresponding to each string returned by each management server; the sample serialization module is used to receive the serialized ID corresponding to each string returned by each management server; The serialized ID corresponding to each received string is converted into the corresponding serialized ID from the strings in each sample data.

The invention also discloses a sample serialization device, comprising: a character string receiving module for receiving character strings; the character strings are sent by the serialization server according to the correspondence between the character strings and each management server; The string is obtained from the sample data by the serial server; the string conversion module is used to convert the received string into a serialized ID according to the locally maintained mapping table; among them, the mapping maintained by different management servers The strings in the table are different from each other; the digitized ID return module is used to return the serialization ID corresponding to the string to the corresponding serialization server, so that the serialization server can use the received strings according to each string. For the corresponding serialization ID, convert the strings in each sample data into the corresponding serialization ID.

The embodiments of the present invention include the following advantages: In the embodiments of the present invention, the mapping tables required for serialization are distributed among multiple management servers, and the strings maintained in the mapping tables of different management servers are different from each other, and the digits of the corresponding strings are different from each other. The chemical ID is also different. then serialize The server only needs to send the sample to be serialized and the string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the serialization of the string The ID is returned to the serialization server. The serialized ID, such as a numeric ID. Therefore, the serialization server can convert the samples into digitized samples for subsequent training. In this way, the serialization server does not need to load the mapping table, avoiding the memory shortage of the serialization server. In addition, since the mapping table is distributed to multiple management servers, when the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the number of sample sequences. time and improve serialization efficiency.

110‧‧‧Steps

120‧‧‧Steps

130‧‧‧Steps

140‧‧‧Steps

150‧‧‧Steps

210‧‧‧Steps

220‧‧‧Steps

230‧‧‧Steps

310‧‧‧Steps

312‧‧‧Steps

314‧‧‧Steps

316‧‧‧Steps

318‧‧‧Steps

320‧‧‧Steps

322‧‧‧Steps

324‧‧‧Steps

326‧‧‧Steps

328‧‧‧Steps

330‧‧‧Steps

332‧‧‧Steps

410‧‧‧String Extraction Module

420‧‧‧Management server determination module

430‧‧‧String sending module

440‧‧‧Serialized ID receiving module

450‧‧‧Sample Serialization Module

510‧‧‧String receiving module

520‧‧‧String Conversion Module

530‧‧‧Digital ID Return Module

600‧‧‧Scheduling Server

601‧‧‧Notification Module

700‧‧‧Serialization Server

701‧‧‧Sample acquisition module

702‧‧‧String Extraction Module

703‧‧‧String remainder module

704‧‧‧First remainder determination module

705‧‧‧String sending module

706‧‧‧Serialized ID receiving module

707‧‧‧Sample Serialization Module

708‧‧‧Output Module

800‧‧‧per management server

801‧‧‧String receiving module

802‧‧‧String Conversion Module

803‧‧‧Digital ID Return Module

Fig. 1 is a flow chart of steps of a method embodiment of a sample serialization described from the serialization server side of the present invention; Fig. 2 is a flow chart of steps of an embodiment of a sample serialization method described from the management server side of the present invention 3 is a flow chart of the steps of a sample serialization method embodiment of the present invention; FIG. 4 is a structural block diagram of a sample serialization device embodiment of the present invention; FIG. 5 is a sample serialization device embodiment of the present invention. Figure 6 is a structural block diagram of a sample serialization system embodiment of the present invention. block diagram.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the drawings and specific embodiments.

One of the core concepts of the embodiments of the present invention is to disperse the mapping tables required for serialization into multiple management servers. The strings maintained in the mapping tables of different management servers are different from each other, and the serialization IDs of the corresponding strings are different from each other. Also different. Then the serialization server only needs to extract the character string from the sample data for the sample data to be serialized, and send the character string to the corresponding management server according to the corresponding relationship between the character string and each management server. server; the management server then obtains the serialization ID of the string and returns it to the serialization server. The serialization server can then convert the samples into digitized samples for subsequent training. In this way, the serialization server does not need to load the mapping table, avoiding the memory shortage of the serialization server. In addition, since the mapping table is distributed to multiple management servers, when the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the number of sample sequences. time and improve serialization efficiency.

Example 1

Referring to FIG. 1, a flow chart of steps of a sample serialization method embodiment of the present invention is shown, which may specifically include the following steps: Step 110: Acquire each character string in the sample to be serialized; in the embodiment of the present invention, the serialization server first receives the sample data to be serialized. In a preferred embodiment, before step 110, it further includes : Step S100, acquiring each sample data to be serialized; in this embodiment of the present invention, there may be one or more serialization server slaves. Each serialization server can obtain a batch of sample data processed by the serialization server according to the notification of the scheduling server coordinator.

In the embodiment of the present invention, each serialization server, each management server, and the scheduling server can form a training cluster for machine training.

In another preferred embodiment of the present invention, the step of acquiring each sample data to be serialized includes: sub-step S11 , acquiring the data belonging to the current serialization server after the scheduling server has evenly distributed all the sample data The device's own batch of sample data.

For example, there are two serialization servers in the training cluster, serialization server A and serialization server B. There are a total of 10,000 pieces of sample data, and the scheduling server can divide the 10,000 pieces of sample data into two parts, each with 5,000 pieces, and notify serialization server A and serial server B to obtain the corresponding 5,000 pieces of sample data.

Of course, the sub-step S11 is only a preferred manner of the present invention, and other allocation manners are also possible, which are not limited in the embodiment of the present invention. For example, according to the performance allocation of the serialization server, at this time, after receiving the uploaded sample data, the scheduling server can allocate it as a serialization server. Before sample data, first obtain the hardware performance of the serialized server, and allocate the corresponding proportion of sample data within a certain range of hardware performance.

Further, in this embodiment of the present invention, each serialization server, after acquiring the sample data serialized by it, extracts a character string from the sample. For example, a sample is as follows:

There are two columns in the sample data. The first column is the label column, which indicates whether the user clicked. The value is 1, which means the user clicked, and the value is 0, which means the user did not click. The second column is the feature column, and the value is the sample data. All characteristics of , separated by commas.

Then the serialization server of the present invention extracts "user_id_123", "age_1", "sex_1", and "age_comb_city3" from the feature column.

It can be understood that the above examples are only examples for describing the extracted character string, and the present invention is not limited thereto, and sample data in other formats can also be used.

It should be noted that, in the embodiment of the present invention, when a character string is extracted from the sample data, a non-purely digitized character string is extracted. For example, the aforementioned "user_id_123", "age_1", "sex_1", "age_comb_city3". If the features in the feature column have pure numbers, they will not be extracted.

In the embodiment of the present invention, the format of the sample data can be analyzed in advance to determine the method to extract character strings from the sample data, such as what kind of extraction template to use to extract the data. Of course, it can be predetermined by the scheduling server in what way Extract the string and notify each serialization server.

Of course, in the embodiment of the present invention, when the sample data is serialized, it can be serialized one by one, that is, a string of sample data is extracted and sent to the corresponding management servers. After the sample data is serialized Serialize the next item. It can also be serialized in batches, that is, sending a batch of sample data strings at a time to the corresponding management servers.

Step 120, according to the correspondence between each character string and each management server, determine the management server corresponding to each character string; the serialization server in the embodiment of the present invention may send the above-mentioned extracted character string to the corresponding management server Server master. In this embodiment of the present invention, the character string is maintained by a mapping table belonging to a certain management server. In the embodiment of the present invention, the corresponding relationship between the character string and the management server can be agreed in a certain way.

In a preferred embodiment of the present invention, the step of determining the management server corresponding to each character string according to the corresponding relationship between each character string and each management server includes: sub-step S21: The corresponding hash value is divided by the number of management servers to obtain the remainder; in sub-step S22, the management server corresponding to the character string is determined according to the corresponding relationship between the remainder and the management server.

In the embodiment of the present invention, taking the aforementioned string "user_id_123" as an example, the hash value hash_value of the string is calculated, and then the hash_value is divided by the total number P of management servers, and the remainder is taken. The formula is hash_value%P .

In the embodiment of the present invention, the corresponding relationship between each of the remainders and the management server is preset.

For example, there are 2 management servers, and the corresponding remainders of 2 are 0 and 1. Then you can first assign 0 to management server A and 1 to management server B. Then, the strings with the remainder of 0 after the hash_value divided by 2 are sent to management server A; the strings with the remainder of 1 after the hash_value divided by 2 are sent to the management server B.

In the embodiment of the present invention, in order to facilitate the direct correspondence between the remainder and the management server, the management server can be directly named according to the aforementioned remainder, then after calculating the remainder, it is possible to directly know which management server the remainder is.

In another preferred embodiment of the present invention, after the step of acquiring each character string in the sample to be serialized, the method further includes: step S31 , deduplicating each character string.

In the embodiment of the present invention, in order to reduce the calculation amount of the management server and the network occupation amount, each character string may be deduplicated first.

Therefore, the string sent to the management server is unique each time, and there will be no repeated string transmission, correspondingly no repeated serialized ID will be returned, and no additional network bandwidth will be occupied. The string received by the management server is also unique each time, and the string is only calculated once in one calculation, and it will not be repeated, which reduces the calculation amount of the management server.

Step 130, sending the string to the corresponding management server, so that each management server converts the received string into a corresponding serialized ID according to the mapping table maintained by the management server; wherein, different management servers maintain The character strings in the mapping table of each management server are different from each other; in the embodiment of the present invention, each management server may obtain the character string maintained by the management server in advance, and then construct each management server's own mapping table. The mapping table is a comparison table of strings and serialized IDs.

In the embodiment of the present invention, the serialized ID is a digital ID, because in the training process, the number is most easily brought into the formula for calculation.

In the embodiment of the present invention, for each character string, the hash value of the character string can be divided by the number of all management servers, and the remainder is obtained, and the remainder also corresponds to the management server. As mentioned above, there are two management servers, then 0 corresponds to management server A, and 1 corresponds to management server B. Then, the character string can be sent to the corresponding management server according to the corresponding relationship between the remainder and the management server. The management server can then build a mapping table based on the string.

In practical applications, after each serialization server obtains its samples, it first extracts all strings of all samples, calculates the hash value of each string, and divides the hash value of each string by the management server's hash value. Calculate the total number and take the remainder, and then send the character string to the corresponding management server according to the corresponding relationship between the remainder and the management server.

After receiving the string, the management server generates a serialized ID for the string. Then build a mapping table of strings and corresponding serialized IDs.

For the management server, after receiving the string, query the serialization ID of the string from the locally maintained mapping table, and then return the serialization ID corresponding to the string to the serialization server. in practical application , the management server can return the string to the serialization server along with its corresponding serialization ID.

Step 140: Receive the serialized ID corresponding to each character string returned by each management server; after the serialization server has sent each character string of the sample data, it can receive the serialized ID corresponding to each of the above-mentioned character strings returned by the management server ID.

Step 150: Convert the strings in each sample data into corresponding serialization IDs according to the received serialization IDs corresponding to each string.

After receiving the serialization ID of each string, the serialization server converts the string in the sample data into the corresponding serialization ID. For example, the serialized ID of the aforementioned "user_id_123" is 11, the serialized ID of "age_1" is 13, the serialized ID of "sex_1" is 24, and the serialized ID of "age_comb_city3" is 55. Then the converted serialized sample data is:

Then, the serialized sample data can be used for subsequent machine training, speeding up the training speed and improving the training efficiency.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. The complete mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Secondly, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

Embodiment 2

Referring to FIG. 2, a flow chart of steps of a sample serialization method embodiment of the present invention is shown, which may specifically include the following steps: Step 210, receiving a character string; the character string is managed by the serialization server according to the character string and each The correspondence between the servers is sent; the character string is obtained from the sample data by the serial server; in the embodiment of the present invention, each management server receives the character string sent by one or several serialization servers.

In the embodiment of the present invention, on the serialization server side, for the sample data to be serialized, a character string can be extracted from it, and then the management server is determined according to the corresponding relationship between the character string and each management server, and then the character string to the management server.

For each serialization server, it determines the management server according to the corresponding relationship between the character string and each management server, and sending the character string to the management server includes: sub-step S51, hashing the hash corresponding to the character string Divide the value by the number of management servers to get the remainder; Sub-step S52, according to the corresponding relationship between the remainder and the management server, determine the management server corresponding to the character string.

In a preferred embodiment of the present invention, a mapping table that needs to be maintained by each management server can be constructed in real time, then before step 210, it further includes: step S201, obtaining a batch of character strings belonging to the current management server; wherein, A batch of character strings belonging to the current management server is different from character strings belonging to other management servers; in this embodiment of the present invention, multiple management servers may be set, and then for each management server, a Batch strings, strings obtained by different management servers are different from each other.

In this embodiment of the present invention, each management server may obtain character strings maintained by the management server in advance, and then construct its own mapping table for each management server.

In the embodiment of the present invention, for each character string, the hash value of the character string may be divided by the number of all management servers, and the remainder is obtained, and the remainder also corresponds to the management server. As mentioned above, there are two management servers, then 0 corresponds to management server A, and 1 corresponds to management server B. Then, the character string can be sent to the corresponding management server according to the corresponding relationship between the remainder and the management server. The management server can then build a mapping table based on the string.

In practical applications, after each serialization server obtains its samples, it first extracts all strings of all samples, calculates the hash value of each string, and divides the hash value of each string by the management server's hash value. The total number and the remainder are taken, and then according to the corresponding relationship between the remainder and the management server, the character The string is sent to the corresponding management server.

Wherein, the remainder corresponding to a batch of character strings belonging to the current management server belongs to the current management server; the remainder is obtained by dividing the hash value corresponding to the character string by the number of each management server.

Step S202, serialize the character string, and construct a mapping table between the character string and the serialized ID; after receiving the character string, the management server generates a serialized ID for the character string. Then build a mapping table of strings and corresponding serialized IDs.

Preferably, the step of serializing the character string and constructing a mapping table between the character string and the serialized ID includes: sub-step S41, obtaining the information in each management server before the sorting of the current management server. The first total number N1 of character strings; for example, the management server has A, B, C, and the order is also as A, B, C. For the first management server A, it has 110 strings; for the second management server B, it has 90 strings, and for the third management server, it has 100 strings.

Then, the first total number N1 of character strings in each management server before management server A is N1=0.

Before the management server B, there is the management server A, and the first total number N1=110.

Before the management server C, there are the management server A and the management server B, and the first total number N1=200.

Sub-step S42, the second total number N2 is obtained by adding the first total number N1 to the number M of character strings of the current management server; In sub-step S43, [N1+1, N2] is used as the range for serializing the character string by the current management server.

The number of strings in management server A is M=110, then the serialization range of strings in management server A is [1,110], then for the strings in management server A, they can be sequenced from 1 to 110 in sequence Chemical ID.

The number of strings in management server B is 90, then the serialization range of strings in management server B is [111, 200], then the strings in management server B can be serialized corresponding to 111 to 200 in sequence ID.

The number of strings in management server C is 100, then the serialization range of strings in management server B is [201, 300], then for the strings in management server C, they can be serialized in sequence corresponding to 201 to 300 ID.

Step 220, according to the mapping table maintained locally, convert the received string into a serialized ID; wherein, the strings in the mapping tables maintained by different management servers are different from each other; for one management server, because It maintains a mapping table, which contains strings and their corresponding serialized IDs. Since the received strings are maintained by the management server, the management server can use the mapping table maintained locally by the management server. , which converts the received string to a serialized ID. For example, according to the correspondence between the character string and the digital ID in the mapping table, look up the digital ID corresponding to the character string, and then return the found digital ID to the corresponding serialization server.

In another preferred embodiment of the present invention, the local maintenance The mapping table, the step of converting the received string into serialized ID includes: sub-step S61, querying whether there is the character string in the locally maintained mapping table; sub-step S62, if there is any in the locally maintained mapping table If there is no such string in the locally maintained mapping table, a serialization ID is generated for the string, and the string and the The corresponding serialization ID is added to the mapping table.

In this embodiment of the present invention, the sample obtained by the serialization server may contain unrecorded character strings in the mapping table of the management server. In this case, the management server may generate a serialization ID for it, and then convert the character Strings and serialization IDs are recorded in the mapping table. At the same time, the serialization ID corresponding to the string is returned to the corresponding serialization server.

In practical applications, non-overlapping serialization ranges can be pre-defined for each monitoring server, and the management server can assign serialization IDs in the serialization range to the string. Allocate a unique serialization range again.

Step 230: Return the serialization ID corresponding to the string to the corresponding serialization server, so that the serialization server can convert the string in each sample data according to the received serialization ID corresponding to each string for the corresponding serialization ID.

Of course, in this embodiment of the present invention, after receiving the string, the management server may record that the string was sent by which serialization server, then After finding the serialization ID corresponding to the string, you can return the corresponding string and serialization ID to the corresponding serialization server according to the record.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. The complete mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Secondly, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

Embodiment 3

Referring to FIG. 3 , a flow chart of steps of a preferred embodiment of a sample serialization method of the present invention is shown.

In this embodiment, in order to describe the serialization method more clearly, the description is made from the perspective of scheduling the server, serializing the server, and managing the overall architecture of the server.

In this embodiment of the present invention, a mapping table may be created for each management server in combination with the scheduling server and the serialization server. As in step S30-step S38.

Step S32, the scheduling server evenly distributes all the sample data, and notifies each serialization server to obtain a batch of sample data belonging to each serialization server according to the distribution result.

Before the entire training starts, after the scheduling server obtains the identification information of all the sample data, it can evenly distribute all the sample data. For example, the sample data is evenly distributed to N serial servers according to the serial number of the sample data. The dispatch server notifies each serialization server of the allocation structure, so that each serialization server obtains its own sample data. At the same time, the scheduling server notifies the serialization server to perform the string serialization process, so that it does not serialize the sample data first, because the management server does not have a mapping table at this time.

In step S34, each serialization server obtains its own batch of sample data according to the notification from the scheduling server, and integrates and sends all the character strings in the sample data to the management server.

In practical applications, after each serialization server obtains the above-mentioned first equally divided sample data, it can extract all the strings of the batch of data from these sample data according to the pre-configured extraction rules, and then The string is deduplicated, and then the deduplicated string is sent to each management server according to the sending rules. The sending rule includes: dividing the hash value corresponding to the string by the total number of management servers to obtain the remainder, for example, sending each string to the management server corresponding to the remainder according to the corresponding relationship between the remainder and the management server middle.

Step S36, the management server receives the words sent by each serialization server String; Step S38, after the management server receives all the strings belonging to the management server, serializes the strings, and builds a mapping table between strings and serialized IDs.

In the embodiment of the present invention, each serialization server can send the character string to the management server through a network connection, and after the character string is sent, the network connection with the corresponding management server can be disconnected. Then the management server can judge whether the serialization server has finished sending its string through the interruption of the network connection. After the management server determines that all serialization servers have sent the strings, the strings can be serialized, and a mapping table between strings and serialization IDs can be constructed.

Of course, in practical applications, the management server may also use other methods to determine that it has received all character strings belonging to itself. For example, a completion identifier is pre-agreed, and the serialization server sends the completion identifier to each management server after the string is sent, and then each management server records the completion identifier of the serialization server. After the completion flag of the conversion server is determined, it is determined that the management server has received all character strings belonging to the management server. The specific method is not limited in the embodiments of the present invention.

After the management server completes the construction of the aforementioned mapping table, the scheduling server may coordinate the serialization servers to perform serialization operations on the sample data. As in steps 310-332.

Step 310, the scheduling server notifies each serialization server to obtain its own sample data; For each serialization server, perform the following steps: step 312, read sample data according to the notification; step 314, extract each character string from the sample data; of course, in practical applications, for the extracted character string, also It will be deduplicated and then step 316 will be executed.

Step 316, for each character string, divide the hash value corresponding to the character string by the number of management servers to obtain the remainder; Step 318, determine the management server corresponding to the character string according to the correspondence between the remainder and the management server server; Step 320, sending the character string to the corresponding management server.

For the management server, the following steps are performed: Step 322 , receiving a character string; receiving the character string sent by the serialization server in step 320 .

Step 324: Convert the received string into a serialized ID according to the locally maintained mapping table.

The mapping table has been constructed in steps S32-S38.

Step 326: Return the serialization ID corresponding to the string to the corresponding serialization server.

Then, for each serialization server, perform the following steps: Step 328, receive the serialization ID corresponding to each character string returned by each management server; Step 330, according to the received serialization ID corresponding to each character string, Convert the strings in each sample material to the corresponding serialized IDs.

Step 332, output the serialized sample data.

In this way, the serialized sample data can be used for machine training.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. The complete mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Finally, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

Then, in the process of combining steps S32-S38, in the process of constructing the mapping table, the strings of all samples are scattered to multiple serialization servers for extraction, and the extraction speed is fast, so that the construction speed of the mapping table is accelerated. Secondly, the construction of the mapping table is distributed among multiple management servers, and each management server does not need to build a complete mapping table, but only needs to build a part of the mapping table, and the construction speed of the mapping table is accelerated. Thirdly, the location of building the mapping table is changed to the management server. The traditional serialization server does not need to perform the construction process of the mapping table, nor does it need to store the mapping table, which reduces the serialization server. the burden of the device.

It should be noted that, for the sake of simple description, the method embodiments are described as a series of action combinations, but those with ordinary knowledge in the technical field should know that the embodiments of the present invention are not subject to the described action sequence. , because according to the embodiment of the present invention, some steps may be performed in other order or simultaneously. Secondly, those with ordinary knowledge in the technical field should also know that the embodiments described in the specification are all preferred embodiments, and the related actions are not necessarily required by the embodiments of the present invention.

Embodiment 4

Referring to FIG. 4 , there is shown a structural block diagram of an embodiment of a sample serialization apparatus of the present invention, which may specifically include the following modules: a string extraction module 410 for obtaining each string in the sample to be serialized; wherein , before the character string extraction module 410, it further includes: a sample data acquisition module S400 for obtaining each sample data to be serialized; a management server determination module 420 for according to the relationship between each character string and each management server The corresponding relationship between each character string is determined, and the management server corresponding to each character string is determined; the character string sending module 430 is used to send the character string to the corresponding management server for each management server to maintain according to the mapping table. , convert the received string into the corresponding serialized ID; among them, different The strings in the mapping table maintained by the management server are different from each other; the serialization ID receiving module 440 is used to receive the serialized ID corresponding to each string returned by each management server; the sample serialization module 450 is used for Convert the strings in each sample data to the corresponding serialization IDs according to the serialization IDs corresponding to the received strings.

In another preferred embodiment of the present invention, the management server determination module 420 includes: a character string remainder module, which is used to divide the hash value corresponding to the character string by the number of management servers to obtain Remainder; the first remainder determination module is used to determine the management server corresponding to the character string according to the corresponding relationship between the remainder and the management server.

In another preferred embodiment of the present invention, after the character string extracting module 410, it further includes: a deduplication module for deduplicating each character string.

In another preferred embodiment of the present invention, the string extracting module 410 includes a first sample data obtaining module before the Serialize the server's own batch of sample data.

This embodiment can be applied to the serialization server side.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. Finish The entire mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Secondly, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

Embodiment 5

Referring to FIG. 5 , there is shown a structural block diagram of another embodiment of the sample serialization apparatus of the present invention, which may specifically include the following modules: a character string receiving module 510 for receiving character strings; the character strings are serialized by The server sends the string according to the correspondence between the string and each management server; the string is obtained from the sample data by the serial server; the string conversion module 520 is used to convert the received data according to the locally maintained mapping table. The received string is converted into a serialized ID; wherein, the strings in the mapping tables maintained by different management servers are different from each other; the digitized ID return module 530 is used to return the serialized ID corresponding to the string to The corresponding serialization server, for the serialization server to convert the strings in each sample data into corresponding serialization IDs according to the serialization IDs corresponding to the received strings.

This embodiment can be applied to the management server side.

In a preferred embodiment of the present invention, before the character string receiving module 510 includes: a character string obtaining module for obtaining a batch of character strings belonging to the current management server; wherein, the character string belonging to the current management server A batch of strings is different from strings belonging to other management servers; a mapping table building module is used to serialize the strings and build a mapping table between strings and serialization IDs.

In another preferred embodiment of the present invention, the mapping table construction module includes: a first quantity acquisition module for acquiring the first number of strings in each management server before the current management server is sorted The total number N1; the second number obtaining module is used to obtain the second total number N2 by adding the first total number N1 to the number M of the character strings of the current management server; the serialization range determination module is used to obtain the second total number N2; [N1+1,N2] as the current management server to serialize the range of strings.

In another preferred embodiment of the present invention, the character string conversion module includes: a query module for querying whether the character string exists in the locally maintained mapping table; a first digitized ID acquisition module, Used to obtain the serialization ID corresponding to the character string if there is the character string in the locally maintained mapping table; a module is generated for if the character string is not included in the locally maintained mapping table, for the character string yields a serialized ID, and converts the character The string and the corresponding serialization ID are added to the mapping table.

In another preferred embodiment of the present invention, the batch of character strings belonging to the current management server includes: the remainder corresponding to each character string in the batch of character strings belongs to the current management server; the remainder is The hash value corresponding to the character string is obtained by dividing the number of each management server.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. The complete mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Secondly, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

Embodiment 6

Referring to FIG. 6, it shows a structural block diagram of another embodiment of the sample serialization system of the present invention, which may specifically include the following modules: a scheduling server 600, a plurality of serialization servers 700, a plurality of management Server 800. Only three serialization servers 700 and three management servers 800 are shown in the figure, and the number of various servers can be set according to actual requirements.

The scheduling server 600 includes: a notification module 601, which is used for the scheduling server to notify each serialization server to obtain its own sample data; in a preferred embodiment of the present invention, in practical applications, the scheduling server 600 It also includes: an even distribution module, which is used to distribute all sample data evenly, and notify each serialization server to obtain a batch of sample data belonging to each serialization server according to the distribution result.

Before the scheduling server 600 starts the whole training, the notification module is also used to notify the serialization server to perform the string serialization process, so that it does not serialize the sample data first, because the management server does not have a mapping table at this time .

Wherein, each serialization server 700 includes: a sample acquisition module 701 for reading sample data according to the notification; a character string extraction module 702 for extracting each character string from the sample data; In applications, the character string extraction module 702 is also used to deduplicate the extracted character string before entering.

The character string remainder module 703 is used to divide the hash value corresponding to the character string by the number of management servers for each character string to obtain the remainder; the first remainder determination module 704 is used to determine the remainder according to the remainder and management server The corresponding relationship of the server, to determine the management server corresponding to the string.

The character string sending module 705 is used for sending the character string to the corresponding management server; the serialization ID receiving module 706 is used for receiving the serialized ID corresponding to each character string returned by each management server; the sample sequence The transformation module 707 is configured to convert the strings in each sample data into corresponding serialization IDs according to the received serialization IDs corresponding to the respective strings.

The output module 708 is used for outputting the serialized sample data.

In another embodiment of the present invention, in order to provide support for the management server to create a mapping table, the serialization server 700 includes: an integrated sending module for each serialization server to obtain its own A batch of sample data, and all the strings in the sample data are integrated and sent to the management server.

Each management server 800 includes: a character string receiving module 801 for receiving character strings; and receiving character strings sent by the character string sending module 705 .

The string conversion module 802 is used to convert the received string into a serialized ID according to the locally maintained mapping table; the digitized ID return module 803 is used to return the serialized ID corresponding to the string For the corresponding serialization server, in another embodiment of the present invention, the management server 800 also creates a mapping table through the following modules: a character string acquisition module, used to obtain the information belonging to the current management server itself Among them, the batch of strings belonging to the current management server is different from the strings belonging to other management servers; the strings obtained by the string acquisition module can be sent by the integrated sending module of the serialization server to get the string.

The mapping table building module is used for serializing the string and constructing a mapping table between strings and serialization IDs.

In the embodiment of the present invention, first, the mapping table required for serialization is distributed among multiple management servers, the strings maintained in the mapping tables of different management servers are different from each other, and the digitized IDs of the corresponding strings are also different. The complete mapping table is distributed to multiple management servers. When the management server finds the serialized ID of the string, the query time is short, which can reduce the query time of the serialized ID of the string, thereby reducing the serialization of samples. time and improve serialization efficiency.

Secondly, the serialization server only needs to send the sample to be serialized and the character string in it to the corresponding management server according to the corresponding relationship with each management server; then the management server will obtain the character string The serialization ID of the string is returned to the serialization server. In this way, the serialization server itself does not store the complete mapping table required for serialization, which avoids the memory shortage of the serialization server and improves the performance of the serialization server.

As for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts.

Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

It should be understood by those of ordinary skill in the art that embodiments of the embodiments of the present invention may be provided as a method, an apparatus, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may employ computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. form.

In a typical configuration, the computer device includes one or more processors (CPUs), an input/output interface, a network interface, and memory. Memory may include non-persistent storage in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of read only memory (ROM) or flash memory ( flash RAM). Memory is an example of a computer-readable medium. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and storage of information can be accomplished by any method or technology. Information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM) , read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, cassette tape, magnetic tape storage or other magnetic storage device or any other non-transmission medium that can be used for storage that can be accessed by a computing device information. As defined herein, computer-readable media does not include non-persistent computer-readable media (transitory media), such as modulated data signals and carrier waves.

Embodiments of the present invention are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal equipment to produce a machine that causes the instructions to be executed by the processor of the computer or other programmable data processing terminal equipment Means are created for implementing the functions specified in the flow or flows of the flowchart and/or the block or blocks of the block diagrams.

These computer program instructions may also be stored in computer readable storage capable of directing a computer or other programmable data processing terminal equipment to operate in a particular manner, such that the instructions stored in the computer readable storage produce an article of manufacture comprising the instruction means , the instruction means implement the functions specified in the flow or flow of the flowchart and/or the block or blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce computer-implemented processing, so that the computer or other programmable terminal equipment The instructions executed on the above provide steps for implementing the functions specified in the flow diagram flow or flow diagrams and/or the block diagram flow diagram block or blocks.

While the preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may be made by those of ordinary skill in the art once the basic progressive concepts become known. Therefore, the scope of the appended claims is intended to be construed to include the preferred embodiment as well as all changes and modifications that fall within the scope of the embodiments of the present invention.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or terminal device comprising a list of elements includes not only those elements, but also a non-exclusive list of elements. other elements, or also include elements inherent to such a process, method, article or terminal equipment. Without further limitation, an element defined by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

A sample serialization method and a sample serialization device provided by the present invention have been introduced in detail above. Specific examples are used in this paper to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only for help. Understand the method of the present invention and its core idea; at the same time, for those with ordinary knowledge in the technical field, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, this specification The contents should not be construed as limiting the present invention.

Claims (14)

A sample serialization method, comprising: acquiring each character string in a sample to be serialized; determining a management server corresponding to each character string according to the correspondence between each character string and each management server; Send it to the corresponding management server, so that each management server can convert the received string into the corresponding serialized ID according to the mapping table maintained by the management server; wherein, the strings in the mapping table maintained by different management servers interact with each other. Not the same; receive the serialized ID corresponding to each string returned by each management server; convert the string in each sample data into the corresponding serialized ID according to the serialized ID corresponding to each received string, wherein, After the step of acquiring each string in the sample to be serialized, the method further includes: deduplicating each string. The method according to item 1 of the scope of the application, wherein the step of determining the management server corresponding to each character string according to the corresponding relationship between each character string and each management server comprises: corresponding the character string to the management server. The hash value of is divided by the number of management servers to obtain the remainder; according to the corresponding relationship between the remainder and the management server, the management server corresponding to the string is determined. According to the method described in item 1 or 2 of the claimed scope, the , before the step of acquiring each character string in the sample to be serialized, further includes: acquiring a batch of sample data belonging to the current serialization server after all sample data are evenly distributed by the scheduling server. A sample serialization method, comprising: receiving a character string; the character string is sent by a serialization server according to the correspondence between the character string and each management server; the character string is obtained by the serial server from sample data; Convert the received string into a serialized ID according to the locally maintained mapping table; wherein, the strings in the mapping tables maintained by different management servers are different from each other; return the serialized ID corresponding to the string to the The corresponding serialization server is used for the serialization server to convert the strings in each sample data into the corresponding serialization IDs according to the serialization IDs corresponding to the received strings; Before the step, it also includes: obtaining a batch of character strings belonging to the current management server; wherein, a batch of character strings belonging to the current management server is different from character strings belonging to other management servers; serializing the character strings , and build a mapping table of strings and serialized IDs. The method according to item 4 of the scope of the patent application, wherein the step of serializing the string and constructing a mapping table between the string and the serialized ID includes: Obtain the first total number N1 of character strings in each management server before the sorting of the current management server; add the first total number N1 to the number M of character strings of the current management server to obtain the second total number N2 ; Take [N1+1,N2] as the current management server's range of string serialization. The method according to claim 4 or 5, wherein the step of converting the received character string into a serialized ID according to a locally maintained mapping table includes: querying whether there is any in the locally maintained mapping table the character string; if the character string is in the locally maintained mapping table, obtain the serialization ID corresponding to the character string; if there is no such character string in the locally maintained mapping table, generate a sequence for the character string The serialized ID is added, and the string and the corresponding serialized ID are added to the mapping table. The method according to claim 4 or 5, wherein the batch of character strings belonging to the current management server includes: the remainder corresponding to each character string in the batch of character strings belongs to the current management server; The remainder is obtained by dividing the hash value corresponding to the character string by the number of each management server. A sample serialization device, comprising: a character string extraction module for obtaining each character string in a sample to be serialized; a management server determination module for obtaining each character string according to each character string and each management server The correspondence between the servers determines the management server corresponding to each character string; the character string sending module is used to send the character string to the corresponding management server for each management server to maintain according to the corresponding management server. A mapping table, which converts the received strings into corresponding serialized IDs; among them, the strings in the mapping tables maintained by different management servers are different from each other; the serialized ID receiving module is used to receive the return of each management server The serialized ID corresponding to each string; the sample serialization module is used to convert the string in each sample data into the corresponding serialized ID according to the serialized ID corresponding to each received string, where in After the string extraction module, it also includes: a deduplication module, which is used to deduplicate each string. The device according to item 8 of the scope of the application, wherein the management server determination module comprises: a character string remainder module for dividing the hash value corresponding to the character string by the number of management servers, The remainder is obtained; the first remainder determination module is used to determine the management server corresponding to the character string according to the corresponding relationship between the remainder and the management server. The device according to claim 8 or 9, wherein, before the character string extraction module, it includes: a first sample data acquisition module for acquiring after the scheduling server has evenly distributed all the sample data , which belongs to the current serialization server's own batch of sample data. A sample serialization device, comprising: a character string receiving module for receiving character strings; the character string is sent by a serialization server according to the correspondence between the character string and each management server; the character string is sent by a serialization server The server obtains from the sample data; the string conversion module is used to convert the received string into a serialized ID according to the locally maintained mapping table; among them, the strings in the mapping table maintained by different management servers They are different from each other; the digitized ID return module is used to return the serialized ID corresponding to the string to the corresponding serialization server, so that the serialization server can receive the serialized ID corresponding to each string. , convert the character strings in each sample data into corresponding serialized IDs, wherein, before the character string receiving module includes: a character string obtaining module for obtaining a batch of character strings belonging to the current management server; Among them, a batch of strings belonging to the current management server is different from strings belonging to other management servers; a mapping table building module is used to serialize the strings and build a mapping between strings and serialization IDs surface. The device according to claim 11, wherein the mapping table construction module comprises: a first quantity acquisition module, configured to acquire the character string in each management server before the sorting of the current management server The first total quantity N1; the second quantity obtaining module is used to obtain the second total quantity N2 by adding the first total quantity N1 to the number M of the character strings of the current management server; the serialization range determination module, using Yu takes [N1+1,N2] as the current pipe The range that the management server serializes the string. The device according to claim 11 or 12, wherein the character string conversion module includes: a query module for querying whether the character string exists in the locally maintained mapping table; the first digitized ID Obtain a module for obtaining the serialization ID corresponding to the string if the string is in the locally maintained mapping table; generate a module for if the string is not in the locally maintained mapping table, then A serialization ID is generated for the string, and the string and the corresponding serialization ID are added to the mapping table. The device according to claim 11 or 12, wherein the batch of character strings belonging to the current management server includes: the remainder corresponding to each character string in the batch of character strings belongs to the current management server; The remainder is obtained by dividing the hash value corresponding to the character string by the number of each management server.

Images