WO2022206054A1

WO2022206054A1 - Material incremental processing method and apparatus, electronic device, and computer storage medium

Info

Publication number: WO2022206054A1
Application number: PCT/CN2021/140584
Authority: WO
Inventors: 李贵阳; 闫创
Original assignee: 北京沃东天骏信息技术有限公司
Priority date: 2021-03-31
Filing date: 2021-12-22
Publication date: 2022-10-06
Also published as: CN113095874A

Abstract

Provided are a material increment processing method and apparatus, a device, and a storage medium. The method comprises: obtaining a material increment instruction (S101), the material increment instruction being used for indicating that material data stored in a source location is changed; performing attribute analysis on the material increment instruction, and determining, from among multiple preset queues according to the attribute analysis result, a target queue corresponding to the material increment instruction (S102); and using the target queue to process the material increment instruction to obtain target material information (S103). In this way, according to the attribute analysis result of the material increment instruction, the corresponding target queue is determined from among multiple preset queues, and the material increment instruction can be processed in real time by using the target queue, such that not only can the real-time performance and throughput of material incremental processing be improved, but the speed of processing the material incremental instruction can be increased, thereby shortening the time for the material change to take effect.

Description

A material incremental processing method, device, electronic device and computer storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202110351985.2 and the filing date of March 31, 2021, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The present application relates to the technical field of data processing, and in particular, to a material increment processing method, device, electronic device and computer storage medium.

Background technique

In the computer field, users are often required to upload the data of a specific item (generally referred to as materials), and then the system executes the corresponding plan according to the uploaded materials. For example, in the advertising delivery system, users need to upload advertising materials on the delivery platform. At this time, the delivery platform will store the advertising materials in the database, and then the material transmission module will transfer the advertising materials to the specified location (usually a network disk), Then, the advertising system obtains advertising materials for advertising delivery. That is to say, the advertising materials need to be transferred from the database to the designated location before the advertisement can be delivered.

However, in practical applications, when the user modifies the advertising material, it is usually hoped that the changed content will take effect as soon as possible. However, due to the poor real-time performance and small task throughput of the current material incremental processing process, it takes a long time for material changes to take effect.

SUMMARY OF THE INVENTION

The present application provides a material incremental processing method, device, electronic equipment and computer storage medium, which can not only improve the real-time performance and throughput of material incremental processing, but also improve the processing speed, thereby shortening the time for material change to take effect. .

The technical solution of the present application is realized as follows:

In a first aspect, an embodiment of the present application provides a method for processing material increments, the method comprising:

Obtain a material increment instruction; wherein, the material increment instruction is used to indicate that the material data stored in the source location is changed;

Perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result;

Use the target queue to process the material increment instruction to obtain target material information.

In a second aspect, an embodiment of the present application provides a material increment processing device, and the material increment processing device includes an acquisition unit, an analysis unit, and a processing unit; wherein,

an acquiring unit, configured to acquire a material increment instruction; wherein, the material increment instruction is used to indicate that the material data stored in the source location is changed;

an analysis unit, configured to perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result;

The processing unit is configured to process the material increment instruction by using the target queue to obtain target material information.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes a memory and a processor; wherein,

the memory for storing a computer program executable on the processor;

The processor is configured to execute the steps of the method according to the first aspect when running the computer program.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores a computer program, and when the computer program is executed by a plurality of processors, implements the steps of the method described in the first aspect.

The embodiments of the present application provide a material increment processing method, device, electronic device and computer storage medium, by acquiring a material increment instruction, wherein the material increment instruction is used to indicate that the material data stored in the source location is changed; Perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result; use the target queue to process the material increment instruction, and obtain Target material information. In this way, according to the attribute analysis result of the material increment instruction, the corresponding target queue is determined from multiple preset queues, so that the material increment instruction can be processed in real time by using the target queue, which can not only improve the real-time performance and throughput of material increment processing It can also improve the processing speed of material increment instructions, thereby shortening the time for material changes to take effect.

Description of drawings

1 is a schematic flowchart of a material incremental processing method provided by the related art;

FIG. 2 is a schematic flowchart of a material increment processing method provided by an embodiment of the present application;

3 is a schematic diagram 1 of an application scenario of a material increment processing method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram 2 of an application scenario of a material increment processing method provided by an embodiment of the present application;

FIG. 5 is a schematic diagram 3 of an application scenario of a material increment processing method provided by an embodiment of the present application;

6 is a schematic structural diagram of a material increment processing device provided by an embodiment of the present application;

7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of the composition and structure of an electronic device according to an embodiment of the present application.

Detailed ways

In order to have a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same or a different subset of all possible embodiments, and Can be combined with each other without conflict. It should also be pointed out that the term "first\second\third" involved in the embodiments of the present application is only to distinguish similar objects, and does not represent a specific ordering of objects. It is understood that "first\second\third" "Three" may be interchanged where permitted in a specific order or sequence, so that the embodiments of the present application described herein can be implemented in sequences other than those illustrated or described herein.

Before the embodiments of the present application are described in further detail, the nouns and terms involved in the embodiments of the present application will be described first. The nouns and terms involved in the embodiments of the present application are applicable to the following explanations:

Advertiser: The user who places the ad.

Advertising material: The content displayed within the advertisement. Specifically, the basic forms of advertising materials are text, picture, and flash type (a file type).

Remote Procedure Call (RPC): A protocol for requesting services from a remote computer over a network without requiring knowledge of the underlying network technology.

A data structure that can be called Protocol Buffer, or PB for short; specifically, it refers to a language-independent, platform-independent, and extensible method of serializing structured data.

It can be understood that the advertising delivery system is a platform for advertisers to upload advertising materials and set delivery methods, also known as a delivery platform (Platform). Here, the delivery platform generally corresponds to a specific database (called a material library, such as MySQL) for storing the advertising materials uploaded by the advertiser. Under this structure, when the delivery platform detects the material change, it needs to convert the advertising material information into PB data in units of advertising units and transmit it to the recall system (Retrieval). Bidding queue, displayed to users after successful bidding. Among them, the data interaction method agreed between the delivery platform and the recall system is the Network File System (NFS). Here, the network file system is also called a network disk.

In practical applications, all operations of advertisers need to be sent to the recall system in quasi-real time, such as uploading an advertisement image, pausing the advertisement plan, and expecting it to take effect at the minute level. After a few minutes, the user is browsing the corresponding advertisement space (such as a certain advertisement space). App banners) may display corresponding ads. In order to achieve this effect, the material transmission service needs to have strong real-time performance and large throughput.

For the modification operation of the advertiser, in the related art, the modified material transmission process needs to be realized by polling the change event. The specific principle is as follows: Create a task queue (task_queue) table in the database (MySQL) in the delivery platform module. When the delivery platform (Platform) monitors any operation of the advertiser, it will add a change record in it. The dimensions may be plans/units/creatives, etc.; then, the material transfer service obtains this value based on the offset that was processed last time (that is, to determine the unexecuted change record) and the window size that needs to be processed this time (that is, the task throughput). The next change instruction that needs to be processed, query the material library to obtain the modified material data, splicing the modified material data, and obtain the material unit data corresponding to the advertising unit (material unit data can also be called material information, generally PB data is used format representation) and persisted to the network disk.

In the actual execution process, for the material transmission method of polling change events, each operation of writing change events needs to be coupled in the Platform process. The process of writing the task_queue table and the process of changing material data ensure atomicity through database transactions. . The material transmission service needs to periodically pull the change records, and then reversely check the material information according to the change records, splicing it into advertising material data, and then writing it to the network disk.

Taking an advertiser's "modify plan budget" operation as an example, see FIG. 1 , which shows a schematic diagram of an application scenario of a material increment processing method provided by the related art. As shown in Figure 1, the application scenario includes a delivery platform module, a material transfer module, and a recall inversion module (Retrieval). In the task_queue table, it can include an identification field (Auto_pk), a type field (Task_type), and a content field (Task_rfid) and the Created time field.

At this time, the process of material transmission is as follows: the platform module needs to first verify the validity of the input parameters (that is, the validity of the input parameters); secondly, the schedule table is modified according to the input parameters, and the task_queue table is written, especially On the other hand, modifying the plan table (this step is embodied in modifying the budget material campaign field) and writing to the task_queue table need to be placed in a transaction to ensure atomicity; then, record the operation log to the log configuration tool (Elastic Search, ES ) for the advertiser to view; finally, the material transfer module periodically pulls the change record, retrieves the material of the changed advertising unit according to the material table of the change record, and re-splices it, obtains the material information corresponding to the advertising unit, and stores the material information Write to the network disk for recalling the inverted module.

However, when the above method is used to realize material transmission, for the delivery platform module, the records of the task_queue table are strongly coupled with the normal advertiser operations, and the perception of the change records of material transmission also depends on the frequency of regularly pulling the task_queue table. In this case, it is difficult to control the time period threshold for pulling the task_queue table and the number of modified records in the pulling task_queue table, which may lead to some defects. For example, the delivery logic is strongly coupled, and all material-related operations must rewrite the task_queue Tables; and splicing advertising material data according to the change records need to be executed sequentially, and the throughput is low; in addition, due to the poor real-time performance of regular polling, it takes a long time for material changes to take effect.

Based on this, an embodiment of the present application provides a material increment processing method, the basic idea of which is: acquiring a material increment instruction; wherein the material increment instruction is used to indicate that the material data stored in the source location is changed; Perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result; use the target queue to process the material increment instruction, and obtain Target material information. In this way, according to the attribute analysis result of the material increment instruction, the corresponding target queue is determined from multiple preset queues, so that the material increment instruction can be processed in real time by using the target queue, which can not only improve the real-time performance and throughput of material increment processing It can also improve the processing speed of material increment instructions, thereby shortening the time for material changes to take effect.

The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The methods in the embodiments of the present application may be executed by an electronic device. In some embodiments, the electronic device may be implemented as a terminal, and may also be implemented as a server, which is not limited in this embodiment of the present application.

In the embodiment of the present application, referring to FIG. 2 , it shows a schematic flowchart of a material increment processing method provided by the embodiment of the present application. As shown in Figure 2, the method may include:

S101: Obtain a material increment instruction.

It should be noted that the embodiments of the present application can be applied to the field of e-commerce material delivery, the field of advertising material delivery, and the like. It should be understood that, in the case where the information is changed, when the modified information needs to be transmitted from one location to another location, the method provided by the implementation of the present application can be adopted.

The embodiments of the present application all take the advertisement material delivery scenario as an example for subsequent description.

In practical applications, the advertising material delivery scenario generally includes a delivery platform module, a material transmission module, and a recall inversion module. First, for the advertiser side, it is necessary to upload and modify the advertising materials through the delivery platform module; secondly, the delivery platform module stores the received advertising materials in the source location (such as a database); thirdly, the material transmission module determines according to the database. The modified advertising unit is re-spliced into the new material information of the advertising unit; finally, the material transmission module transmits the new material information of the advertising unit to the target location (such as a network file storage system), waiting for subsequent advertising.

In order to complete the above process, it is necessary to determine the material increment instruction. Here, the material increment instruction is used to indicate that the material data at the source location is changed, and the source location refers to the data storage location on the side of the material delivery user (such as an advertiser).

In some embodiments of the present application, taking the source location as a database as an example, the determining at least one material increment instruction may include: collecting log information of the database when a material change event occurs in the database; The log information of the database is filtered by using preset keywords, and the material increment instruction is generated according to the filtering result.

It should be noted that, in the case where the source location is a database, the log information of the database can be filtered by using a preset keyword, thereby generating a material increment instruction. Here, the preset keyword is pre-configured according to the application scenario, and is used to extract a field value of the database log to obtain the changed data location, and then generate the material increment instruction according to the changed data location.

In some embodiments of the present application, when a material change event occurs in the database, log information (such as a binary log) of the database is collected. Here, you can use the built-in tools of the database to detect the database. When a material change event occurs in the database, the binary log of the database (equivalent to log information) is obtained; then, the log of the database is filtered according to the preset keywords. and extraction to generate material increment instructions. Here, the material change event includes not only modifying the existing material data, but also writing new material data.

In the related art, it is also necessary to create a task_queue table in the database. When the delivery platform module monitors any operation of the advertiser, a change record is added in it, and then the material transmission module is processed according to the last offset ( That is, determine the last processed change record) and the window size to be processed this time (that is, determine the number of change records to be processed this time), and obtain the material increment instruction that needs to be processed this time, so as to carry out subsequent operations.

That is to say, in the related art, the material delivery logic is strongly coupled with the advertiser, and all material-related operations must remember to rewrite the task_queue table, which is inefficient. However, in the embodiment of the present application, the database's own tools are used to monitor modification events, and use the database's own tools to monitor modification events. The binary log is used to obtain material increment instructions. The operation is simple and efficient, and the task_queue table is no longer required.

In this way, the material increment instruction is obtained through the above mechanism for subsequent processing of the material increment instruction.

S102: Perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction from a plurality of preset queues according to the attribute analysis result.

It should be noted that, in the relevant information, after obtaining the material increment instruction, generally only a single queue is used for execution, and the task throughput is small and the real-time performance is poor. Therefore, the embodiment of the present application adopts a multi-queue processing mode, and according to the attributes of the material increment instruction, it can be allocated to the target queues among the multiple preset queues. Here, multiple preset queues can be executed separately by different processes to improve task throughput and processing speed.

It should be understood that the material increment instruction may indicate a specific instruction for modifying one advertising material, or may include multiple specific instructions for respectively instructing the modification of multiple advertising materials. In other words, for the material delivery method in the embodiment of the present application, if there are multiple specific instructions, the specific instructions will be allocated to different preset queues according to the attribute analysis result, thereby improving throughput and real-time performance.

In some embodiments of the present application, the multiple preset queues may include a first preset queue and a second preset queue, and determining the target queue from the multiple preset queues according to the attribute analysis result may include: In the case that the attribute analysis result satisfies the preset screening condition, the first preset queue is determined as the target queue, and the material increment instruction is stored in the first preset queue; In the case that the attribute analysis result does not meet the preset screening conditions, the second preset queue is determined as the target queue, and the material increment instruction is stored in the second preset queue; wherein, the The first preset queue is different from the second preset queue.

It should be noted that, for the attribute analysis result of the material increment instruction, if the attribute analysis result satisfies the preset filter conditions, the material increment instruction is allocated to the first preset queue; if the attribute analysis result does not meet the preset filter conditions, then Allocate the material increment instruction to the second preset queue. Here, the first preset queue is used to store some instructions with low execution speed or low priority (that is, material increment instructions that meet the preset screening conditions), which can also be called the slow queue, and the second preset queue is used for It is used to store other common instructions (that is, material increment instructions that do not meet the preset screening conditions), which can also be called standard queues. Due to the difference in the material increment instructions included, the execution speed of the slow queue is slower than that of the standard queue.

In other words, multiple preset queues can include slow queues and standard queues. For a small number of special material increment instructions, such as many modifications, long execution time, and low credit score of advertisers, this part of the material will be Incremental instructions are put into the slow queue for execution; for most common material incremental instructions, they are put into the standard queue. Here, the preset filter conditions can be determined according to actual application scenarios.

It should be noted that, for the first preset queue and the second preset queue, there is also a token bucket mechanism. Generally speaking, the token bucket capacity of the first preset queue is smaller than the token bucket capacity of the second preset queue. . Here, the principle of the token bucket is: put the token into the bucket at a constant speed, if the request needs to be processed, you need to get a token from the bucket first, when there is no token in the bucket, then Denial of service. In other words, the capacity of the token bucket represents the maximum number of tasks in the queue. By adjusting the size of the token bucket capacity, the maximum number of tasks in the first preset queue and the second preset queue can be controlled to avoid system failure.

In this way, according to the analysis result of the material increment instruction, the corresponding target queue can be determined in multiple preset queues, and the material increment instruction can be processed in a targeted manner, thereby improving the efficiency of material transmission.

S103: Based on the material data stored in the source location, process the material increment instruction through the target queue to obtain target material information.

It should be noted that, for the target queue, the corresponding material data is obtained from the source location according to the material increment instruction for processing, so as to obtain the target material information.

It should be noted that the material increment instruction may include a first material increment instruction and a second material increment instruction, and the first material increment instruction is stored in the first preset queue, and the first material increment instruction is stored in the first preset queue. Two material increment instructions are stored in the second preset queue. Therefore, in some embodiments of the present application, using the target queue to process the material increment instruction to obtain target material information may include: using the first preset queue and the second preset queue A queue is set to process the first material increment instruction and the second material increment instruction in parallel to obtain the target material information.

It should be noted that if the material increment instruction includes the first material increment instruction and the second material increment instruction, and after attribute analysis, the first material increment instruction is stored in the first preset queue, and the second material increment instruction is stored in the first preset queue. If the quantity instruction is stored in the second preset queue, the first preset queue and the second preset queue can be executed in parallel through two specific processes, so as to obtain the material information of the first material increment instruction and the second material increment instruction material information.

In this way, the concurrent multi-queue mode can ensure that the material increment command can be executed efficiently, and the classification of slow queues and standard queues can avoid the blockage of a certain task, so that the material increment command takes effect in high real-time, and multiple Preset queues also increase task bandwidth and expand task throughput.

In some embodiments of the present application, the obtaining target material information based on the processing of the material increment instruction through the target queue may include: acquiring the material increment instruction from the target queue, Determine the target material identifier according to the material increment instruction; determine the target material data set from the material data stored in the source location according to the material identifier; perform data splicing processing on the target material data set to obtain the target Material information.

It should be noted that the material data is stored in the source location in the form of an advertisement unit, that is, the source location stores multiple material identifiers and respective material data sets of the multiple material identifiers. Here, a material ID represents a specific advertising plan, or a material ID represents an advertising unit.

Therefore, the acquisition process of the target material information includes the following steps: first, take out the material increment instruction from the target queue, and determine the modified target material identifier according to the material increment instruction; then, according to the target material identifier, from the The target material data set corresponding to the material identifier is determined in the material data stored in the source location; finally, the data in the target material data set is spliced to obtain target material information.

In one application scenario of the present application, the material data corresponding to an advertisement can be divided into multiple levels, such as user (user) data, plan (campaign) data, unit (ad_group) data, and creative (ad) data. This is called an ad unit. In other words, for the database, the respective data of multiple ad units are stored, and the data of one ad unit includes data of four different levels of user/campaign/ad_group/ad. At this time, there will be an independent material identification (Identity Document, ID) for each advertisement, and the material ID of the modified material will be carried in the material increment instruction, but the specific modification information will not be recorded.

It should also be noted that, according to the foregoing content, in the field of advertising materials, the material increment instruction specifically indicates a certain material unit that has been modified. In practical application scenarios, advertisers may repeatedly modify a material unit in a short period of time. In this case, in order to avoid redundant operations, the first preset queue and the second preset queue can be designed without The form of an ordered set, thus avoiding repeated operations. For example, the material transmission module determines from the log of the database that the material unit 335 has been modified, and generates (material increment instruction: material unit 335); and assigns (material increment instruction: material unit 335) to the second pre-order Set up a queue, if before the modification of (material increment instruction: material unit 335) is processed, the material transfer module determines from the database log again that material unit 335 has been modified twice, and once again generates (material increment instruction : material unit 335). However, since the second preset queue is an unordered set, the new material increment instruction will not be put into the second preset queue again. On the other hand, when (material increment instruction: material unit 335) is executed, the latest material data needs to be acquired from the database for splicing, and the result has already covered two modifications.

Therefore, by defining the second preset queue as an unordered set, deduplication can be performed naturally and redundant operations can be avoided. However, there is also a disadvantage of unordered collections, that is, due to the randomness of execution, when new material increment instructions flow in continuously, it is possible that an old material increment instruction will never be flushed out for processing. Therefore, this drawback can be avoided by using the flush queue mechanism.

In some embodiments of the present application, the plurality of preset queues further include a third preset queue, and the method may further include: starting a timer; The material increment instruction stored in the second preset queue is moved to the third preset queue.

It should be noted that the plurality of preset queues further include a third preset queue, and the third preset queue refers to a flushing queue of the second preset queue. In the embodiment of the present application, the material increment instruction stored in the second preset queue may be moved to the third preset queue according to a preset preset duration.

In the embodiment of the present application, after the attribute analysis is performed, the common material increment instruction is put into the second preset queue; All unprocessed material increment instructions in the preset queue are moved to the third preset queue. Correspondingly, each time a material increment instruction is processed, the third preset queue is preferentially processed, thereby avoiding the disadvantage that a certain material increment instruction cannot be processed for a long time. Here, the third preset queue is also an unordered set.

In some embodiments of the present application, the obtaining the material increment instruction from the target queue may include: in the case that the attribute analysis result satisfies a preset screening condition, obtaining the material increment instruction from the first preset queue Obtain the material increment instruction from ; if the attribute analysis result does not meet the preset screening conditions, determine whether the third preset queue is empty; if the third preset queue is not empty, then The material increment instruction is acquired from the third preset queue; if the third preset queue is empty, the material increment instruction is acquired from the second preset queue.

It should be noted that, for the first preset queue, the material increment instructions in the first preset queue can be fetched and executed in sequence. For the second preset queue and the third preset queue, since the third preset queue is actually to prevent the instructions in the second preset queue from not being executed for a long time, the third preset queue takes precedence over the second preset queue. The default queue is executed. In other words, when the third preset queue is not empty, the instructions in the third preset queue need to be executed first; when the third preset queue is empty, the instructions in the second preset queue are executed. Material increment instruction.

It should also be noted that, for the material increment instruction of the first preset queue, the second preset queue or the third preset queue, execution may fail due to program failure, logical problem or physical failure. Therefore, in some embodiments of the present application, the plurality of preset queues may further include a fourth preset queue, and in the case that the processing of the material increment instruction fails, the method may further include: based on the material Incremental instruction, a retry instruction is generated in the fourth preset queue; according to the retry instruction, the material increment instruction is retried through the fourth preset queue.

It should be noted that in the event that the processing of the material increment instruction fails, a retry instruction will be generated in the fourth preset queue accordingly, so that the failed material increment instruction will be executed again in the fourth preset queue. , to avoid blocking a queue. It should be understood that the first preset queue, the second preset queue/third preset queue, and the fourth preset queue are performed concurrently through three processes (the second preset queue and the third preset queue share one process). .

In some embodiments of the present application, in the event that the material increment instruction fails to be processed, the method may further include: generating alarm information based on the material increment instruction; wherein the alarm information is used to inform the management The personnel prompts that the material increment instruction is abnormal.

It should be noted that, in the case that the processing of the material increment instruction fails, alarm information can also be generated so as to prompt that the material increment instruction is in an abnormal state, so as to remind the management personnel to handle it. In addition, if the material increment instruction in the fourth preset queue fails to be executed again, the processing strategy of infinite retry and alarm (ie, alarm) may be adopted.

In some embodiments of the present application, the method further includes: transmitting the target material information to a target location; sorting the target material information at the target location, and performing material delivery according to the sorting result.

It should be noted that, by processing the preset queues (the first preset queue, the second preset queue, the third preset queue, and the fourth preset queue), the target material information can be obtained, and the target material information can be transmitted. Go to the target location (usually a distributed network file system), and then recall the inverted module to obtain the target material information from the target location and enter the sorting step, so as to carry out subsequent material delivery.

It can be understood that, through the above processing, the material is transferred from the source position to the target position, which improves the real-time performance and throughput of material transmission.

The embodiment of the present application provides a material increment processing method, by acquiring a material increment instruction; wherein, the material increment instruction is used to indicate that the material data stored in the source location is changed; attribute the material increment instruction parsing, determining a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute parsing result; using the target queue to process the material increment instruction to obtain target material information. In this way, according to the attribute analysis result of the material increment instruction, the corresponding target queue is determined from multiple preset queues, so that the material increment instruction can be processed in real time by using the target queue, which can not only improve the real-time performance and throughput of material increment processing It can also improve the processing speed of material increment instructions, thereby shortening the time for material changes to take effect.

In some embodiments of the present application, referring to FIG. 3 , it shows a schematic diagram of an application scenario of a material increment processing method provided by an embodiment of the present application. As shown in FIG. 3 , the application scenario includes a delivery platform module 201 , a material transmission module 202 and a recall inversion module 203 , and the delivery platform module includes a database 2011 .

It can be understood that, the material increment processing method provided in the embodiment of the present application focuses on the interaction between the delivery platform module 201 and the material transmission module 202 and internal processes.

Based on this application scenario, taking the advertiser modifying the planned budget as an example, the process of incremental material processing is as follows:

First, the delivery platform module 201 receives the modification instruction of the advertiser, and for the modification instruction, performs the steps of parameter validity check, modification plan table and recording operation log; here, modifying the plan table refers to modifying the plan table according to the operation of the advertiser The material data in the database 2011 and the recording operation log are used to present the corresponding modification records to the advertiser.

Secondly, after detecting the modification event of the database 2011, obtain the binary log (Binlog) of the database 2011, filter the Binlog through the configured listening keyword (equivalent to the aforementioned preset keyword), determine the material increment instruction, and The material increment instruction is transmitted to the increment instruction queue of the material transmission module 202 by means of a message queue (Message Queue, MQ) (equivalent to the aforementioned multiple preset queues).

Then, in the material transmission module 202, according to the material increment instruction in the increment instruction queue, the material table in the database is reversely checked to obtain the target material information (or material unit data) of the PB data structure, and the material unit is Data is written to the network disk.

Finally, the recall and inversion module 203 takes out the material unit data (equivalent to the aforementioned target material information) from the network disk, and enters a specific sorting and placing process.

In other words, the core of the material increment processing method provided by the embodiment of the present application is mainly to upgrade the information interaction mode, so that the responsibilities of each module are clearer and a self-closed loop is formed; the overall application architecture can also be optimized, and the material online ( Ad modifications take effect) time. For the delivery platform module, there is no need to write the task_queue table separately for each operation that needs to trigger the material increment. After the advertising material data falls into the database, the changes of the database table can be obtained by collecting the binary log (Binlog) at the database level. , configure the table/field changes to be detected (equivalent to monitoring keywords) for filtering. It should be understood that this process needs to adjust the Binlog mode of the database to a row (Row) mode. Finally, use the message queue cluster tool as a message relay, and this tool can also be used as a stress test tool for playback at selected sites.

As can be seen from the above, in FIG. 3 , the “incremental instruction queue” in the material transmission module 202 actually corresponds to the core logic of the entire material transmission process, so it will be described in detail. For the incremental instruction queue, the middleware that is mainly relied on is Redis (an open source in-memory data structure storage system).

In some embodiments of the present application, referring to FIG. 4 , it shows a schematic diagram of an application scenario of another material increment processing method provided by an embodiment of the present application. As shown in FIG. 4 , the material transmission module 202 includes an instruction receiving sub-module 2021 , a token bucket current limiting sub-module 2022 , a multi-queue processing sub-module 2023 , a unit assembly sub-module 2024 and a result output sub-module 2025 . Among them, the command receiving sub-module 2021 is responsible for converting external commands into a form that can be processed inside the material transmission, the token bucket current limiting sub-module 2022 is used to manage the number of commands that can be processed by a certain queue at the same time, and the multi-queue processing sub-module 2023 In order to ensure that the data processing is not lost in order (that is, the order of data in the queue is guaranteed without data loss) and real-time stability, the unit assembly sub-module 2024 uses a multi-threaded method to assemble the material information in parts, and outputs the sub-module through the final result. Module 2025 persists the data to the network disk.

For each of the above modules, the descriptions are as follows:

The instruction receiving sub-module 2021 is mainly responsible for converting external information (that is, the material modification information determined by the delivery platform module) into an object for internal processing of material transmission. In addition to receiving the material incremental instructions formed by the database Binlog, it also provides a Hyper Text Transfer Protocol (HTTP)/RPC protocol interface, which supports sending instructions by manual triggering to trigger the incremental process.

The token bucket current limiting sub-module 2022 is mainly responsible for limiting the number of instructions processed in a certain queue at the same time. Through the configuration center tool, the number of tokens in the token bucket can be dynamically adjusted to ensure the stable operation of the service. In other words, the token bucket current limiting sub-module 2022 is mainly used to limit the system to only process a preset number at the same time (ie number of tokens), for the excess part, you need to wait for the token bucket space.

For the multi-queue processing sub-module 2023, it includes the slow queue slow (equivalent to the aforementioned first preset queue)/standard queue normal (equivalent to the aforementioned second preset queue, also called a normal queue)/flushing queue flush ( It is equivalent to the aforementioned third preset queue, which may also be referred to as a buffer queue)/retry queue retry (equivalent to the aforementioned fourth preset queue) and execution queue running. These different queues perform their respective duties to ensure that the incremental instructions can be executed in a timely manner without losing the order and being stable in real time.

In the embodiment of the present application, for the slow queue, the number of tokens in the token bucket of the slow queue is small, and the execution process is the same as that of other queues. As the name implies, the execution speed of the queue is relatively slow. For some business scenarios, such as the advertiser's low credit score, the system will put all the advertiser's instructions in the slow queue; in addition, if a certain instruction may trigger the reassembly of a large number of units, for example, a system crowd is It is tied to 50w units. If the crowd information changes, all units associated with it will be changed. At this time, this instruction is placed in the slow queue for slow execution, which can ensure that the normal queue is smooth and no backlog.

For the standard queue-flush queue, the normal and flush queues are actually unordered collections (Redis's set function), which is characterized by the ability to naturally deduplicate unordered. Here, advertisers often modify a unit multiple times in a short period of time, so the feature of deduplication is very important, which can effectively improve work efficiency. For normal Binlog and instruction parsing, it will go to the standard queue, but using only one standard queue will lead to the problem of zombie instructions, that is, the standard queue continues to flow into incremental instructions, and a certain instruction may never be popped off the stack theoretically ( Pop). Therefore, it is necessary to periodically flush the standard queue to the flushing queue. When selecting an instruction, first determine whether the flushing queue is empty, and then take the standard queue if it is empty.

For the retry queue, it is very likely that the assembly unit fails due to system jitter or logical abnormality. At this time, an alarm will be triggered, and a retry command will be formed and placed in the retry queue. In some embodiments of the present application, the policy configured by the number of retries may be unlimited retries plus an alarm, which requires manual intervention by an administrator.

For the execution queue, put the unit ID (equivalent to the aforementioned material identifier) into the execution queue before unit assembly, and remove the unit ID from the execution queue after information assembly and result output to ensure that the unit processing is not lost.

The unit assembly sub-module 2024 is mainly responsible for querying the database or adjusting the interface to obtain the latest material data set of the unit according to the unit ID, and splicing all the data in the material data set into the material unit data of the PB data structure. In some embodiments of the present application, a material data set may be divided into four levels: user/campaign/ad_group/ad (user/plan/unit/creative) according to the level of advertising materials, and further subdivided into smart bidding, crowd targeting , keywords, commodity orientation and other eight modules, the unit assembly module uses a multi-threaded parallel method to query and finally summarize.

For the result output sub-module 2025, the previously spliced material unit data (or called unit information, material information) is persisted to the network disk PB file, and the data can be quickly rolled back by retaining the latest 20 small versions .

In some embodiments of the present application, referring to FIG. 5 , it shows a schematic diagram of an application scenario of another material increment processing method provided by an embodiment of the present application. As shown in FIG. 5 , the plurality of preset queues may include a slow queue, a standard queue, a flush queue and a retry queue. Assuming that the material increment instruction can include the first material increment instruction, the second increment instruction, and the third increment instruction, the above material increment instruction is subjected to attribute analysis through the instruction attribute analyzer, and the first material increment is added according to the attribute analysis result. The quantity instruction is stored in the slow queue, the second material increment instruction and the third material increment instruction are stored in the standard queue, and the second material increment instruction is flushed to the flushing queue. At this time, the first material increment instruction is taken out from the slow queue through the first execution queue, and the first material increment instruction is executed through the unit assembly sub-module and the result output sub-module, and the corresponding value of the first material increment instruction is output. Target material information; take out the second material increment instruction from the flushing queue through the second execution queue, execute the second material increment instruction through the unit assembly sub-module and the result output sub-module, and output the corresponding second material increment instruction target material information.

When the flushing queue is empty, take out the third material increment instruction from the standard queue, and execute the third material increment instruction through the unit assembly sub-module and the result output sub-module, and output the target corresponding to the third material increment instruction material information;

In addition, assuming that the execution of the first material increment instruction fails, a retry instruction is generated in the retry queue, the retry instruction is taken out from the retry queue by the third execution queue, and the retry instruction is checked by the unit assembly sub-module and the result output sub-module. Execute the trial command (equivalent to re-executing the first material increment command).

It can be seen from the above that the multi-queue processing sub-module, the unit assembly sub-module and the result output sub-module complement each other and jointly support the core logic of the material incremental processing method; and the first execution queue, the second execution queue and the third execution queue can Concurrent execution improves task throughput and processing speed.

In short, the embodiments of the present application provide a high-throughput, quasi-real-time advertising incremental material transmission solution. Among them, good real-time performance and throughput can enable advertisers' operations to be launched in minutes, and settings such as budget, bid, and orientation can take effect quickly; and its core is the multi-queue design logic of the material incremental processing method, which can ensure the system At the same time of stable operation, the layers are distinct and complement each other, which ultimately ensures the rapid delivery of materials.

It can be understood that the responsibilities of the delivery platform module and the material transmission module are clearer through the processes provided in the embodiments of this application, and this method has strong reusability for incremental information transmission of other business types. The configuration also achieves good flexibility and ease of use.

In some embodiments of the present application, referring to FIG. 6 , it shows a schematic diagram of the composition and structure of a material increment processing device 30 provided in an embodiment of the present application. As shown in FIG. 6, the material increment processing device 30 includes an acquisition unit 301, an analysis unit 302 and a processing unit 303, wherein the acquisition unit 301 is configured to acquire a material increment instruction; wherein, the material increment instruction uses The material data stored at the indicated source location is changed; the parsing unit 302 is configured to perform attribute analysis on the material increment instruction, and determine the target corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result Queue; the processing unit 303 is configured to use the target queue to process the material increment instruction to obtain target material information.

In some embodiments of the present application, the acquiring unit 301 is configured to collect log information of the database when a material change event occurs in the database; and filter the log information of the database by using preset keywords , and generate the material increment instruction according to the filtering result.

In some embodiments of the present application, the parsing unit 302 is further configured to determine the first preset queue as the target queue under the condition that the attribute parsing result satisfies the preset filter condition, and to The material increment instruction is stored in the first preset queue; if the attribute analysis result does not meet the preset screening conditions, the second preset queue is determined as the target queue, and the The material increment instruction is stored in the second preset queue; wherein the first preset queue is different from the second preset queue.

In some embodiments of the present application, the material increment instruction includes a first material increment instruction and a second material increment instruction, and the first material increment instruction is stored in the first preset queue, the second material increment instruction is stored in the second preset queue;

Correspondingly, the processing unit 303 is further configured to use the first preset queue and the second preset queue to perform parallel processing on the first material increment instruction and the second material increment instruction, to obtain The target material information.

In some embodiments of the present application, the processing unit 303 is configured to obtain the material increment instruction from the target queue, and determine a target material identifier according to the material increment instruction; The target material data set is determined from the material data stored in the source location; the data splicing process is performed on the target material data set to obtain the target material information.

In some embodiments of the present application, the parsing unit 302 is further configured so that the plurality of preset queues further include a third preset queue, and starts a timer; when the timing duration of the timer reaches a preset duration, The third preset queue is determined as the target queue, and the material increment instruction stored in the second preset queue is moved to the third preset queue.

In some embodiments of the present application, the processing unit 303 is further configured to obtain the material increment instruction from the first preset queue when the attribute parsing result satisfies a preset filter condition; In the case that the attribute analysis result does not meet the preset screening conditions, determine whether the third preset queue is empty; if the third preset queue is not empty, obtain from the third preset queue the material increment instruction; if the third preset queue is empty, obtain the material increment instruction from the second preset queue.

In some embodiments of the present application, the plurality of preset queues further include a fourth preset queue, and the processing unit 303 is further configured to, in the case that the processing of the material increment instruction fails, based on the material increment amount instruction, and a retry instruction is generated in the fourth preset queue; according to the retry instruction, the material increment instruction is retried through the fourth preset queue.

In some embodiments of the present application, the processing unit 303 is further configured to generate alarm information based on the material increment instruction in the event that the material increment instruction fails to be processed; wherein the alarm information is used for Notify the manager that there is an abnormality in the material increment instruction.

In some embodiments of the present application, the processing unit 303 is further configured to transmit the target material information to the target location; sort the target material information at the target location, and perform material delivery according to the sorting result.

It can be understood that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, and it may also be non-modular. Moreover, each component in this embodiment may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.

If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or The part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, this embodiment provides a computer storage medium storing a computer program, the computer program implementing the steps of the method in any one of the foregoing embodiments when executed by a plurality of processors.

Based on the composition of the above-mentioned material increment processing device 30 and the computer storage medium, see FIG. 7 , which shows a schematic diagram of a hardware structure of an electronic device 40 provided by an embodiment of the present application. As shown in FIG. 7 , the electronic device 40 may include: a communication interface 401 , a memory 402 and a processor 403 ; various components are coupled together through a bus device 404 . It can be understood that the bus device 404 is used to implement the connection communication between these components. In addition to the data bus, the bus device 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity of illustration, the various buses are labeled as bus devices 404 in FIG. 7 . Among them, the communication interface 401 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

memory 402 for storing computer programs that can run on processor 403;

The processor 403 is configured to, when running the computer program, execute:

It can be understood that the memory 402 in this embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically programmable read-only memory (Erasable PROM, EPROM). Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory may be Random Access Memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous link DRAM (Synchronous link DRAM, SLDRAM) ) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 402 of the apparatus and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

The processor 403 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 403 or an instruction in the form of software. The above-mentioned processor 403 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 402, and the processor 403 reads the information in the memory 402, and completes the steps of the above method in combination with its hardware.

It will be appreciated that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Device (DSP Device, DSPD), programmable Logic Devices (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), General Purpose Processors, Controllers, Microcontrollers, Microprocessors, Others for performing the functions described herein electronic unit or a combination thereof.

For a software implementation, the techniques described herein may be implemented through modules (eg, procedures, functions, etc.) that perform the functions described herein. Software codes may be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

In some embodiments of the present application, the processor 403 is further configured to execute the steps of the method described in any one of the preceding embodiments when running the computer program.

In some embodiments of the present application, based on the schematic composition diagram of the above-mentioned material increment processing device 30 , see FIG. 8 , which shows a schematic composition structure diagram of another electronic device 40 provided by the embodiments of the present application. As shown in FIG. 8 , the electronic device 40 includes at least the material increment processing device 30 described in any one of the foregoing embodiments.

For the electronic device 40, since it includes the material increment processing device 30 described in any one of the foregoing embodiments, through the material increment processing device 30, the attribute analysis result of the material increment instruction can be obtained, and then Determine the corresponding target queue from multiple preset queues, and use the target queue to process material incremental instructions in real time, which can not only improve the real-time performance and throughput of material incremental processing, but also improve the processing of material incremental instructions speed, thereby shortening the time for material changes to take effect.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application.

It should be noted that, in this application, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements , but also other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined under the condition of no conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain a new product embodiment.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Industrial Applicability

Claims

A material increment processing method, the method comprises:

Obtain a material increment instruction; wherein, the material increment instruction is used to indicate that the material data stored in the source location is changed;

Perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result;

Use the target queue to process the material increment instruction to obtain target material information.
The material increment processing method according to claim 1, wherein the source location is a database, and the acquiring a material increment instruction comprises:

Collect log information of the database when a material change event occurs in the database;

The log information of the database is filtered by using preset keywords, and the material increment instruction is generated according to the filtering result.
The material increment processing method according to claim 1, wherein the multiple preset queues include a first preset queue and a second preset queue, and the predetermined queue is determined according to the attribute analysis result among the multiple preset queues. The target queue corresponding to the above material increment instruction, including:

In the case that the attribute analysis result satisfies the preset screening condition, the first preset queue is determined as the target queue, and the material increment instruction is stored in the first preset queue;

In the case that the attribute analysis result does not meet the preset screening conditions, determining the second preset queue as the target queue, and storing the material increment instruction in the second preset queue;

Wherein, the first preset queue is different from the second preset queue.
The material increment processing method according to claim 3, wherein the material increment instruction comprises a first material increment instruction and a second material increment instruction, and the first material increment instruction is stored in the a first preset queue, the second material increment instruction is stored in the second preset queue;

Correspondingly, using the target queue to process the material increment instruction to obtain target material information, including:

The first material increment instruction and the second material increment instruction are processed in parallel by using the first preset queue and the second preset queue to obtain the target material information.
The material increment processing method according to claim 3, wherein the processing of the material increment instruction by using the target queue to obtain target material information comprises:

Obtain the material increment instruction from the target queue, and determine the target material identifier according to the material increment instruction;

According to the target material identifier, determine the target material data set from the material data stored in the source location;

Perform data splicing processing on the target material data set to obtain the target material information.
The material increment processing method according to claim 5, wherein the plurality of preset queues further comprises a third preset queue, and the method further comprises:

start the timer;

When the timing duration of the timer reaches a preset duration, the third preset queue is determined as the target queue, and the material increment instruction stored in the second preset queue is moved to the second preset queue. Three preset queues.
The material increment processing method according to claim 6, wherein the acquiring the material increment instruction from the target queue comprises:

Obtain the material increment instruction from the first preset queue under the condition that the attribute analysis result satisfies the preset screening condition;

In the case that the attribute analysis result does not meet the preset screening conditions, determine whether the third preset queue is empty;

If the third preset queue is not empty, acquiring the material increment instruction from the third preset queue;

If the third preset queue is empty, the material increment instruction is acquired from the second preset queue.
The material increment processing method according to claim 1, wherein the plurality of preset queues further include a fourth preset queue, and in the case that the processing of the material increment instruction fails, the method further comprises:

generating a retry instruction in the fourth preset queue based on the material increment instruction;

According to the retry instruction, retry processing is performed on the material increment instruction through the fourth preset queue.
The material increment processing method according to claim 8, wherein, in the case that the material increment instruction processing fails, the method further comprises:

Based on the material increment instruction, alarm information is generated; wherein, the alarm information is used to prompt a manager that there is an abnormality in the material increment instruction.
The material increment processing method according to any one of claims 1 to 9, wherein the method further comprises:

transmitting the target material information to the target location;

The target material information of the target location is sorted, and material delivery is performed according to the sorting result.
A material incremental processing device, the material incremental processing device includes an acquisition unit, an analysis unit and a processing unit; wherein,

The acquiring unit is configured to acquire a material increment instruction; wherein, the material increment instruction is used to indicate that the material data stored in the source location is changed;

The analyzing unit is configured to perform attribute analysis on the material increment instruction, and determine a target queue corresponding to the material increment instruction in a plurality of preset queues according to the attribute analysis result;

The processing unit is configured to process the material increment instruction by using the target queue to obtain target material information.
An electronic device comprising a memory and a processor; wherein,

the memory for storing a computer program executable on the processor;

The processor is configured to execute the steps of the method according to any one of claims 1 to 10 when running the computer program.
A computer storage medium storing a computer program, the computer program implementing the steps of the method according to any one of claims 1 to 10 when the computer program is executed by a processor.