CN113743975A - Advertisement effect processing method and device - Google Patents
Advertisement effect processing method and device Download PDFInfo
- Publication number
- CN113743975A CN113743975A CN202110126551.2A CN202110126551A CN113743975A CN 113743975 A CN113743975 A CN 113743975A CN 202110126551 A CN202110126551 A CN 202110126551A CN 113743975 A CN113743975 A CN 113743975A
- Authority
- CN
- China
- Prior art keywords
- user association
- hash
- click log
- advertisement
- association key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000694 effects Effects 0.000 title claims abstract description 41
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 48
- 238000005192 partition Methods 0.000 claims description 29
- 238000004590 computer program Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 5
- 238000009825 accumulation Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 22
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 235000014510 cooky Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0242—Determining effectiveness of advertisements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Abstract
The invention discloses an advertisement effect processing method and device, and relates to the technical field of computers. One embodiment of the method comprises: collecting order messages from a message queue, and determining user association keys in the order messages; searching an advertisement click log corresponding to the user association key through the indexing characteristic of the ClickHouse; and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser. According to the implementation mode, the indexing characteristic of the ClickHouse is applied to the advertisement effect system, the corresponding advertisement click log can be obtained based on jda of the order information, the whole system does not need to occupy memory resources, the index calculation amount is small, and the performance of logic calculation is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to an advertisement effect processing method and device.
Background
An advertisement effect system based on HDFS (Hadoop Distributed filesystem) as a storage system means that offline advertisement click logs of N days are stored in the HDFS, and when an order is tracked, the advertisement click logs of the full N days are loaded to a memory and order association architecture system, and then logical operation is performed; the advertisement click log includes a cookie of the user and relevant information of the advertiser, and is usually obtained by clicking an advertisement delivered by the advertiser by the user.
In the process of implementing the invention, the inventor finds that the prior art has at least the following problems: HDFS does not support indexing functionality; when the order is tracked, the clicks in the click logs of the total N days are really related to the order of the batch by less than one ten thousandth, and more irrelevant advertisement click logs exist, so that the existing loading mode of the total advertisement click logs can cause waste of memory resources, and the calculation performance is poor due to overlarge data volume.
Disclosure of Invention
In view of this, embodiments of the present invention provide an advertisement effect processing method and apparatus, which can at least solve the problems in the prior art that, because HDFS does not support an indexing function, all advertisement click logs in a system need to be loaded into a memory when an order is tracked, which results in waste of memory resources and poor computing performance.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided an advertisement effect processing method including:
collecting order messages from a message queue, and determining user association keys in the order messages;
searching an advertisement click log corresponding to the user association key through the indexing characteristic of the ClickHouse;
and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
Optionally, before the collecting the order message from the message queue, the method further includes:
and accessing the advertisement click log stream into the clickwouse in real time through an engine of the clickwouse, and taking a hash user association key in the advertisement click log as an index identifier.
Optionally, the searching, through an index characteristic of clickwouse, an advertisement click log corresponding to the user association key includes:
performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
and determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition.
Optionally, the performing hash processing on the user association key by using a non-encryption type hash function includes:
and judging the effectiveness of the user association key by using an effective function, if the user association key is not inquired in a blacklist, performing hash processing on the user association key by using a non-encryption type hash function, and if the user association key is not inquired in the blacklist, determining that the user association key is invalid, and filtering the user association key without processing.
Optionally, the searching for the advertisement click log corresponding to the user associated key in the partition includes:
counting the number of hash user association keys in the same partition;
if the size of the preset batch is reached, generating a query statement so as to query an advertisement click log corresponding to the hash user association key in the click House;
and if not, continuously counting the number of the hash user association keys until the preset batch size is reached.
Optionally, after attributing the order message to the advertiser who released the advertisement click log, the method further includes:
generating and storing a tracking result of the order information; and
and acquiring a processing link where the order information is currently located, and updating the processing state of the order information based on the processing link.
To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided an advertisement effect processing apparatus including:
the collecting module is used for collecting order messages from the message queue and determining user association keys in the order messages;
the searching module is used for searching the advertisement click log corresponding to the user associated key through the indexing characteristic of the ClickHouse;
and the attribution module is used for attributing the order message to an advertiser who puts the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
Optionally, the system further includes an access module, configured to: and accessing the advertisement click log stream into the clickwouse in real time through an engine of the clickwouse, and taking a hash user association key in the advertisement click log as an index identifier.
Optionally, the search module is configured to: performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
and determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition.
Optionally, the search module is configured to: and judging the effectiveness of the user association key by using an effective function, if the user association key is not inquired in a blacklist, performing hash processing on the user association key by using a non-encryption type hash function, and if the user association key is not inquired in the blacklist, determining that the user association key is invalid, and filtering the user association key without processing.
Optionally, the search module is configured to: counting the number of hash user association keys in the same partition;
if the size of the preset batch is reached, generating a query statement so as to query an advertisement click log corresponding to the hash user association key in the click House;
and if not, continuously counting the number of the hash user association keys until the preset batch size is reached.
Optionally, the system further includes a status updating module, configured to:
generating and storing a tracking result of the order information; and
and acquiring a processing link where the order information is currently located, and updating the processing state of the order information based on the processing link.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an advertisement effect processing electronic device.
The electronic device of the embodiment of the invention comprises: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the advertisement effectiveness processing methods described above.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements any of the advertisement effectiveness processing methods described above.
According to the scheme provided by the invention, one embodiment of the invention has the following advantages or beneficial effects: the click House with the index characteristic is used as a storage system for storing the advertisement click log for N days, so that the problems of memory resource waste and influence on the performance of the logic calculation module are solved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic flow chart of a method for processing advertisement effectiveness according to an embodiment of the present invention;
FIG. 2 is a flow diagram illustrating an alternative advertisement effectiveness processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main modules of an advertisement effectiveness processing apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
FIG. 5 is a schematic block diagram of a computer system suitable for use with a mobile device or server implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Referring to fig. 1, a main flowchart of an advertisement effectiveness processing method according to an embodiment of the present invention is shown, which includes the following steps:
s101: collecting order messages from a message queue, and determining user association keys in the order messages;
s102: searching an advertisement click log corresponding to the user association key through the indexing characteristic of the ClickHouse;
s103: and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
In the above embodiment, for steps S101 and S102, clickwouse is a columnar database management system for online analytical processing (OLAP). The scheme takes ClickHouse with index characteristic as a storage system for storing advertisement click logs for N days. Reference to an advertiser is a legal person, other economic organization, or individual who designs, produces, or distributes advertisements to promote goods or provide services, either by himself or by trust with others. It is an important participant in the market economy and advertising activities, and the subject qualification is that the self organization forms are closely related, and can be legal people or natural people.
The advertisement click log stream is accessed from kafka to clickwouse in real-time by the clickwouse's kafka (open source streaming platform). In addition, the Click House can also store offline advertisement Click log streams, and the part of advertisement Click log streams need to be accessed in an offline manner, so that N days of total advertisement Click logs are stored in the Click House. Typically, the default time limit is 15 days, and since the order placement time begins, the order tracking time searches 15 days of advertisement click logs from the order placement time onward.
Order messages are collected from the jmq message queue at a predetermined period, for example, 10 minutes of order messages. Using jda (i.e. user association key, such as user id, non-order id) in the order message as input, the click index module of clickhaus e is used to obtain the advertisement click log related to the order message, which is specifically obtained in the following manner as described in fig. 2.
For step S103, order messages collected from the jmq message queue and the obtained advertisement click log are input into the effect tracking module, and preset parameters such as campaign configuration information, advertiser custom configuration information, blacklist offline data and the like are also loaded into the effect tracking module. The effect tracking module performs logical operation of order tracking to determine which order sku should be attributed to which advertiser corresponding to the previously determined advertisement click log, and then outputs the operation result to the HDFS. It should be noted that the storage of the operation result and the storage of the source data may be different, because the reading of the operation result does not need to use an indexing mechanism.
The term "advertisement effect" generally refers to the social influence and effect that can be produced by advertisement information transmitted through an advertisement medium. This effect and effect includes two aspects: the advertisement broadcasting effect and the advertisement selling effect are achieved. The two advertisement effects are measured, so that enterprises can effectively make advertisement strategies, the advertisement cost is reduced, and the advertisement benefit is improved.
The advertising effect is divided into a narrow sense and a broad sense. The narrow advertising effect refers to the economic benefit obtained by the advertisement, i.e., the degree to which the advertisement propagates to promote the sale of the product, i.e., the sales effect brought by the advertisement. The generalized advertisement effect refers to the achievement degree of the purpose of the advertisement activity, and is the sum of direct or indirect changes caused by the advertisement information in the transmission process, including the economic benefit, the psychological benefit and the social benefit of the advertisement.
Upon obtaining the advertiser to whom the order is attributed, the order's current processing status may be updated, such as one of placing an order, paying, completing, and canceling. Generally, a current processing state is determined according to a current processing link of an order, and the current processing state is stored in jimdb; and asynchronously writing the current processing state into the mysql during storage so as to update the processing state of the order in the mysql. And (5) carrying out follow-up real-time order state message flow to update the order processing state in real time.
According to the method provided by the embodiment, the index characteristic of the clickwouse is applied to the advertisement effect system, the corresponding advertisement click log can be obtained based on jda of the order information, the whole method does not need to occupy memory resources, the index calculation amount is small, and the performance of logic calculation is improved.
Referring to fig. 2, a flow chart of an alternative advertisement effectiveness processing method according to an embodiment of the present invention is shown, which includes the following steps:
s201: accessing the advertisement click log stream into a click H out in real time through a click House engine, and taking a Hash user association key in the advertisement click log as an index mark;
s202: collecting order messages from a message queue, and determining user association keys in the order messages;
s203: using an effective function to judge the effectiveness of the user association key;
s204: if the query in the blacklist does not exist, performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
s205: otherwise, determining that the user association key is invalid, and filtering without processing;
s206: determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition according to the index characteristic of ClickHouse;
s207: and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
In the above embodiment, for the steps S202 and S207, reference may be made to the description shown in fig. 1, and details are not repeated here.
In the above embodiment, regarding steps S201, S203 to S206, the present embodiment mainly describes how to obtain an advertisement click log from clickwouse, and the specific implementation logic is as follows:
1. when receiving the advertisement click log stream from kafka, the ClickHouse directly obtains jda after hash from the advertisement click log stream and uses the jda as the index identification of the advertisement click log stream;
2. collect 10 minutes (time is merely an example) of order messages from the jmq message queue, with the 10 minutes of order messages as input;
3. before jda in the order message is processed by the hash (or after it is actually processed), the validity function jda is used to judge the validity of jda, and if the validity is invalid when the validity is in the set packet of the blacklist jda, the validity can be directly filtered to not participate in the query of the advertisement click log;
4. if the query does not exist, carrying out hash processing on jda fields in the order message by using a MurmurHash function to obtain a new field hash jda; the MurmurHash is a non-encryption type Hash function, is suitable for general Hash retrieval operation, is used for converting a character string into a 64-bit Hash value, and has the main effects of saving the storage space of data and indexes on one hand and reducing the index search range during query on the other hand.
5. If the number is jda, sorting the hashed jda in partitions, wherein the number of the partitions is partition _ nums at present, and the value can be adjusted along with specific service characteristics, namely the number of concurrency of subsequent queries. Since the ClickHouse is stored based on index sorting, in order to make good use of this sequential property, the hash jda is also sorted in partitions before querying, so as to reduce the search range of the data in the ClickHouse. And the data of the same hash jda are divided into the same partition, and the data in the same partition are sorted.
In addition, the hash jda in the same partition is queried for a batch, and the hash jda of a certain batch is queried for one batch (batch). If the preset batch size is not reached, repeatedly collecting the hash jda step, if the preset batch size is not reached, generating an sql statement, and inquiring an advertisement click log responding to the order message in a ClickHouse; where BatchSize is the batch size, and is typically used in batch operations on databases, to improve performance, for example: the BatchSize is 1000, which is 1000 pieces of data processed by each database interaction.
In the method provided by the embodiment, the click log of the advertisement is written into the click log from kafka in real time by using the click log as a storage system, and an index is established by using jda after hash in the click log; when the follow-up order tracking and the advertising effect are accumulated, jda in the order information is directly used, and the relevant advertisement click log can be obtained from the Clickhouse.
The embodiment of the invention applies the indexing characteristic of the ClickHouse to the advertisement effect system, so that jda based on order information can obtain the corresponding advertisement click log; by using the processed hash jda as an index, the size of the index data can be reduced; and (3) putting the hash jda of all the advertisement click logs into different partitions, performing subsequent batch processing in the same partition, and well utilizing the sequential storage characteristic of the Clickhouse data to reduce query IO and improve query efficiency.
Referring to fig. 3, a schematic diagram of main modules of an advertisement effectiveness processing apparatus 300 according to an embodiment of the present invention is shown, including:
a collecting module 301, configured to collect order messages from a message queue, and determine a user association key in the order messages;
the searching module 302 is configured to search, through an index characteristic of clickwouse, an advertisement click log corresponding to the user associated key;
an attributing module 303, configured to attribute the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log, and a preset parameter, so as to accumulate putting effects of the advertiser.
The apparatus for implementing the present invention further includes an access module 304 (not shown in the figure), configured to:
and accessing the advertisement click log stream into the clickwouse in real time through an engine of the clickwouse, and taking a hash user association key in the advertisement click log as an index identifier.
In the device implemented by the present invention, the search module 302 is configured to:
performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
and determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition.
In the device implemented by the present invention, the search module 302 is configured to:
and judging the effectiveness of the user association key by using an effective function, if the user association key is not inquired in a blacklist, performing hash processing on the user association key by using a non-encryption type hash function, and if the user association key is not inquired in the blacklist, determining that the user association key is invalid, and filtering the user association key without processing.
In the device implemented by the present invention, the search module 302 is configured to:
counting the number of hash user association keys in the same partition;
if the size of the preset batch is reached, generating a query statement so as to query an advertisement click log corresponding to the hash user association key in the click House;
and if not, continuously counting the number of the hash user association keys until the preset batch size is reached.
The apparatus further includes a status update module 305 (not shown) for:
generating and storing a tracking result of the order information; and
and acquiring a processing link where the order information is currently located, and updating the processing state of the order information based on the processing link.
In addition, the detailed implementation of the device in the embodiment of the present invention has been described in detail in the above method, so that the repeated description is not repeated here.
FIG. 4 illustrates an exemplary system architecture 400 to which embodiments of the invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (by way of example only). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 401, 402, 403.
The terminal devices 401, 402, and 403 may be various electronic devices having display screens and supporting web browsing, and the server 405 may be a server that provides various services.
It should be noted that the method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a collection module, a lookup module, and an attribution module. Where the names of these modules do not in some way constitute a limitation on the modules themselves, for example, an attribution module may also be described as an "attribution and accumulation module".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
collecting order messages from a message queue, and determining user association keys in the order messages;
searching an advertisement click log corresponding to the user association key through the indexing characteristic of the ClickHouse;
and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
According to the technical scheme of the embodiment of the invention, the corresponding advertisement click log can be obtained based on jda of the order information; by using the processed hash jda as an index, the size of the index data can be reduced; and (3) putting the hash jda of all the advertisement click logs into different partitions, performing subsequent batch processing in the same partition, and well utilizing the sequential storage characteristic of the Clickhouse data to reduce query IO and improve query efficiency.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. An advertisement effect processing method, comprising:
collecting order messages from a message queue, and determining user association keys in the order messages;
searching an advertisement click log corresponding to the user association key through the indexing characteristic of the ClickHouse;
and attributing the order message to an advertiser who puts in the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
2. The method of claim 1, further comprising, prior to said collecting order messages from a message queue:
and accessing the advertisement click log stream into the clickwouse in real time through an engine of the clickwouse, and taking a hash user association key in the advertisement click log as an index identifier.
3. The method of claim 2, wherein the searching for the advertisement click log corresponding to the user associated key through the indexing characteristic of clickwouse comprises:
performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
and determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition.
4. The method of claim 3, wherein hashing the user association key using a non-cryptographic hash function comprises:
and judging the effectiveness of the user association key by using an effective function, if the user association key is not inquired in a blacklist, performing hash processing on the user association key by using a non-encryption type hash function, and if the user association key is not inquired in the blacklist, determining that the user association key is invalid, and filtering the user association key without processing.
5. The method of claim 3, wherein the searching for the advertisement click log corresponding to the user associated key in the partition comprises:
counting the number of hash user association keys in the same partition;
if the size of the preset batch is reached, generating a query statement so as to query an advertisement click log corresponding to the hash user association key in the click House;
and if not, continuously counting the number of the hash user association keys until the preset batch size is reached.
6. The method of any of claims 1-5, further comprising, after said attributing the order message to an advertiser that placed the ad click log:
generating and storing a tracking result of the order information; and
and acquiring a processing link where the order information is currently located, and updating the processing state of the order information based on the processing link.
7. An advertisement effectiveness processing apparatus, comprising:
the collecting module is used for collecting order messages from the message queue and determining user association keys in the order messages;
the searching module is used for searching the advertisement click log corresponding to the user associated key through the indexing characteristic of the ClickHouse;
and the accumulation module is used for attributing the order message to an advertiser who puts the advertisement click log based on the order message, the advertisement click log and preset parameters so as to accumulate the putting effect of the advertiser.
8. The apparatus of claim 7, wherein the lookup module is configured to:
performing hash processing on the user association key by using a non-encryption type hash function to obtain a hash user association key;
and determining a partition corresponding to the Hash user association key, and searching an advertisement click log corresponding to the Hash user association key in the partition.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110126551.2A CN113743975A (en) | 2021-01-29 | 2021-01-29 | Advertisement effect processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110126551.2A CN113743975A (en) | 2021-01-29 | 2021-01-29 | Advertisement effect processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113743975A true CN113743975A (en) | 2021-12-03 |
Family
ID=78728183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110126551.2A Pending CN113743975A (en) | 2021-01-29 | 2021-01-29 | Advertisement effect processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113743975A (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060116989A1 (en) * | 2004-11-30 | 2006-06-01 | Srikanth Bellamkonda | Efficient data aggregation operations using hash tables |
| CN103605756A (en) * | 2013-11-22 | 2014-02-26 | 北京国双科技有限公司 | Data processing method and data processing device for on-line analysis processing |
| WO2014115671A1 (en) * | 2013-01-22 | 2014-07-31 | 株式会社ぐるなび | Advertising main site server, advertising main site server managing method, and advertising main site managing program |
| CN105468749A (en) * | 2015-11-26 | 2016-04-06 | 上海晶赞科技发展有限公司 | Data processing method and device |
| CN107330034A (en) * | 2017-06-26 | 2017-11-07 | 百度在线网络技术（北京）有限公司 | A kind of log analysis method and device, computer equipment, storage medium |
| CN108985838A (en) * | 2018-07-10 | 2018-12-11 | 天津璧合信息技术有限公司 | A kind of method and device obtaining ad click crowd phone number |
| CN109150757A (en) * | 2017-06-16 | 2019-01-04 | 腾讯科技（深圳）有限公司 | A kind of advertising resource processing method, device and electronic equipment |
| CN110134683A (en) * | 2019-04-23 | 2019-08-16 | 中国地质大学（武汉） | The partition zone optimizing research method and system that magnanimity element stores in relational database |
| CN110399479A (en) * | 2018-04-20 | 2019-11-01 | 北京京东尚科信息技术有限公司 | Search for data processing method, device, electronic equipment and computer-readable medium |
| CN110866001A (en) * | 2018-08-21 | 2020-03-06 | 北京京东尚科信息技术有限公司 | Method and device for determining order to be processed |
| CN111125521A (en) * | 2019-12-13 | 2020-05-08 | 上海喜马拉雅科技有限公司 | Information recommendation method, device, equipment and storage medium |
| CN111461754A (en) * | 2019-01-18 | 2020-07-28 | 北京京东尚科信息技术有限公司 | Method and device for determining flow source of order |
| CN111488377A (en) * | 2020-04-08 | 2020-08-04 | 北京奇艺世纪科技有限公司 | Data query method and device, electronic equipment and storage medium |
| CN111784380A (en) * | 2020-06-05 | 2020-10-16 | 北京沃东天骏信息技术有限公司 | Advertisement putting attribution method and device |
| CN112085332A (en) * | 2020-08-06 | 2020-12-15 | 微民保险代理有限公司 | Data processing method, data processing device, computer equipment and storage medium |
-
2021
- 2021-01-29 CN CN202110126551.2A patent/CN113743975A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060116989A1 (en) * | 2004-11-30 | 2006-06-01 | Srikanth Bellamkonda | Efficient data aggregation operations using hash tables |
| WO2014115671A1 (en) * | 2013-01-22 | 2014-07-31 | 株式会社ぐるなび | Advertising main site server, advertising main site server managing method, and advertising main site managing program |
| CN103605756A (en) * | 2013-11-22 | 2014-02-26 | 北京国双科技有限公司 | Data processing method and data processing device for on-line analysis processing |
| CN105468749A (en) * | 2015-11-26 | 2016-04-06 | 上海晶赞科技发展有限公司 | Data processing method and device |
| CN109150757A (en) * | 2017-06-16 | 2019-01-04 | 腾讯科技（深圳）有限公司 | A kind of advertising resource processing method, device and electronic equipment |
| CN107330034A (en) * | 2017-06-26 | 2017-11-07 | 百度在线网络技术（北京）有限公司 | A kind of log analysis method and device, computer equipment, storage medium |
| CN110399479A (en) * | 2018-04-20 | 2019-11-01 | 北京京东尚科信息技术有限公司 | Search for data processing method, device, electronic equipment and computer-readable medium |
| CN108985838A (en) * | 2018-07-10 | 2018-12-11 | 天津璧合信息技术有限公司 | A kind of method and device obtaining ad click crowd phone number |
| CN110866001A (en) * | 2018-08-21 | 2020-03-06 | 北京京东尚科信息技术有限公司 | Method and device for determining order to be processed |
| CN111461754A (en) * | 2019-01-18 | 2020-07-28 | 北京京东尚科信息技术有限公司 | Method and device for determining flow source of order |
| CN110134683A (en) * | 2019-04-23 | 2019-08-16 | 中国地质大学（武汉） | The partition zone optimizing research method and system that magnanimity element stores in relational database |
| CN111125521A (en) * | 2019-12-13 | 2020-05-08 | 上海喜马拉雅科技有限公司 | Information recommendation method, device, equipment and storage medium |
| CN111488377A (en) * | 2020-04-08 | 2020-08-04 | 北京奇艺世纪科技有限公司 | Data query method and device, electronic equipment and storage medium |
| CN111784380A (en) * | 2020-06-05 | 2020-10-16 | 北京沃东天骏信息技术有限公司 | Advertisement putting attribution method and device |
| CN112085332A (en) * | 2020-08-06 | 2020-12-15 | 微民保险代理有限公司 | Data processing method, data processing device, computer equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| 郭心语;刘鹏;周敏奇;周傲英;: "网络广告定向技术综述", 华东师范大学学报(自然科学版), no. 03 * |
| 陈磊;刘奕群;茹立云;马少平;: "基于用户日志挖掘的搜索引擎广告效果分析", 中文信息学报, no. 06 * |
| 黄必栋;: "网络威胁情报处理系统的设计与开发", 网络安全技术与应用, no. 08 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10121169B2 (en) | Table level distributed database system for big data storage and query | |
| US10528970B2 (en) | Systems, methods, and devices for pipelined processing of online advertising performance data | |
| Bordin et al. | DSPBench: A suite of benchmark applications for distributed data stream processing systems | |
| US9965772B2 (en) | System and method for unifying user-level data across different media platforms | |
| CN104182506A (en) | Log management method | |
| CN110738477A (en) | reconciliation method, device, computer equipment and storage medium | |
| CN108415964A (en) | Tables of data querying method, device, terminal device and storage medium | |
| CN110019367B (en) | Method and device for counting data characteristics | |
| US11720708B2 (en) | Privacy preserving data collection and analysis | |
| WO2019143370A1 (en) | Method, system, and computer program product for real-time data aggregation | |
| CN109299089A (en) | The calculating and storage method and calculating of a kind of label data of drawing a portrait and storage system | |
| CN108985805B (en) | Method and device for selectively executing push task | |
| CN113360500A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN113190558A (en) | Data processing method and system | |
| CN109544207A (en) | A kind of information processing method, storage medium and server | |
| US10331713B1 (en) | User activity analysis using word clouds | |
| CN113743975A (en) | Advertisement effect processing method and device | |
| CN114357280A (en) | Information pushing method and device, electronic equipment and computer readable medium | |
| CN114817297A (en) | Method and device for processing data | |
| CN110781238B (en) | Client view caching method and device based on combination of Redis and Hbase | |
| CN112817979A (en) | Advertisement delivery information attribution method and system | |
| CN112184370A (en) | Method and device for pushing product | |
| CN113434754A (en) | Method and device for determining recommended API (application program interface) service, electronic equipment and storage medium | |
| CN110020364B (en) | Method and device for determining flow source of page access | |
| CN113535768A (en) | Production monitoring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |