CN112416982A - Method and device for calculating real-time user characteristics - Google Patents
Method and device for calculating real-time user characteristics Download PDFInfo
- Publication number
- CN112416982A CN112416982A CN202110093431.7A CN202110093431A CN112416982A CN 112416982 A CN112416982 A CN 112416982A CN 202110093431 A CN202110093431 A CN 202110093431A CN 112416982 A CN112416982 A CN 112416982A
- Authority
- CN
- China
- Prior art keywords
- data processing
- data
- platform
- configuration file
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000012545 processing Methods 0.000 claims abstract description 185
- 238000004590 computer program Methods 0.000 claims description 15
- 241001178520 Stomatepia mongo Species 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000013075 data extraction Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 description 33
- 230000008569 process Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000003111 delayed effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The embodiment of the application provides a method and a device for calculating real-time user characteristics, which relate to the technical field of databases, wherein the method for calculating the real-time user characteristics comprises the steps of judging whether a characteristic configuration file exists in a streaming data processing platform or not when a data processing result is detected to be stored in the database; when the feature configuration file exists in the streaming data processing platform, judging whether the feature configuration file corresponds to a data processing result; when the feature configuration file corresponds to the data processing result, loading user data corresponding to the data processing result in the database; and calculating according to the feature configuration file and the user data to obtain the user features. Therefore, by implementing the implementation mode, the user characteristics can be automatically calculated according to the characteristic configuration file while the database is dropped, so that the real-time acquisition of the user characteristics is realized, and the personalized user characteristics are acquired by using the characteristic configuration file.
Description
Technical Field
The application relates to the technical field of databases, in particular to a method and a device for calculating real-time user characteristics.
Background
With the rapid development of society, more and more data information appears in front of people, so that people develop database technology and store the data information by using a database. However, in practice, after the data information is stored in the database, the staff member usually uses the database to calculate the user data corresponding to the data information to obtain the user characteristics of the user. However, this method for acquiring user characteristics does not have real-time acquisition, and cannot achieve personalized user characteristic acquisition effects.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method and an apparatus for calculating a real-time user characteristic, which can automatically perform calculation of a user characteristic according to a characteristic configuration file while dropping a database, thereby implementing real-time acquisition of a user characteristic, and implement personalized user characteristic acquisition by using the characteristic configuration file.
A first aspect of an embodiment of the present application provides a method for calculating a real-time user characteristic, including:
when the data processing result is stored in the database, judging whether a feature configuration file exists in the streaming data processing platform;
when the feature configuration file exists in the streaming data processing platform, judging whether the feature configuration file corresponds to the data processing result;
when the feature configuration file corresponds to the data processing result, loading user data corresponding to the data processing result in the database;
and calculating according to the feature configuration file and the user data to obtain the user features.
In the implementation process, the method can detect whether the service data is processed and put in storage in real time, and immediately acquire the corresponding configuration file and the user data to calculate the user characteristics when detecting that the service data is processed and put in storage. Therefore, by implementing the implementation mode, the user-defined calculation of the user characteristics can be realized through the specific configuration file, so that more accurate user characteristics can be obtained through calculation; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
Further, before the step of determining whether a feature profile exists in the streaming data processing platform when it is detected that the data processing result is stored in the database, the method further includes:
acquiring service data stored in a data hub platform and a kafka platform which are included in a streaming data processing platform;
performing data processing on the service data to obtain a data processing result;
and storing the data processing result into a mongo database.
In the implementation process, the method can acquire the service data in the datahub platform and the kafka platform in advance, then process the service data through the flink to obtain the corresponding data processing result, and finally store the data processing result into the mongo database. Therefore, by implementing the implementation mode, the business data can be subjected to data processing and database dropping through the flink, so that the business data can be processed on the basis of ensuring the efficiency, and meanwhile, the process of calculating the user characteristics can be triggered to be carried out in real time.
Further, before the step of acquiring the service data in the streaming data processing platform, the method further includes:
the business data stored in the mysql database are input into a datahub platform included in a streaming data processing platform through a data transmission service, and the business data are input into a kafka platform included in the streaming data processing platform through a database change monitoring service.
In the implementation process, the method can transmit the service data in the mysql database to the database platform and the kafka platform in advance, so that the service data can be correspondingly stored in the two platforms, the database platform can more effectively distribute the data and the like, the kafka platform can complete the output of the configuration file and the calculation of the user characteristics in cooperation with flink, and the stability and the accuracy of the user characteristic calculation method are further ensured.
Further, performing data processing on the service data according to a preset processing scheme to obtain a data processing result; the preset processing scheme comprises at least one of a data extraction processing scheme, a data conversion processing scheme, a data loading processing scheme, a null data processing scheme, a garbage data processing scheme and a delay data processing scheme.
In the implementation process, the method can perform ETL processing on the service data through the flink, and a result obtained by the processing is used as a data processing result and is stored in the mongo database together with the service data. Therefore, by implementing the embodiment, the database dropping processing can be performed after the ETL processing is performed on the service data, so that the quality of the database dropping data is higher.
Further, when it is detected that the data processing result is stored in the database, the step of determining whether the feature configuration file exists in the streaming data processing platform includes:
when the data processing result is stored in the database, judging whether an sql feature configuration file exists in a kafka platform included in the streaming data processing platform;
and when the sql feature configuration file exists in the kafka platform, triggering and executing the step of judging whether the feature configuration file corresponds to the data processing result.
In the implementation process, the method can acquire the sql-like configuration file, so that subsequent user characteristics can be calculated according to the sql-like configuration file, personalized user characteristic calculation is achieved, and user characteristics with higher quality and higher pertinence can be acquired.
A second aspect of the embodiments of the present application provides a device for computing real-time user characteristics, where the device for computing real-time user characteristics includes:
the first judging unit is used for judging whether a characteristic configuration file exists in the streaming data processing platform or not when the data processing result is stored in the database;
a second judging unit, configured to, when the feature configuration file exists in the streaming data processing platform, judge whether the feature configuration file corresponds to the data processing result;
a loading unit, configured to load, when the feature configuration file corresponds to the data processing result, user data corresponding to the data processing result in the database;
and the calculating unit is used for calculating according to the feature configuration file and the user data to obtain the user features.
In the implementation process, the device can realize the user-defined calculation of the user characteristics through a specific configuration file, so that more accurate user characteristics can be calculated; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
Further, the computing device further comprises:
the acquisition unit is used for acquiring the service data stored in the datahub platform and the kafka platform included in the streaming data processing platform;
the processing unit is used for carrying out data processing on the service data to obtain a data processing result;
and the storage unit is used for storing the data processing result to the mongo database.
In the implementation process, the device can perform data processing and database dropping on the service data through the flink, thereby completing the processing of the service data on the basis of ensuring the efficiency, and simultaneously triggering the process of user characteristic calculation to be performed in real time.
Further, the storage unit is further configured to input the service data stored in the mysql database into a datahub platform included in the streaming data processing platform through the data transmission service, and input the service data into a kafka platform included in the streaming data processing platform through the database change monitoring service.
In the implementation process, the device can transmit the service data in the mysql database to the datahub platform and the kafka platform in advance, so that the service data can be correspondingly stored in the two platforms, the datahub platform can more effectively distribute the data and the like, the kafka platform can complete the output of the configuration file and the calculation of the user characteristics in cooperation with flink, and the stability and the accuracy of the user characteristic calculation method are further ensured.
A third aspect of embodiments of the present application provides an electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the method for calculating the real-time user characteristics according to any one of the first aspect of embodiments of the present application.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores computer program instructions, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the method for calculating the real-time user characteristics according to any one of the first aspect of the embodiments of the present application.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flowchart of a method for calculating a real-time user characteristic according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of another method for calculating a real-time user characteristic according to an embodiment of the present disclosure;
FIG. 3 is a schematic structural diagram of a computing device for real-time user features according to an embodiment of the present application;
FIG. 4 is a schematic structural diagram of another computing device for real-time user characterization according to an embodiment of the present application;
fig. 5 is a system flow diagram of a method for calculating a real-time user characteristic according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example 1
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a method for calculating a real-time user characteristic according to an embodiment of the present disclosure. The method for calculating the real-time user characteristics comprises the following steps:
s101, when the data processing result is stored in a database, judging whether a feature configuration file exists in a streaming data processing platform, if so, executing a step S102; if not, the flow is ended.
In this embodiment, the method may trigger a real-time calculation process of the user characteristics according to a message generated when the service data is stored in the database.
In this embodiment, the method may execute a configuration file written before the computer program is started, where the configuration file includes a program entry class, a message source, a database instance where the original data is located, a feature configuration file source, and the like.
In this embodiment, the feature profile may contain the event message table name, the table-resident repository, the dependency table information, the feature calculation logic, and to which kafka to send.
S102, judging whether the feature configuration file corresponds to a data processing result, if so, executing steps S103-S104; if not, the flow is ended.
In this embodiment, the data processing result corresponds to the service data handled by the user.
And S103, loading user data corresponding to the data processing result in the database.
In this embodiment, the user data is data of a user handling the service data.
And S104, calculating according to the feature configuration file and the user data to obtain the user features.
In this embodiment, the user characteristics are used to represent the omnidirectional characteristics of the user.
In this embodiment, the method may find a corresponding feature configuration file in real time according to the data processing result to perform real-time feature calculation of the corresponding logic, and store the user features obtained by calculation in the database.
In the embodiment, the method has the characteristics of better expansibility, usability and the like. Regarding the expansibility, the method can realize the pluggable property of different message queues and data storage in a mode of rapidly developing or directly accessing a third-party plug-in through a system used by the method; regarding availability, the method and the system can realize automatic restart of failure caused by server exception through an open source flink computing framework used by the method and the system, thereby ensuring high availability of system tasks; regarding the usability, the method and the device can realize the rapid development of the user characteristics in a configuration mode, complete the characteristic development, the test and the online within a few minutes, and further realize that the user can also carry out the user characteristic development, the test and the online operation under the condition that the user does not know the program development and only knows the business logic.
In the embodiment of the present application, the execution subject of the method may be a computing device such as a computer and a server, and is not limited in this embodiment.
In this embodiment, an execution subject of the method may also be an intelligent device such as a smart phone and a tablet computer, which is not limited in this embodiment.
It can be seen that, by implementing the method for calculating the real-time user characteristics described in this embodiment, it can be detected whether the service data is processed and stored in a storage in real time, and when it is detected that the service data is processed and stored in a storage, the corresponding configuration file and the user data are immediately acquired to calculate the user characteristics. Therefore, by implementing the implementation mode, the user-defined calculation of the user characteristics can be realized through the specific configuration file, so that more accurate user characteristics can be obtained through calculation; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
Example 2
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a method for calculating a real-time user characteristic according to an embodiment of the present application. As shown in fig. 2, the method for calculating the real-time user characteristics includes:
s201, inputting the business data stored in the mysql database into a dahub platform included in the streaming data processing platform through the data transmission service, and inputting the business data into a kafka platform included in the streaming data processing platform through the database change monitoring service.
In this embodiment, the method may write the service data into the datahub platform in a DTS manner, or may write the service data into the kafka platform by canal.
S202, acquiring service data stored in a datahub platform and a kafka platform included in the streaming data processing platform.
And S203, performing data processing on the service data to obtain a data processing result.
In this embodiment, the method may perform ETL processing on the service data, and store the processed data processing result in mongo and send a message backward. The ETL includes conversion of data format, null processing, garbage data processing, and delayed data processing.
As an optional implementation, the step of performing data processing on the service data to obtain a data processing result includes:
performing data processing on the service data according to a preset processing scheme to obtain a data processing result; the preset processing scheme comprises at least one of a data extraction processing scheme, a data conversion processing scheme, a data loading processing scheme, a null data processing scheme, a garbage data processing scheme and a delay data processing scheme.
In this embodiment, the data extraction processing scheme is used to refer to a data processing scheme for extracting data from a database.
In this embodiment, the data conversion processing scheme is used to refer to a data processing scheme that converts data from one representation form to another representation form.
In this embodiment, the data loading processing scheme is used to refer to a data processing scheme for storing the converted data in the database.
In this embodiment, the null data processing scheme is used to refer to a data processing scheme for deleting a record containing a missing value or a data processing scheme for interpolating a missing value.
In this embodiment, the garbage data processing scheme is used to refer to a data processing scheme for deleting or skipping garbage data.
In this embodiment, the delayed data processing scheme is used to refer to a data processing scheme that performs hysteresis or delay processing on data.
And S204, storing the data processing result into the mongo database.
In this embodiment, the method can preferentially complete processing of the service data. The processing procedure may include synchronization, processing, and library dropping of the service data.
In this embodiment, the service data is processed by ETL and dropped into a library and then sent to a message for triggering real-time feature calculation, so that the user features can be calculated immediately after an event occurs.
S205, when the data processing result is stored in the database, judging whether the kafka platform included in the streaming data processing platform has an sql feature configuration file, if so, executing the step S206; if not, the flow is ended.
In this embodiment, since there is a certain logic (i.e., user requirement logic) required for calculating the features of the user, these implementation logics have already been configured in the method, and an sql-configured feature configuration file is obtained. The principle is that the user requirements are converted into sql statements, and then converted into feature configuration files according to the logic of sql.
S206, judging whether the feature configuration file corresponds to the data processing result, if so, executing the steps S207-S208; if not, the flow is ended.
In this embodiment, if the feature configuration file does not correspond to the data processing result, the feature configuration file corresponding to the data processing result is obtained, and the subsequent steps S207 to S208 are triggered.
And S207, loading user data corresponding to the data processing result in the database.
And S208, calculating according to the feature configuration file and the user data to obtain the user features.
In this embodiment, the flash (Apache flash) is an open source stream processing framework developed by the Apache software foundation, and the core of the framework is a distributed stream data stream engine written in Java and Scala.
In this embodiment, mysql is a relational database.
In this embodiment, canal may be used to parse the database incremental log and also to provide incremental data subscription and consumption.
In this embodiment, DTS (Data Transmission Service, DTS for short) refers to a Data Transmission Service.
In this embodiment, the DataHub platform is a processing platform for Streaming Data (Streaming Data), and provides functions of publishing (Publish), subscribing (Subscribe), and distributing Streaming Data.
In this embodiment, the Kafka proof is an open source stream processing platform written in Scala and Java.
In this embodiment, ETL (Extract-Transform-Load) is used to describe the process of extracting (Extract), converting (Transform), and loading (Load) data from the source end to the destination end.
In this embodiment, Config is used to refer to a feature profile.
In this embodiment, Mongo is used to refer to the Mongo database.
Referring to fig. 5, fig. 5 is a system flow diagram of the method, and for the explanation of the system flow diagram, reference may be made to embodiment 1 or embodiment 2.
For example, the method may obtain the service data in mysql through dts and canal, and then obtain the service data through the datahub platform and the kafka platform, respectively, so that the flink passes through the datahub platform and the kafka platform to the service data. And when the flink acquires the service data, carrying out simple ETL processing on the service data, and enabling the processing result to fall into the monogo database, so that the flink directly acquires the sql-like feature configuration file in the kafka platform according to the time of falling into the database, and simultaneously extracting the user data in the monogo database, so that the flink can calculate the user features according to the feature configuration file and the user data.
For example, the user is wangwu, and wangwu transacts a business, and the transacted business generates corresponding data to perform database storage. Meanwhile, the flink can detect that service data handled by the king five are in a database, so that the sql-like feature configuration file corresponding to the king five and the user information of the king five are called immediately, and the user features of the king five are calculated through the flink according to the feature configuration file of the king five and the user information of the king five.
Therefore, by implementing the method for calculating the real-time user characteristics described in the embodiment, the user-defined calculation of the user characteristics can be realized through a specific configuration file, so that more accurate user characteristics can be calculated; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
Example 3
Referring to fig. 3, fig. 3 is a schematic structural diagram of a computing device for real-time user features according to an embodiment of the present application. As shown in fig. 3, the real-time user feature calculating device includes:
a first judging unit 310, configured to, when it is detected that the data processing result is stored in the database, judge whether a feature configuration file exists in the streaming data processing platform;
a second judging unit 320, configured to, when a feature configuration file exists in the streaming data processing platform, judge whether the feature configuration file corresponds to a data processing result;
a loading unit 330, configured to load, when the feature configuration file corresponds to the data processing result, user data corresponding to the data processing result in the database;
the calculating unit 340 is configured to perform calculation according to the feature configuration file and the user data to obtain the user feature.
In the embodiment of the present application, for explanation of a computing device of real-time user characteristics, reference may be made to the description in embodiment 1 or embodiment 2, and details are not repeated in this embodiment.
Therefore, the computing device for implementing the real-time user characteristics described in this embodiment can implement user-defined computation of user characteristics through a specific configuration file, so as to obtain more accurate user characteristics through computation; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
Example 4
Referring to fig. 4, fig. 4 is a schematic structural diagram of a computing device for real-time user features according to an embodiment of the present disclosure. The real-time user feature calculation device shown in fig. 4 is optimized by the real-time user feature calculation device shown in fig. 3. As shown in fig. 4, the computing device further includes:
an obtaining unit 350, configured to obtain service data stored in a datahub platform and a kafka platform included in a streaming data processing platform;
the processing unit 360 is configured to perform data processing on the service data to obtain a data processing result;
and the storage unit 370 is used for storing the data processing result into the mongo database.
As an optional implementation manner, the storage unit 370 is further configured to input the service data stored in the mysql database into the datahub platform included in the streaming data processing platform through the data transmission service, and input the service data into the kafka platform included in the streaming data processing platform through the database change monitoring service.
As an optional implementation manner, the processing unit 360 is specifically configured to perform data extraction processing, data conversion processing, data loading processing, null processing, garbage data processing, and delayed data processing on the service data, so as to obtain a data processing result.
As an optional implementation manner, the first determining unit 310 is specifically configured to determine, when it is detected that the data processing result is stored in the database, whether an sql feature configuration file exists in a kafka platform included in the streaming data processing platform; and when the sql-based feature profile exists in the kafka platform, the second judging unit 320 is triggered to execute the operation of judging whether the feature profile corresponds to the data processing result.
In the embodiment of the present application, for explanation of a computing device of real-time user characteristics, reference may be made to the description in embodiment 1 or embodiment 2, and details are not repeated in this embodiment.
Therefore, the computing device for implementing the real-time user characteristics described in this embodiment can implement user-defined computation of user characteristics through a specific configuration file, so as to obtain more accurate user characteristics through computation; meanwhile, the effect of real-time calculation of the user characteristics can be realized in a data drop database real-time calculation mode.
An embodiment of the present application provides an electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a method for calculating a real-time user characteristic in any one of embodiment 1 or embodiment 2 of the present application.
The embodiment of the present application provides a computer-readable storage medium, which stores computer program instructions, and when the computer program instructions are read and executed by a processor, the computer program instructions execute the method for calculating the real-time user characteristics according to any one of embodiment 1 or embodiment 2 of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for computing real-time user characteristics, the method comprising:
when the data processing result is stored in the database, judging whether a feature configuration file exists in the streaming data processing platform;
when the feature configuration file exists in the streaming data processing platform, judging whether the feature configuration file corresponds to the data processing result;
when the feature configuration file corresponds to the data processing result, loading user data corresponding to the data processing result in the database;
and calculating according to the feature configuration file and the user data to obtain the user features.
2. The method of claim 1, wherein prior to the step of determining whether the feature profile exists in the streaming data processing platform when the data processing result is stored in the database, the method further comprises:
acquiring service data stored in a data hub platform and a kafka platform which are included in a streaming data processing platform;
performing data processing on the service data to obtain a data processing result;
and storing the data processing result into a mongo database.
3. The method for calculating the real-time user characteristics according to claim 2, wherein the step of acquiring the service data in the streaming data processing platform is preceded by the step of:
the business data stored in the mysql database are input into a datahub platform included in a streaming data processing platform through a data transmission service, and the business data are input into a kafka platform included in the streaming data processing platform through a database change monitoring service.
4. The method for calculating the real-time user characteristics according to claim 2, wherein the step of performing data processing on the service data to obtain a data processing result comprises:
performing data processing on the service data according to a preset processing scheme to obtain a data processing result; the preset processing scheme comprises at least one of a data extraction processing scheme, a data conversion processing scheme, a data loading processing scheme, a null data processing scheme, a garbage data processing scheme and a delay data processing scheme.
5. The method for calculating the real-time user characteristics according to claim 1, wherein the step of determining whether the characteristic configuration file exists in the streaming data processing platform when the data processing result is stored in the database comprises:
when the data processing result is stored in the database, judging whether an sql feature configuration file exists in a kafka platform included in the streaming data processing platform;
and when the sql feature configuration file exists in the kafka platform, triggering and executing the step of judging whether the feature configuration file corresponds to the data processing result.
6. A real-time user feature computing device, the real-time user feature computing device comprising:
the first judging unit is used for judging whether a characteristic configuration file exists in the streaming data processing platform or not when the data processing result is stored in the database;
a second judging unit, configured to, when the feature configuration file exists in the streaming data processing platform, judge whether the feature configuration file corresponds to the data processing result;
a loading unit, configured to load, when the feature configuration file corresponds to the data processing result, user data corresponding to the data processing result in the database;
and the calculating unit is used for calculating according to the feature configuration file and the user data to obtain the user features.
7. The real-time user profile computing device of claim 6, further comprising:
the acquisition unit is used for acquiring the service data stored in the datahub platform and the kafka platform included in the streaming data processing platform;
the processing unit is used for carrying out data processing on the service data to obtain a data processing result;
and the storage unit is used for storing the data processing result to the mongo database.
8. The real-time user profile computing apparatus according to claim 7, wherein the storage unit is further configured to input the business data stored in the mysql database into a datahub platform included in the streaming data processing platform through a data transmission service, and input the business data into a kafka platform included in the streaming data processing platform through a database change monitoring service.
9. An electronic device, characterized in that the electronic device comprises a memory for storing a computer program and a processor for executing the computer program to cause the electronic device to perform the method of calculating the real-time user characteristics of any one of claims 1 to 5.
10. A readable storage medium having stored thereon computer program instructions which, when read and executed by a processor, perform the method of calculating real-time user characteristics of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110093431.7A CN112416982B (en) | 2021-01-25 | 2021-01-25 | Method and device for calculating real-time user characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110093431.7A CN112416982B (en) | 2021-01-25 | 2021-01-25 | Method and device for calculating real-time user characteristics |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112416982A true CN112416982A (en) | 2021-02-26 |
| CN112416982B CN112416982B (en) | 2021-09-21 |
Family
ID=74782817
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110093431.7A Active CN112416982B (en) | 2021-01-25 | 2021-01-25 | Method and device for calculating real-time user characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112416982B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102609460B (en) * | 2012-01-13 | 2015-02-04 | 中国科学院计算技术研究所 | Method and system for microblog data acquisition |
| US20150235143A1 (en) * | 2003-12-30 | 2015-08-20 | Kantrack Llc | Transfer Learning For Predictive Model Development |
| CN106372969A (en) * | 2016-09-06 | 2017-02-01 | 国家电网公司 | Power user feature identification method and system |
| CN106951552A (en) * | 2017-03-27 | 2017-07-14 | 重庆邮电大学 | A kind of user behavior data processing method based on Hadoop |
| CN107844634A (en) * | 2017-09-30 | 2018-03-27 | 平安科技（深圳）有限公司 | Polynary universal model platform modeling method, electronic equipment and computer-readable recording medium |
| CN109492422A (en) * | 2018-09-04 | 2019-03-19 | 航天信息股份有限公司 | A kind of data processing method and system based on user behavior information |
| CN111242318A (en) * | 2020-01-13 | 2020-06-05 | 拉扎斯网络科技（上海）有限公司 | Business model training method and device based on heterogeneous feature library |
| CN111382150A (en) * | 2020-03-19 | 2020-07-07 | 交通银行股份有限公司 | Real-time computing method and system based on Flink |
| CN111382347A (en) * | 2018-12-28 | 2020-07-07 | 广州市百果园信息技术有限公司 | Object feature processing and information pushing method, device and equipment |
-
2021
- 2021-01-25 CN CN202110093431.7A patent/CN112416982B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150235143A1 (en) * | 2003-12-30 | 2015-08-20 | Kantrack Llc | Transfer Learning For Predictive Model Development |
| CN102609460B (en) * | 2012-01-13 | 2015-02-04 | 中国科学院计算技术研究所 | Method and system for microblog data acquisition |
| CN106372969A (en) * | 2016-09-06 | 2017-02-01 | 国家电网公司 | Power user feature identification method and system |
| CN106951552A (en) * | 2017-03-27 | 2017-07-14 | 重庆邮电大学 | A kind of user behavior data processing method based on Hadoop |
| CN107844634A (en) * | 2017-09-30 | 2018-03-27 | 平安科技（深圳）有限公司 | Polynary universal model platform modeling method, electronic equipment and computer-readable recording medium |
| CN109492422A (en) * | 2018-09-04 | 2019-03-19 | 航天信息股份有限公司 | A kind of data processing method and system based on user behavior information |
| CN111382347A (en) * | 2018-12-28 | 2020-07-07 | 广州市百果园信息技术有限公司 | Object feature processing and information pushing method, device and equipment |
| CN111242318A (en) * | 2020-01-13 | 2020-06-05 | 拉扎斯网络科技（上海）有限公司 | Business model training method and device based on heterogeneous feature library |
| CN111382150A (en) * | 2020-03-19 | 2020-07-07 | 交通银行股份有限公司 | Real-time computing method and system based on Flink |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112416982B (en) | 2021-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106656536B (en) | Method and equipment for processing service calling information | |
| CN106202235B (en) | Data processing method and device | |
| US20180365085A1 (en) | Method and apparatus for monitoring client applications | |
| US8856313B2 (en) | Systems and methods for using provenance information for data retention in stream-processing | |
| CN108256870B (en) | Method and device for generating description information, updating and processing data based on topological structure | |
| CN109918678B (en) | Method and device for identifying field meaning | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN110399268A (en) | A kind of method, device and equipment of anomaly data detection | |
| CN111708760B (en) | Model migration deployment method and device, electronic equipment and storage medium | |
| CN111400170A (en) | Data permission testing method and device | |
| CN109359109B (en) | Data processing method and system based on distributed stream computing | |
| CN109271453B (en) | Method and device for determining database capacity | |
| CN107016028B (en) | Data processing method and apparatus thereof | |
| CN112416982B (en) | Method and device for calculating real-time user characteristics | |
| CN111552703B (en) | Data processing method and device | |
| CN111427871B (en) | Data processing method, device and equipment | |
| CN110955547B (en) | Management method, system, equipment and readable storage medium for interface abnormal information | |
| CN110955760B (en) | Evaluation method of judgment result and related device | |
| CN108234196B (en) | Fault detection method and device | |
| US20160364813A1 (en) | System for creating a linkage family tree including non-requested entities and detecting changes to the tree via an intelligent change detection system | |
| CN114338494B (en) | Service dependency topological relation obtaining method and device, storage medium and electronic equipment | |
| CN111625853B (en) | Snapshot processing method, device and equipment and readable storage medium | |
| CN114020571A (en) | Monitoring method and monitoring equipment for index server | |
| CN116418858A (en) | Service release method and device | |
| CN116719710A (en) | Data processing method, device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: Room 716, 7 / F, building 2, 28 Andingmen East Street, Dongcheng District, Beijing Patentee after: Beijing Easy Yikang Information Technology Co.,Ltd. Address before: Room 716, 7 / F, building 2, 28 Andingmen East Street, Dongcheng District, Beijing Patentee before: BEIJING QINGSONGCHOU INFORMATION TECHNOLOGY Co.,Ltd. |