CN110908883A - User portrait data monitoring method, system, equipment and storage medium - Google Patents
User portrait data monitoring method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN110908883A CN110908883A CN201911120466.4A CN201911120466A CN110908883A CN 110908883 A CN110908883 A CN 110908883A CN 201911120466 A CN201911120466 A CN 201911120466A CN 110908883 A CN110908883 A CN 110908883A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- message queue
- storing
- user representation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention provides a method, a system, equipment and a storage medium for monitoring user portrait data, wherein the method comprises the following steps: collecting user portrait data from each node in a user portrait data link; storing the user representation data in a message queue; adopting a data flow calculation engine to carry out real-time statistics on the user image data in the message queue; and storing the statistical result of the data flow calculation engine in a statistical database. By adopting the scheme of the invention, the user portrait data link is monitored in a full link mode, and the data statistical efficiency is improved by combining the message queue and the data flow calculation engine, so that the data of each node in the user portrait data link can be captured and analyzed in time, and the problem can be found in time.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a user portrait data monitoring method, a user portrait data monitoring system, user portrait data monitoring equipment and a storage medium.
Background
Big data is of increasing importance to companies, and the security of data assets is also being addressed frequently within companies that enable various departments of business. After the calculation of the big data department, the data of the user portrait system is pushed to each business system for use, and is specifically applied to use scenes such as operation, advertisement, report forms and the like, and the downstream business system puts higher requirements on the upstream data quality. If the data lack necessary monitoring in the circulation process of each system, the data quality of the key nodes cannot be managed and controlled.
In the existing monitoring system, a downstream service system provides data exception checking on service logic when using data, such as data duplication judgment, data quantity exception judgment, data inaccuracy judgment and the like, but this is only judgment established on specific service logic, and is applicable to a service a but not necessarily applicable to B service, and does not have a unified standard. In addition, in the existing system, if a problem occurs in the data in the circulation process, the test or code review is required, or necessary information is marked in the code, the information is output to a local file in the form of an output log, the log is browsed when the problem occurs, and a possible problem point is searched according to the log; the time and labor costs of these methods are high, and when data exception checking is coupled in a service system, the performance of the service is affected, and each time data is used, the development cost is increased.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide a method, a system, a device and a storage medium for monitoring user portrait data, which are combined with a message queue and a data stream calculation engine to improve data statistics efficiency and perform full link monitoring on a user portrait data link.
The embodiment of the invention provides a method for monitoring user image data, which comprises the following steps:
collecting user portrait data from each node in a user portrait data link;
storing the user representation data in a message queue;
adopting a data flow calculation engine to carry out real-time statistics on the user image data in the message queue;
and storing the statistical result of the data flow calculation engine in a statistical database.
Optionally, after storing the statistical result of the data stream computing engine, the method further includes the following steps:
processing the statistical result in the statistical database by adopting a preset abnormal detection rule;
and if the statistic result is detected to be abnormal, triggering an alarm system.
Optionally, the user portrait data includes a portrait data ID, a user tag ID, a dotting event type, and dotting event record data;
the user portrait data in the message queue is counted, wherein the counting comprises one or more of counting transmission data amount of each transmission node of the user portrait data, counting data processing amount of each user label, counting data processing amount of each user and counting data processing amount of each dotting event type.
Optionally, after storing the user representation data in a message queue, the method further includes the following steps:
storing user representation data in the message queue in a distributed database.
Optionally, the message queue is a Kafka message queue, and the data stream calculation engine is a Flink data stream calculation engine or a Spark data stream calculation engine;
storing user representation data in the message queue in a distributed database includes storing user representation data in the message queue in an HDFS system and updating a Hive table of the HDFS system.
Optionally, storing the user representation data in the message queue in a distributed database includes storing the user representation data in the message queue in a distributed database using a GAIA component.
Optionally, the user representation data link includes a user representation data generation node, a user representation data distribution node, and a user representation data usage node.
Optionally, after storing the user representation data in the message queue in the distributed database, the method further includes the following steps:
obtaining user portrait data from the distributed database;
performing a consistency check on user representation data having the same representation data ID obtained from different nodes in the user representation data link;
if there is an inconsistency, the image data ID where the inconsistency has occurred and the node where the inconsistency has occurred are determined.
The embodiment of the invention also provides a user portrait data monitoring system, which is applied to the user portrait data monitoring method, and the system comprises:
the data acquisition module is used for acquiring user portrait data from each node of a user portrait data link and storing the user portrait data in a message queue;
the data processing module is used for carrying out real-time statistics on the user image data in the message queue by adopting a data stream calculation engine;
and the result storage module is used for storing the statistical result of the data stream calculation engine in a statistical database.
The embodiment of the invention also provides a user portrait data monitoring device, which comprises:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the user representation data monitoring method via execution of the executable instructions.
An embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program is executed to implement the steps of the user portrait data monitoring method.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
The user portrait data monitoring method, the system, the equipment and the storage medium provided by the invention have the following advantages:
the invention solves the problems in the prior art, monitors the user portrait data link in a full link manner, and improves the data statistical efficiency by combining the message queue and the data flow calculation engine, thereby timely capturing and analyzing the data of each node in the user portrait data link, finding out the problems in time, being suitable for data monitoring of each service system in the user portrait system, and not influencing the normal work of the existing service system.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a flow chart of a method for monitoring user image data in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating data collection in a user representation data monitoring method according to an embodiment of the invention;
FIG. 3 is a schematic diagram illustrating anomaly detection in a user profile data monitoring method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a user representation data monitoring system according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a user representation data monitoring system with added alerts, in accordance with an embodiment of the present invention;
FIG. 6 is a schematic view of a user representation data monitoring apparatus in accordance with an embodiment of the present invention;
fig. 7 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
As shown in fig. 1, an embodiment of the present invention provides a method for monitoring user image data, where the method includes the following steps:
s100: collecting user portrait data from each node in a user portrait data link;
s200: storing the user representation data in a message queue; in this embodiment, the message queue may be a Kafka message queue, where Kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java, and is a high throughput distributed publish-subscribe messaging system that can process all the flow data of a consumer's actions in a web site; the Kafka message queue is used as an example for illustration, but the invention is not limited to this, and in other alternative embodiments, the message queue may adopt other types of queues, and all fall within the protection scope of the invention;
s300: adopting a data flow calculation engine to carry out real-time statistics on the user image data in the Kafka message queue; in this embodiment, the data stream computation engine is a Flink data stream computation engine, wherein the Flink data stream computation engine is an open source stream processing framework developed by Apache software foundation, and the core of the Flink data stream computation engine is a distributed stream data stream engine written in Java and Scala. The Flink data stream computing engine executes any stream data program in a data parallel and pipeline mode, and a system can execute batch processing and stream processing programs when a pipeline of the Flink data stream computing engine runs; the following description takes a Flink data stream calculation engine as an example, but the present invention is not limited thereto, and in other alternative embodiments, the data stream calculation engine may also be other types of calculation engines, such as a Spark data stream calculation engine;
s400: the statistics of the data flow computation engine are stored in a statistics database, for example in MySQL (relational database management system).
In this embodiment, step S100 is performed to acquire full link data of the user portrait data link, generate user portrait data to be sent to each service system, and record key events of necessary paths in a data tracking mode for tracking, alarming, analyzing, debugging, and the like of abnormal data. And through step S200 and step S300, the data in the Kafka message queue is consumed by adopting the data stream computing engine, and the data statistical efficiency is improved by combining the Kafka message queue and the Flink data stream computing engine, so that the data of each node in the user portrait data link can be captured and analyzed in time, after the statistical result is stored through step S400, the problem can be found in time through the analysis of the statistical result, and the consistency, reliability and real-time performance of the user portrait data in the user portrait system are improved.
FIG. 2 is a schematic diagram of data acquisition in a user portrait data monitoring method according to an embodiment of the present invention. In this embodiment, the user representation data link includes a user representation data generation node, a user representation data distribution node, and a user representation data usage node. The user portrait data use node is mainly data used by each business system, including advertisement, operation and the like. The user portrait data generation node may be further specifically divided into a user portrait data calculation node and a user tag addition node. Each node is buried, and user portrait data is collected in a Kafka message queue by sdk (software development kit).
In this embodiment, the user representation data includes a representation data ID, a user tag ID, a dotting event type, and dotting event record data. The image data ID is a unique identifier of the image data, is unchangeable when the image data ID is circulated among all nodes in a full data link, and can be used for data consistency check in subsequent off-line processing. The dotting event record data may include dotting event behavior, dotting role, dotting description, dotting time point, and further, the user representation data may further include information such as extension field.
In step S300, the Flink dataflow computation engine is used to perform statistics on the user portrait data in the Kafka message queue, including one or more of statistics on transmission data amount of each transmission node of the user portrait data, statistics on data processing amount of each user tag, statistics on data processing amount of each user, and statistics on data processing amount of each dotting event type. For example, the statistics may include the amount of computation and data for each tag of the user representation system, the amount of data transferred at various stages during the transfer process, etc. by the current time. Different statistical tasks can be set in the Flink data flow calculation engine, and the statistical tasks are started to carry out data statistics.
FIG. 3 is a schematic diagram illustrating an anomaly detection method in a user image data monitoring method according to an embodiment of the present invention. In this embodiment, the step S400: after storing the statistical result of the data stream calculation engine, the method further includes S500: the steps of anomaly detection and warning specifically include: processing the statistical result in the statistical database by adopting a preset abnormal detection rule; and if the statistic result is detected to be abnormal, triggering an alarm system.
For example, in the short message marketing campaign, the abnormal detection rule is set in advance to detect whether the short message delivery amount exceeds a preset delivery amount threshold value, the delivered buried point event information is consumed in real time by using a Flink engine in the delivery process, and when the conditions such as overdischarge occur, an alarm system can be triggered, so that unnecessary loss is effectively avoided through manual intervention.
Further, in this embodiment, the alarm threshold in the abnormal alarm system may also be configured, for example, configured to configure a delivered data volume threshold, a delivered delay threshold, etc., and compared with data counted in real time by the Flink data stream calculation engine, if the threshold is exceeded, the alarm information is sent to the relevant staff in the form of short message, telephone, etc.
In this embodiment, the step S200: after storing the user portrait data in the Kafka message queue, the method further includes S600: storing user portrait data offline; specifically, the method comprises the following steps: storing user representation data in the Kafka message queue in a distributed database.
In this embodiment, the distributed database may be an HDFS system, and after storing the user image data in an HDFS (Hadoop distributed file system) system, a Hive table of the HDFS system is updated. Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading, and is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. However, the present invention is not limited to this, and in other embodiments, other types of distributed databases may be selected for offline storage of user portrait data.
In this embodiment, storing the user representation data in the Kafka message queue in a distributed database includes storing the user representation data in the Kafka message queue in a distributed database using a GAIA component.
In this embodiment, the step S600: after storing the user representation data in the Kafka message queue in the distributed database, the method further includes S700: specifically, step S700 includes the following steps:
s710: obtaining user portrait data from the distributed database;
s720: performing a consistency check on user representation data having the same representation data ID obtained from different nodes in the user representation data link;
s730: if there is an inconsistency, the image data ID where the inconsistency has occurred and the node where the inconsistency has occurred are determined.
In this embodiment, the Hive table stores data of each node in calculation and transmission of the user representation system, and specific fields include: portrait data ID, portrait system tag ID, dotting event type, dotting role, event behavior, user ID, dotting description, dotting time point, extension field, for data consistency and reliability verification of each stage; if data are missing, data comparison of each node can be carried out in the Hive table, and abnormal data and abnormal processing nodes are checked. After the image data ID where the inconsistency has occurred and the node where the inconsistency has occurred are determined, the worker may be notified through an alarm system.
As shown in fig. 4, an embodiment of the present invention further provides a user portrait data monitoring system, which is applied to the user portrait data monitoring method, and the system includes:
a data acquisition module M100, configured to acquire user portrait data from each node of a user portrait data link and store the user portrait data in a message queue, in this embodiment, the user portrait data may be acquired through sdk and stored in a Kafka message queue;
a data processing module M200, configured to perform real-time statistics on the user portrait data in the Kafka message queue by using a data stream calculation engine, in this embodiment, the data stream calculation engine may be a Flink data stream calculation engine or a Spark data stream calculation engine, where Spark is a fast and general calculation engine designed specifically for large-scale data processing;
and a result storage module M300, configured to store the statistical result of the data flow calculation engine in a statistical database, for example, in MySQL.
In this embodiment, the data acquisition module M100 performs full link data acquisition on the user portrait data link, generates user portrait data to be sent to each service system, and records the key events of the necessary paths in a data tracking mode for tracking, alarming, analyzing, debugging and the like of abnormal data. And data in the message queue is consumed by the data flow computing engine through the data processing module M200, and the data statistical efficiency is improved by combining the message queue and the data flow computing engine, so that data of each node in the user portrait data link can be captured and analyzed in time, after the statistical result is stored through the result storage module M300, problems can be found in time through analysis of the statistical result, and the consistency, reliability and real-time performance of user portrait data in the user portrait system are improved.
The data processing module M200 may count the transmission data amount of each transmission node of the user portrait data, count the data processing amount of each user tag, count the data processing amount of each user, count the data processing amount of each dotting event type, and the like, when the data stream calculation engine is used to count the user portrait data in the message queue.
As shown in fig. 5, in this embodiment, the user portrait data monitoring system may further include an anomaly alarm module M400, configured to process the statistical result in the statistical database by using a preset anomaly detection rule; and if the statistical result is detected to be abnormal, triggering an abnormal alarm system, and alarming by adopting the modes of nails, short messages, telephones, mails and the like.
Further, the user portrait data monitoring system of the present invention may further include an alarm configuration module M500, configured to set an alarm threshold in the abnormality detection rule of the abnormality alarm module M400, and a worker may directly configure the alarm threshold on a web page, for example, configure a delivered data amount threshold, a delivered delay threshold, and the like.
In this embodiment, the data processing module M200 is further configured to store the user portrait data in the Kafka message queue in a distributed database, and obtain offline user portrait data from the distributed database; performing a consistency check on user representation data having the same representation data ID obtained from different nodes in the user representation data link; if there is an inconsistency, the image data ID where the inconsistency has occurred and the node where the inconsistency has occurred are determined.
The embodiment of the invention also provides user portrait data monitoring equipment, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the user representation data monitoring method via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the electronic device 600 is embodied in the form of a general purpose computing device. The combination of the electronic device 600 may include, but is not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting different platform combinations (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code that can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
An embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program is executed to implement the steps of the user portrait data monitoring method. In some possible embodiments, aspects of the present invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of this specification, when the program product is run on the terminal device.
Referring to fig. 7, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, compared with the prior art, the method, system, device and storage medium for monitoring user portrait data provided by the present invention have the following advantages:
the invention solves the problems in the prior art, monitors the user portrait data link in a full link manner, and improves the data statistical efficiency by combining the message queue and the data flow calculation engine, thereby timely capturing and analyzing the data of each node in the user portrait data link, finding out the problems in time, being suitable for data monitoring of each service system in the user portrait system, and not influencing the normal work of the existing service system.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (11)
1. A user portrait data monitoring method is characterized by comprising the following steps:
collecting user portrait data from each node in a user portrait data link;
storing the user representation data in a message queue;
adopting a data flow calculation engine to carry out real-time statistics on the user image data in the message queue;
and storing the statistical result of the data flow calculation engine in a statistical database.
2. The method of monitoring user portrait data of claim 1, further comprising, after storing statistics of the data flow calculation engine, the steps of:
processing the statistical result in the statistical database by adopting a preset abnormal detection rule;
and if the statistic result is detected to be abnormal, triggering an alarm system.
3. A user representation data monitoring method as claimed in claim 1 wherein said user representation data includes a representation data ID, a user tag ID, a dotting event type and a dotting event record data;
the user portrait data in the message queue is counted, wherein the counting comprises one or more of counting transmission data amount of each transmission node of the user portrait data, counting data processing amount of each user label, counting data processing amount of each user and counting data processing amount of each dotting event type.
4. The method of monitoring user representation data as recited in claim 1, further comprising, after storing said user representation data in a message queue, the steps of:
storing user representation data in the message queue in a distributed database.
5. The user representation data monitoring method of claim 4, wherein the message queue is a Kafka message queue and the data stream calculation engine is a Flink data stream calculation engine or a Spark data stream calculation engine;
storing user representation data in the message queue in a distributed database includes storing user representation data in the message queue in an HDFS system and updating a Hive table of the HDFS system.
6. The method of user representation data monitoring of claim 4, wherein said storing user representation data in said message queue in a distributed database includes using a GAIA component to store user representation data in said message queue in a distributed database.
7. The user representation data monitoring method of claim 6, wherein the user representation data link includes a user representation data calculation node, a user tag addition node, a user representation data distribution node, and a user representation data usage node.
8. The method of monitoring user representation data of claim 1, wherein after storing said user representation data in said message queue in a distributed database, further comprising the steps of:
obtaining user portrait data from the distributed database;
performing a consistency check on user representation data having the same representation data ID obtained from different nodes in the user representation data link;
if there is an inconsistency, the image data ID where the inconsistency has occurred and the node where the inconsistency has occurred are determined.
9. A user representation data monitoring system, for use in a user representation data monitoring method as claimed in any one of claims 1 to 8, the system comprising:
the data acquisition module is used for acquiring user portrait data from each node of a user portrait data link and storing the user portrait data in a message queue;
the data processing module is used for carrying out real-time statistics on the user image data in the message queue by adopting a data stream calculation engine;
and the result storage module is used for storing the statistical result of the data stream calculation engine in a statistical database.
10. A user portrait data monitoring apparatus, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the user representation data monitoring method of any of claims 1-8 via execution of the executable instructions.
11. A computer readable storage medium storing a program, wherein the program is executed to implement the steps of the user representation data monitoring method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911120466.4A CN110908883B (en) | 2019-11-15 | 2019-11-15 | User portrait data monitoring method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911120466.4A CN110908883B (en) | 2019-11-15 | 2019-11-15 | User portrait data monitoring method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110908883A true CN110908883A (en) | 2020-03-24 |
| CN110908883B CN110908883B (en) | 2022-08-30 |
Family
ID=69817460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911120466.4A Active CN110908883B (en) | 2019-11-15 | 2019-11-15 | User portrait data monitoring method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110908883B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111640436A (en) * | 2020-05-15 | 2020-09-08 | 北京青牛技术股份有限公司 | Method for providing a dynamic customer representation of a call partner to an agent |
| CN111652658A (en) * | 2020-06-11 | 2020-09-11 | 北京妙医佳健康科技集团有限公司 | Portrait fusion method, apparatus, electronic device and computer readable storage medium |
| CN111741360A (en) * | 2020-06-19 | 2020-10-02 | 深圳市酷开网络科技有限公司 | Image application method and device based on open-source column type database and storage medium |
| CN111930748A (en) * | 2020-08-07 | 2020-11-13 | 北京百度网讯科技有限公司 | Data tracking method, device, equipment and storage medium for streaming computing system |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112100220A (en) * | 2020-09-22 | 2020-12-18 | 福建天晴在线互动科技有限公司 | System for realizing real-time monitoring of illegal account group |
| CN113220530A (en) * | 2021-05-14 | 2021-08-06 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and platform |
| CN113297620A (en) * | 2021-06-08 | 2021-08-24 | 哈尔滨无限力创网络科技有限公司 | Big data security processing system based on user portrait |
| CN113553320A (en) * | 2021-07-29 | 2021-10-26 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106502772A (en) * | 2016-10-09 | 2017-03-15 | 国网浙江省电力公司信息通信分公司 | Electric quantity data batch high speed processing method and system based on distributed off-line technology |
| CN109284195A (en) * | 2018-08-27 | 2019-01-29 | 广东电网有限责任公司信息中心 | A kind of real-time representation data calculation method and system |
-
2019
- 2019-11-15 CN CN201911120466.4A patent/CN110908883B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106502772A (en) * | 2016-10-09 | 2017-03-15 | 国网浙江省电力公司信息通信分公司 | Electric quantity data batch high speed processing method and system based on distributed off-line technology |
| CN109284195A (en) * | 2018-08-27 | 2019-01-29 | 广东电网有限责任公司信息中心 | A kind of real-time representation data calculation method and system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111640436A (en) * | 2020-05-15 | 2020-09-08 | 北京青牛技术股份有限公司 | Method for providing a dynamic customer representation of a call partner to an agent |
| CN111640436B (en) * | 2020-05-15 | 2024-04-19 | 北京青牛技术股份有限公司 | Method for providing dynamic customer portraits of conversation objects to agents |
| CN111652658A (en) * | 2020-06-11 | 2020-09-11 | 北京妙医佳健康科技集团有限公司 | Portrait fusion method, apparatus, electronic device and computer readable storage medium |
| CN111741360A (en) * | 2020-06-19 | 2020-10-02 | 深圳市酷开网络科技有限公司 | Image application method and device based on open-source column type database and storage medium |
| CN111930748B (en) * | 2020-08-07 | 2023-08-08 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for tracking data of streaming computing system |
| CN111930748A (en) * | 2020-08-07 | 2020-11-13 | 北京百度网讯科技有限公司 | Data tracking method, device, equipment and storage medium for streaming computing system |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112100220A (en) * | 2020-09-22 | 2020-12-18 | 福建天晴在线互动科技有限公司 | System for realizing real-time monitoring of illegal account group |
| CN113220530A (en) * | 2021-05-14 | 2021-08-06 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and platform |
| CN113220530B (en) * | 2021-05-14 | 2022-07-19 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and platform |
| CN113297620A (en) * | 2021-06-08 | 2021-08-24 | 哈尔滨无限力创网络科技有限公司 | Big data security processing system based on user portrait |
| CN113553320B (en) * | 2021-07-29 | 2022-09-02 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and device |
| CN113553320A (en) * | 2021-07-29 | 2021-10-26 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110908883B (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908883B (en) | User portrait data monitoring method, system, equipment and storage medium | |
| CN109960635B (en) | Monitoring and alarming method, system, equipment and storage medium of real-time computing platform | |
| US10489711B1 (en) | Method and apparatus for predictive behavioral analytics for IT operations | |
| CN112631913B (en) | Method, device, equipment and storage medium for monitoring operation faults of application program | |
| US9535981B2 (en) | Systems and methods for filtering low utility value messages from system logs | |
| CN110704873B (en) | Method and system for preventing sensitive data from being leaked | |
| CN111274094B (en) | Interface early warning method, system, equipment and storage medium | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN112000806A (en) | Abnormal log monitoring and analyzing method, system, equipment and storage medium | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| CN111245672A (en) | Monitoring method and system for general extensible tracking service full link | |
| CN114548706A (en) | Early warning method for business risk and related equipment | |
| US20220138032A1 (en) | Analysis of deep-level cause of fault of storage management | |
| CN110912757A (en) | Service monitoring method and server | |
| CN113760666A (en) | System exception processing method, device and storage medium | |
| CN112087320A (en) | Abnormity positioning method and device, electronic equipment and readable storage medium | |
| CN113760677A (en) | Abnormal link analysis method, device, equipment and storage medium | |
| CN110011845B (en) | Log collection method and system | |
| US11816210B2 (en) | Risk-based alerting for computer security | |
| CN110851316A (en) | Abnormity early warning method, abnormity early warning device, abnormity early warning system, electronic equipment and storage medium | |
| CN114546780A (en) | Data monitoring method, device, equipment, system and storage medium | |
| CN113342619A (en) | Log monitoring method and system, electronic device and readable medium | |
| CN111835566A (en) | System fault management method, device and system | |
| CN112015623A (en) | Method, device and equipment for processing report data and readable storage medium | |
| CN112825165A (en) | Project quality management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |