CA3184561A1 - Method, device and computer equipment for improving user portrait interface performance - Google Patents
Method, device and computer equipment for improving user portrait interface performanceInfo
- Publication number
- CA3184561A1 CA3184561A1 CA3184561A CA3184561A CA3184561A1 CA 3184561 A1 CA3184561 A1 CA 3184561A1 CA 3184561 A CA3184561 A CA 3184561A CA 3184561 A CA3184561 A CA 3184561A CA 3184561 A1 CA3184561 A1 CA 3184561A1
- Authority
- CA
- Canada
- Prior art keywords
- data
- machine
- label data
- portrait
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000007781 pre-processing Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 29
- 230000006870 function Effects 0.000 claims description 8
- 230000004931 aggregating effect Effects 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 description 7
- 238000004220 aggregation Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a method, apparatus, computer device, and storage medium for improving the performance of a user portrait interface. The method comprises:
preprocessing label data according to a set task instance; wherein the label data includes poi _________________________ nait label data; obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information;
distributing the conditional query statement randomly to any machine of query nodes; obtaining aggregated data from the machine of query nodes according to the conditional query statement; and returning the aggregated data to a client terminal for displaying. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to further improve the performance of a user portrait interface.
preprocessing label data according to a set task instance; wherein the label data includes poi _________________________ nait label data; obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information;
distributing the conditional query statement randomly to any machine of query nodes; obtaining aggregated data from the machine of query nodes according to the conditional query statement; and returning the aggregated data to a client terminal for displaying. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to further improve the performance of a user portrait interface.
Description
METHOD, DEVICE AND COMPUTER EQUIPMENT FOR IMPROVING USER
PORTRAIT INTERFACE PERFORMANCE
Technical Field [0001] The present disclosure relates to the technical field of user portrait, particularly to a method, apparatus, computer device, and storage medium for improving the performance of a user poi __ lt ait interface.
Background
PORTRAIT INTERFACE PERFORMANCE
Technical Field [0001] The present disclosure relates to the technical field of user portrait, particularly to a method, apparatus, computer device, and storage medium for improving the performance of a user poi __ lt ait interface.
Background
[0002] With the development and progress of the society, building user portraits is getting more important, user portraits can be used to truly reflects the user's behavior trajectory, habit characteristics and service needs and so on in a demonstrate multi-dimensional view of data, and enable various fields to improve the capability of services, and provide necessary technical support for data mining in the data analysis.
[0003] Based on user portraits, the requirement of the performance of a user portrait interface by an operating party also appeared, currently, Elasticsearc as a search engine, or offline tasks tool such as spark as data process tool is applied to achieve the requirement of the performance of the user portrait interface.
[0004] However, as the requirement of the performance of the user portrait interface by clients is increased, the response to user's requests within a time limit is required. And Elasticsearc as the search engine has its own limits, the user portrait interface is often times out when large data is aggregated, therefore, the search engine are not able to meet the requirement by the operating party and have negative user experiences.
Invention Content
Invention Content
[0005] Based on this, it is necessary to address the above technical problems and provide a method, apparatus, computer device and storage medium for improving the performance of a user portrait interface.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0006] On one hand, a method for improving the performance of a user portrait interface, comprises:
[0007] Preprocessing label data according to a set task instance;
wherein the label data includes poi ____ halt label data;
wherein the label data includes poi ____ halt label data;
[0008] Obtaining portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
[0009] Distributing the conditional query statement randomly to any machine of query nodes;
[0010] Obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0011] Returning the aggregated data to a client teitninal for displaying to the user.
[0012] In an embodiment, wherein preprocessing label data according to a set task instance, comprises:
[0013] Grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups;
[0014] According to the plurality of the coded data groups, generating a plurality of files with set formats corresponding to the label data;
[0015] Inserting each file with the set format into a corresponding local engine table;
and Date Regue/Date Received 2023-02-21
and Date Regue/Date Received 2023-02-21
[0016] Obtaining the portrait label data from the local engine table and importing a corresponding correlation engine table; wherein the correlation engine table is on the machine of sharding query nodes.
[0017] In an embodiment, wherein the pre-set rule also comprises:
[0018] Coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and
[0019] Grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
[0020] In an embodiment, wherein, inserting each file with the set format into the corresponding local engine table, comprises:
[0021] Inserting each file with the set format into a corresponding blank local engine table; and
[0022] Inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
[0023] In an embodiment, wherein, distributing the conditional query statement randomly to any machine of query nodes, comprises:
[0024] Distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
[0025] In an embodiment, wherein obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes Date Regue/Date Received 2023-02-21 machines, comprises:
[0026] According to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions.
[0027] Wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
[0028] In an embodiment, wherein method further comprises:
[0029] Associating each corresponding correlation engine table with a distributed table;
wherein the distributed table is on the machine of sharding query nodes.
wherein the distributed table is on the machine of sharding query nodes.
[0030] Wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
[0031] In another aspect, an apparatus for improving the performance of a user portrait interface is provided, wherein the apparatus comprises:
[0032] A data processing module configured to preprocess label data according to a set task instance; wherein the label data includes portrait label data;
[0033] An obtaining module configured to obtain portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
[0034] A random distribution module configured to distribute the conditional query statement randomly to any machine of query nodes;
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0035] An aggregating module configured to obtain aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0036] A data display module configured to return the aggregated data to a client teiiiiinal for displaying to the user.
[0037] On the other hand, a computer device, including a memory, a processor and a computer program stored in the memory and run on the processor configured to achieve following steps when the processor executes the computer program:
[0038] Preprocessing label data according to a set task instance;
wherein the label data includes poi ____ Li ait label data;
wherein the label data includes poi ____ Li ait label data;
[0039] Obtaining portrait label information selected by a user, and generating a conditional query statement according to the portrait label infoimation;
[0040] Distributing the conditional query statement randomly to any machine of query nodes;
[0041] Obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0042] Returning the aggregated data to a client teiiiiinal for displaying to the user.
[0043] In another aspect, a computer readable storage medium stored with a computer program configured to achieve following steps when the processor executes the computer program:
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0044] Preprocessing label data according to a set task instance;
wherein the label data includes portrait label data;
wherein the label data includes portrait label data;
[0045] Obtaining portrait label infounation selected by a user, and generating a conditional query statement according to the portrait label information;
[0046] Distributing the conditional query statement randomly to any machine of query nodes;
[0047] Obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0048] Returning the aggregated data to a client terminal for displaying to the user.
[0049] The above-mentioned method, apparatus, computer device, and storage medium for improving the performance of a user portrait interface. Firstly, preprocessing label data according to a set task instance; wherein the label data includes portrait label data;
Then, obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information; distributing the conditional query statement randomly to any machine of query nodes; Finally, obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of poitiait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying. Each machine of sharding query nodes needs the portrait label data that meet the conditional query statement only, and to perform the data aggregation, then to return the aggregation results to the client teiminal. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to further improve the performance of a user portrait interface.
Date Regue/Date Received 2023-02-21 Drawing Description
Then, obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information; distributing the conditional query statement randomly to any machine of query nodes; Finally, obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of poitiait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying. Each machine of sharding query nodes needs the portrait label data that meet the conditional query statement only, and to perform the data aggregation, then to return the aggregation results to the client teiminal. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to further improve the performance of a user portrait interface.
Date Regue/Date Received 2023-02-21 Drawing Description
[0050] Figure 1 is an application environment diagram of a method for improving the performance of a user portrait interface in an embodiment;
[0051] Figure 2 is a flowchart of the method for improving the performance of a user portrait interface in an embodiment;
[0052] Figure 3 is a process diagram of Step 202 of a method for improving the performance of a user portrait interface in an embodiment;
[0053] Figure 4 is a structural diagram of an apparatus for improving the performance of a user portrait interface in an embodiment;
[0054] Figure 5 is an internal structural diagram of a computer device in an embodiment.
Specific embodiment methods
Specific embodiment methods
[0055] In order to make clearer application purposes, technical solutions, and advantages, the present disclosure is further explained in detail with a particular embodiment thereof, and with reference to the drawings. It shall be appreciated that these descriptions are only intended to be illustrative, but not to limit the scope of the disclosure thereto.
[0056] The present application provides a method for improving the performance of a user portrait interface and applies the method in an application environment as shown in Figure 1. In Figure 1, wherein, a client terminal 102 communicates with a server 104 through network. The server 104 preprocesses label data according to a set task instance; wherein the label data includes portrait label data;
Then, obtains portrait label information selected by a user, generates a conditional query statement according to the portrait label information; distributes randomly the Date Regue/Date Received 2023-02-21 conditional query statement to any machine of query nodes; the server 104 obtains aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returns the aggregated data to the client terminal 102 for displaying. Wherein, the client terminal 102 can be but not limited to various personal computer, laptop, smart phone, tablet computer, portable wearable device or sub-server, server 201 can be an independent server or a server cluster composed of a plurality of servers or cloud computing platform to achieve.
Then, obtains portrait label information selected by a user, generates a conditional query statement according to the portrait label information; distributes randomly the Date Regue/Date Received 2023-02-21 conditional query statement to any machine of query nodes; the server 104 obtains aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returns the aggregated data to the client terminal 102 for displaying. Wherein, the client terminal 102 can be but not limited to various personal computer, laptop, smart phone, tablet computer, portable wearable device or sub-server, server 201 can be an independent server or a server cluster composed of a plurality of servers or cloud computing platform to achieve.
[0057] In an embodiment, as shown in Figure 2, a method for improving the performance of a user portrait interface is provided, as an example of the client terminal in Figure 1, comprising following steps:
[0058] Step 202, preprocessing label data according to a set task instance;
wherein the label data includes portrait label data.
wherein the label data includes portrait label data.
[0059] Wherein the task instance is set by ClickHouse system (a column-oriented SQL database management system (DBMS) for online analytical processing (OLAP)), the task instance refers to a Spark (computing engine) task; the label data refers to data described characteristics of an operating entity. The characteristics of the operating entity is described by a plurality of labels for reflecting the operating entity in multi-dimensions; wherein the label data includes portrait label data and other label data, wherein the portrait label data is portrait field data.
[0060] Specifically, according to the set Spark task, preprocessing the label data, wherein the label data preprocess is a process for storing the label data and storing the label data by category in various engine tables in ClickHouse system, it is convenient to call and obtain in a later stage.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0061] Step 204, obtaining portrait label information selected by a user, and generating a conditional query statement according to the portrait label info rmati on.
[0062] Specifically, the portrait label information selected by the user is information of portrait interface, the structured query statement is generated according to the information of portrait interface selected by the suer, then the structured query statement is transferred to the conditional query statement, wherein the conditional query statement is a statement for querying data in the column-oriented database management system directly.
[0063] Step 206, distributing the conditional query statement randomly to any machine of query nodes.
[0064] Specifically, distributing the conditional query statement randomly to any machine of query nodes. Wherein the machines of query nodes are a cluster, a plurality of query nodes machines are set in the cluster, that is, a group of mutually independent computers interconnected through a high-speed network, which form a group and are managed in a single system mode. The configuration of the cluster is used for improving availability and scalability. It is uncertain to which machine of query nodes is randomly distributed the conditional query statement.
[0065] Step 208, obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines.
[0066] Specifically, obtaining aggregated data from the machine of query nodes according to the conditional query statement. When the conditional query Date Regue/Date Received 2023-02-21 statement has been randomly distributed any machine of query nodes, the machine of query nodes distributes the conditional query statement to a plurality of machines of sharding query nodes, each machine of sharding query nodes performs a computing process based on the portrait label data stored on its own machine only and return the result to the machine of query nodes. The machine of query nodes receives results from various machines of sharding query nodes and aggregates the results to obtain the aggregated data. Therefore, the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines.
[0067] Step 210, returning the aggregated data to a client terminal for displaying to the user.
[0068] Specifically, returning the aggregated data to a client terminal for displaying to the user, it is convenient to check out the results of the aggregated data responding to the information of portrait label information selected by the user.
[0069] The above-mentioned method for improving the performance of a user portrait interface. Firstly, preprocessing label data according to a set task instance;
wherein the label data includes portrait label data; Then, obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information; distributing the conditional query statement randomly to any machine of query nodes; Finally, obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying. Each machine of sharding query nodes needs the portrait label data that meet the conditional query statement only, and to perform the data aggregation, then to return the aggregation results to the client terminal. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to Date Regue/Date Received 2023-02-21 further improve the performance of a user portrait interface.
wherein the label data includes portrait label data; Then, obtaining portrait label information selected by a user, generating a conditional query statement according to the portrait label information; distributing the conditional query statement randomly to any machine of query nodes; Finally, obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying. Each machine of sharding query nodes needs the portrait label data that meet the conditional query statement only, and to perform the data aggregation, then to return the aggregation results to the client terminal. The above method can fully use the capability of parallel computing in a distributed system and improve the computing speed to Date Regue/Date Received 2023-02-21 further improve the performance of a user portrait interface.
[0070] In an embodiment, as shown in Figure 3, wherein preprocessing label data according to a set task instance, further comprises the following steps:
[0071] Step 2022, grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups.
[0072] Wherein the pre-set rule refers to coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data.
[0073] Specifically, in a manner of a modulo sharding, the rule is to divide the coded data into N parts, wherein the N is the number of query nodes plus 1, in other words, the number of the machines of query nodes plus one, so that the coded data are uniformly distributed on each machine of sharding query nodes. The coded data is obtained by a remainder for a specific value for determining the coded data should be arranged in which coded data group, the coded data with the same remainder are arranged in a group, and the plurality of coded data groups is obtained. For example, the label data is sorted from minimum to maximum to be 1-m, the number of the machines of sharding query nodes plus one is N, that is, the number of the coded data group is N groups, prior to the coded data grouped into each coded data group, 1-m of the coded data should have a remainder after division by value K, the coded data having the same reminder is grouped together, then the plurality of coded data groups are obtained.
[0074] Step 2024, according to the plurality of the coded data groups, generating a plurality of files with set formats corresponding to the label data.
[0075] Specifically, according to the plurality of the coded data groups, generating a plurality of files with set formats corresponding to the label data, each coded Date Regue/Date Received 2023-02-21 data group generates a file with a set format, wherein the set format is HDFS
(Hadoop Distributed File System) file.
(Hadoop Distributed File System) file.
[0076] Step 2026, inserting each file with the set format into a corresponding local engine table.
[0077] Specifically, as a plurality of files with the set formats are generated, each file with the set format is inserted into the corresponding local engine table, wherein the local engine table is a built-in engine table in ClickHouse.
[0078] Step 2028, obtaining the portrait label data from the local engine table, and importing a corresponding correlation engine table; wherein the correlation engine table is on the machine of sharding query nodes.
[0079] Specifically, obtaining the portrait label data from the local engine table, wherein the files with the set formats stored in each local engine table includes both portrait label data and other label data. The portrait label data is classified as hot data, which refers to online data accessed by the computing nodes frequently, the hot data is often used to be computed nearby since the access of the hot data in a great need and required a highly efficiency. Other label data is classified as cold data, wherein the cold data refers to offline data accessed rarely, and the cold data is stored centrally, has a lower access frequency and a lower efficiency, therefore the cold data can be configured centrally. The hot data is the portrait label data stored in each local engine table in the present application, and the portrait label data obtained from each local engine table is imported into the corresponding correlation engine table. Wherein the correlation engine table is on the machine of sharding query nodes, each correlation engine table in each machine of sharding query nodes stores the corresponding portrait label information, it is convenient to obtain in a later stage, and reduce loading time for data.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0080] The above-mentioned manner of sharding for storing, which divides the label data into the plurality of coded data groups according to a preset rule, then respectively store the plurality of coded data groups into various machines of sharding query nodes, to reduce the access pressure of each machine of sharding query nodes and to improve the performance of the entire database system.
[0081] In an embodiment, wherein, inserting each file with the set format into the corresponding local engine table, further comprises the following steps:
[0082] Inserting each file with the set format into a corresponding blank local engine table; and
[0083] Inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
[0084] Wherein the blank local engine table is a built-in engine table in ClickHouse.
[0085] Specifically, inserting each file with the set founat into a corresponding blank local engine table; and inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view. The materialized view is a database object including a query result, which is a local copy of the remote data, or which is used to generating a summary table based on a data table summation. The storage of the materialized view is based on data of a remote table, also named as snapshots.
This method enable each file with the set format in the bland local engine table to be inserted into the corresponding local engine table, to increase the storing speed.
This method enable each file with the set format in the bland local engine table to be inserted into the corresponding local engine table, to increase the storing speed.
[0086] In an embodiment, wherein, distributing the conditional query statement randomly to any machine of query nodes, further comprises the following steps:
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0087] Distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
[0088] Wherein the distribution of VIP refers to the tasks assigned to multiple operating units in a balance manner. As we know, the general IP address is bound with a physical network interface card while the VIP is not bound with a physical network interface. When a computer in an outer network access to an internal network of a company via a domain, a DNS (The Domain Name System) server in the internal network resolves the domain name to a VIP. When the computer in the outer network obtains the VIP and sends the data packet to the VIP. But the VIP is not connected with any physical devices in the internal network, which is implemented via ARP (Address Resolution Protocol). In other words, the MAC
address (Media Access Control) mapped in the VIP is controllable. The VIP can be dynamically mapped in different MAC addresses in the internal network, which means to be mapped in different devices for load balancing.
address (Media Access Control) mapped in the VIP is controllable. The VIP can be dynamically mapped in different MAC addresses in the internal network, which means to be mapped in different devices for load balancing.
[0089] Specifically, by applying the step of distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes, each machine of query nodes can be balanced, to balance the workload of each machine of query nodes.
[0090] In an embodiment, wherein obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines, further comprises the following steps:
[0091] According to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtaining aggregated data of the Date Regue/Date Received 2023-02-21 portrait label data from each machine of sharding query nodes by calculating bitmap functions.
[0092] Wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
[0093] Specifically, according to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions. Each machine of sharding query nodes performs aggregation computing based on the portrait label data stored on its own machine only via bitmap functions, to obtain the aggregated data of portrait label data, and the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes to improve the entire query speed.
[0094] In an embodiment, wherein method further comprises the following steps:
[0095] Associating each corresponding correlation engine table with a distributed table; wherein the distributed table is on the machine of sharding query nodes.
[0096] Wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
[0097] Wherein the distributed table is a built-in engine table in ClickHouse.
[0098] Specifically, associating each corresponding correlation engine table with a distributed table; wherein the distributed table is on the machine of sharding query nodes.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0099] When the aggregation operation is executed by the machine of query nodes, the aggregated data of portrait label data associated with each correlation engine table is aggregated on the distributed table, to obtain aggregated data and achieve the aggregation operation.
[0100] What should be understood is although the steps of the process diagram of Figures 2-3 are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly provided instruction in this article, there is no strict order in which these steps can be performed, and they can be performed in any other orders. In addition, at least partial steps of Figures 2-3 can include more sub steps or multiple stages, these sub steps or stages are not necessarily completed at the same time but can be executed in different time, the execution order of these sub steps or stages is also not necessarily in sequence order but can be performed alternately with the other steps or sub steps of other steps or at least one part of the other stages.
[0101] In an embodiment, as shown in Figure 4, an apparatus for improving the performance of a user portrait interface is provided, wherein the apparatus comprises: a data processing module 302, an obtaining module 304, a random distribution module 306, an aggregating module 308 and a data display module 310.
[0102] A data processing module 302 is configured to preprocess label data according to a set task instance; wherein the label data includes portrait label data;
[0103] An obtaining module 304 is configured to obtain portrait label information selected by a user, and generating a conditional query statement according to the portrait label infoimation;
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0104] A random distribution module 306 is configured to distribute the conditional query statement randomly to any machine of query nodes;
[0105] An aggregating module 308 is configured to obtain aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0106] A data display module 310 is configured to return the aggregated data to a client terminal for displaying to the user.
[0107] In an embodiment, the data processing module 302 is further configured to:
[0108] Group by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups;
[0109] According to the plurality of the coded data groups, generate a plurality of files with set formats corresponding to the label data;
[0110] Insert each file with the set format into a corresponding local engine table; and
[0111] Obtain the portrait label data from the local engine table, and import a corresponding correlation engine table; wherein the correlation engine table is on the machine of sharding query nodes.
[0112] In an embodiment, wherein the pre-set rule of the apparatus also comprises:
[0113] Coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and Date Regue/Date Received 2023-02-21
[0114] Grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
[0115] In an embodiment, the data processing module 302 is further configured to:
[0116] Insert each file with the set format into a corresponding blank local engine table;
and
and
[0117] Insert each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
[0118] In an embodiment, the random distribution module 306 is further configured to:
[0119] distribute the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
[0120] In an embodiment, the aggregating module 308 is further configured to:
[0121] According to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtain aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions.
[0122] Wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
[0123] In an embodiment, the apparatus is further configured to:
[0124] Associate each corresponding correlation engine table with a distributed table;
wherein the distributed table is on the machine of sharding query nodes.
Date Regue/Date Received 2023-02-21
wherein the distributed table is on the machine of sharding query nodes.
Date Regue/Date Received 2023-02-21
[0125] Wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
[0126] For the specific limitation of an apparatus for improving the performance of a user portrait interface can refer to the above-mentioned method for improving the performance of a user portrait interface, which will not be repeated here.
Each module of the above apparatus for improving the performance of a user portrait interface can be achieved fully or partly by software, hardware, and their combinations. The above modules can be embedded in the processor or independent of the processor in computer device and can store in the memory of computer device in form of software, so that the processor can call and execute the operations corresponding to the above modules.
Each module of the above apparatus for improving the performance of a user portrait interface can be achieved fully or partly by software, hardware, and their combinations. The above modules can be embedded in the processor or independent of the processor in computer device and can store in the memory of computer device in form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0127] In an embodiment, a computer device is provided, the computer device can be a server whose internal structure diagram is shown in Figure 5. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide calculation and control capabilities. The memory of the computer device includes non-volatile storage medium and internal memory. The memory of non-volatile storage medium has an operation system, computer programs and database. The internal memory provides an environment for the operation system and computer program running in a non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a method for improving the performance of a user portrait interface.
[0128] The skilled in the art can understand that the structure shown in Figure 5 is only partial structural diagram related this application solution and not constitute limitation to the computer device applied on the current application solution, the specific computer device can include more or less components than what is shown in the figure, Date Regue/Date Received 2023-02-21 or combinations of some components or different components to what is shown in the figure.
[0129] Ti an embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and ran on the processor configured to achieve the following steps when the processor executes the computer program:
[0130] Preprocessing label data according to a set task instance; wherein the label data includes portrait label data;
[0131] Obtaining portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
[0132] Distributing the conditional query statement randomly to any machine of query nodes;
[0133] Obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0134] Returning the aggregated data to a client terminal for displaying to the user.
[0135] In an embodiment, the processor is configured to achieve the following steps when the processor executes the computer program of preprocessing label data according to a set task instance:
[0136] Grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0137] According to the plurality of the coded data groups, generating a plurality of files with set formats corresponding to the label data.
[0138] Inserting each file with the set format into a corresponding local engine table.
[0139] Obtaining the portrait label data from the local engine table and importing a corresponding correlation engine table; wherein the correlation engine table is on the machine of sharding query nodes.
[0140] In an embodiment, the processor is further configured to achieve the following steps when the processor executes the computer program:
[0141] Coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and
[0142] Grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
[0143] In an embodiment, the processor is further configured to achieve the following steps when the processor executes the computer program of inserting each file with the set format into the corresponding local engine table:
[0144] Inserting each file with the set format into a corresponding blank local engine table; and
[0145] Inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
[0146] In an embodiment, the processor is further configured to achieve the following steps when the processor executes the computer program of distributing Date Regue/Date Received 2023-02-21 the conditional query statement randomly to any machine of query nodes:
[0147] Distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
[0148] In an embodiment, the processor is further configured to achieve the following steps when the processor executes the computer program of obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines, further comprises the following steps:
[0149] According to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions.
[0150] Wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
[0151] In an embodiment, the processor is further configured to achieve the following steps when the processor executes the computer program:
[0152] Associating each corresponding correlation engine table with a distributed table; wherein the distributed table is on the machine of sharding query nodes.
[0153] Wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0154] In an embodiment, a computer readable storage medium is provided, the medium stored with computer program and the processor performs the following steps when executing the computer program:
[0155] Preprocessing label data according to a set task instance; wherein the label data includes portrait label data;
[0156] Obtaining portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
[0157] Distributing the conditional query statement randomly to any machine of query nodes;
[0158] Obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and
[0159] Returning the aggregated data to a client terminal for displaying to the user.
[0160] In an embodiment, the computer program of preprocessing label data according to a set task instance is executed by the processor to achieve the following steps:
[0161] Grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups.
[0162] According to the plurality of the coded data groups, generating a plurality of files with set formats corresponding to the label data.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
[0163] Inserting each file with the set format into a corresponding local engine table.
[0164] Obtaining the portrait label data from the local engine table, and importing a corresponding correlation engine table; wherein the correlation engine table is on the machine of sharding query nodes.
[0165] In an embodiment, the computer program is executed by the processor to achieve the following steps:
[0166] Coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and
[0167] Grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
[0168] In an embodimentõ the computer program of inserting each file with the set format into the corresponding local engine table is executed by the processor to achieve the following steps:
[0169] Inserting each file with the set format into a corresponding blank local engine table; and
[0170] Inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
[0171] In an embodiment, the computer program of distributing the conditional query statement randomly to any machine of query nodes, is executed by the processor to achieve the following steps:
[0172] Distributing the conditional query statement randomly in a manner of a Date Regue/Date Received 2023-02-21 random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
[0173] In an embodiment, the computer program of obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines, is executed by the processor to achieve the following step:
[0174] According to both the conditional query statement and each portrait label data from each machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions.
[0175] Wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
[0176] In an embodiment, the computer program is executed by the processor to achieve the following steps:
[0177] Associating each corresponding correlation engine table with a distributed table; wherein the distributed table is on the machine of sharding query nodes.
[0178] Wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
[0179] The skilled in the art can understand that all or partial of procedures from the above-mentioned methods can be perfoimed by computer program instructions through related hardware, the mentioned computer program can be stored in a non-volatile Date Recue/Date Received 2023-02-21 material computer readable storage medium, this computer can include various embodiment procedures from the abovementioned methods when execution. Any reference to the memory, the storage, the database, or the other media used in each embodiment provided in current application can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programable ROM (PROM), electrically programmable ROM (EPRPMD), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an instruction but not limited to, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAMD), synchronous DRAM (SDRAM), dual data rate SDRAM
(DDRSDRAM), enhanced SRAM (ESDRAM), synchronal link (Synchlink) DRAM
(SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
(DDRSDRAM), enhanced SRAM (ESDRAM), synchronal link (Synchlink) DRAM
(SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0180] The technical features of the above-mentioned embodiments can be randomly combined, for concisely statement, not all possible combinations of technical features in the abovementioned embodiments are described. However, if there are no conflicts in the combinations of these technical features, it shall be within the scope of the present description.
[0181] The above-mentioned embodiments are only several embodiments in this disclosure and the description is more specific and detailed but cannot be understood as the limitation of the scope of the invention patent. Evidently those ordinary skilled in the art can make various modifications and variations to the disclosure without departing from the spirit and scope of the disclosure. Therefore, the appended claims are intended to be construed as encompassing the described embodiment and all the modifications and variations coming into the scope of the disclosure.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
Claims (10)
1. A method for improving the performance of a user portrait interface, comprises:
preprocessing label data according to a set task instance; wherein the label data includes portrait label data;
obtaining poi __________________________________________________________ tiait label information selected by a user, and generating a conditional query statement according to the portrait label information;
distributing the conditional query statement randomly to any machine of query nodes;
obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying to the user.
preprocessing label data according to a set task instance; wherein the label data includes portrait label data;
obtaining poi __________________________________________________________ tiait label information selected by a user, and generating a conditional query statement according to the portrait label information;
distributing the conditional query statement randomly to any machine of query nodes;
obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and returning the aggregated data to a client terminal for displaying to the user.
2. The method according to claim 1, wherein preprocessing label data according to a set task instance, comprising:
grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups;
according to the plurality of the coded data groups, generating a plurality of files with set founats corresponding to the label data;
inserting each file with the set format into a corresponding local engine table; and obtaining the portrait label data from the local engine table and importing a corresponding correlation engine table; wherein the correlation engine table is on the Date Regue/Date Received 2023-02-21 machine of sharding query nodes.
grouping by coding the label data with a pre-set rule according to the set task instance, to obtain a plurality of coded data groups;
according to the plurality of the coded data groups, generating a plurality of files with set founats corresponding to the label data;
inserting each file with the set format into a corresponding local engine table; and obtaining the portrait label data from the local engine table and importing a corresponding correlation engine table; wherein the correlation engine table is on the Date Regue/Date Received 2023-02-21 machine of sharding query nodes.
3. The method according to claim 2, wherein the pre-set rule also comprises:
coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
coding the label data according to a size order of the label data from minimum to maximum for obtaining the coded data; and grouping the coded data in a manner of a modulo sharding to obtain the plurality of coded data groups.
4. The method according to claim 2, wherein, inserting each file with the set format into the corresponding local engine table, comprising:
inserting each file with the set format into a corresponding blank local engine table; and inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
inserting each file with the set format into a corresponding blank local engine table; and inserting each file with the set format of the corresponding blank local engine table into the corresponding local engine table in a manner of a materialized view.
5. The method according to claim 1, wherein, distributing the conditional query statement randomly to any machine of query nodes, comprising:
distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
distributing the conditional query statement randomly in a manner of a random distribution of Virtual Internet Protocol address (VIP) to any machine of query nodes.
6. The method according to claim 1, wherein obtaining aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of a plurality of portrait label data from machines of a plurality of sharding query nodes, comprising:
according to both the conditional query statement and each portrait label data from each Date Regue/Date Received 2023-02-21 machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions; and wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
according to both the conditional query statement and each portrait label data from each Date Regue/Date Received 2023-02-21 machine of sharding query nodes, obtaining aggregated data of the portrait label data from each machine of sharding query nodes by calculating bitmap functions; and wherein the aggregated data is obtained on a basis of the aggregated data of portrait label data from each machine of sharding query nodes.
7. The method according to claim 6, wherein method further comprising:
associating each corresponding correlation engine table with a distributed table;
wherein the distributed table is on the machine of sharding query nodes; and wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
associating each corresponding correlation engine table with a distributed table;
wherein the distributed table is on the machine of sharding query nodes; and wherein the aggregated data is obtained on the distributed table on a basis of the aggregated data of portrait label data from each corresponding correlation engine table.
8. An apparatus for improving the performance of a user portrait interface, wherein the apparatus compri ses:
a data processing module configured to preprocess label data according to a set task instance; wherein the label data includes portrait label data;
an obtaining module configured to obtain portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
a random distribution module configured to distribute the conditional query statement randomly to any machine of query nodes;
an aggregating module configured to obtain aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and Date Recue/Date Received 2023-02-21 a data display module configured to return the aggregated data to a client terminal for displaying to the user.
a data processing module configured to preprocess label data according to a set task instance; wherein the label data includes portrait label data;
an obtaining module configured to obtain portrait label information selected by a user, and generating a conditional query statement according to the portrait label information;
a random distribution module configured to distribute the conditional query statement randomly to any machine of query nodes;
an aggregating module configured to obtain aggregated data from the machine of query nodes according to the conditional query statement, wherein the aggregated data is obtained on a basis of portrait label data from a plurality of sharding query nodes machines; and Date Recue/Date Received 2023-02-21 a data display module configured to return the aggregated data to a client terminal for displaying to the user.
9. A computer device, including a memory, a processor and a computer program stored in the memory and run on the processor configured to achieve the steps of any methods in claims 1 to 7 when the processor executes the computer program.
10. A computer readable storage medium stored with a computer program configured to achieve the steps of any methods in claim 1 to 7 when the processor executes the computer program.
Date Regue/Date Received 2023-02-21
Date Regue/Date Received 2023-02-21
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111609594.2A CN114357024A (en) | 2021-12-24 | 2021-12-24 | Method, device and computer equipment for improving user portrait interface performance |
| CN202111609594.2 | 2021-12-24 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA3184561A1 true CA3184561A1 (en) | 2023-06-24 |
Family
ID=81102308
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA3184561A Pending CA3184561A1 (en) | 2021-12-24 | 2022-12-21 | Method, device and computer equipment for improving user portrait interface performance |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114357024A (en) |
| CA (1) | CA3184561A1 (en) |
-
2021
- 2021-12-24 CN CN202111609594.2A patent/CN114357024A/en active Pending
-
2022
- 2022-12-21 CA CA3184561A patent/CA3184561A1/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| CN114357024A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10459899B1 (en) | Splitting database partitions | |
| US9996565B2 (en) | Managing an index of a table of a database | |
| US8965914B2 (en) | Grouping identity records to generate candidate lists to use in an entity and relationship resolution process | |
| US10002170B2 (en) | Managing a table of a database | |
| WO2011137189A1 (en) | System and methods for mapping and searching objects in multidimensional space | |
| CN112015820A (en) | Method, system, electronic device and storage medium for implementing distributed graph database | |
| US11422994B2 (en) | Adaptive query processor for query systems with limited capabilities | |
| CN108074210B (en) | Object acquisition system and method for cloud rendering | |
| CN107749887A (en) | A kind of CDN resource allocations, localization method and device and CDN system | |
| US20140201214A1 (en) | Creating a file descriptor independent of an open operation | |
| US20150120697A1 (en) | System and method for analysis of a database proxy | |
| CN109560940B (en) | Charging method and device for content delivery network CDN service | |
| EP3107010A1 (en) | Data integration pipeline | |
| CN111400301A (en) | Data query method, device and equipment | |
| CA3184561A1 (en) | Method, device and computer equipment for improving user portrait interface performance | |
| CN114840608B (en) | Distributed data access method, device and storage medium | |
| CN115098738A (en) | Service data extraction method and device, storage medium and electronic equipment | |
| CN112015797B (en) | Method for reading data and computer equipment | |
| CN112035760B (en) | Task allocation method and computer equipment | |
| CN114490039A (en) | Network card flow secondary allocation method, system, equipment and medium for CPU load balance | |
| CN110427390B (en) | Data query method and device, storage medium and electronic device | |
| CN114238264A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| Elghamrawy | An adaptive load-balanced partitioning module in Cassandra using rendezvous hashing | |
| Carstoiu | Cloud SaaS Infrastructure | |
| US11971946B1 (en) | Global real time on demand rendering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request |
Effective date: 20230928 |
|
| EEER | Examination request |
Effective date: 20230928 |
|
| EEER | Examination request |
Effective date: 20230928 |
|
| EEER | Examination request |
Effective date: 20230928 |
|
| EEER | Examination request |
Effective date: 20230928 |