CN109460412A - Data aggregation method, device, equipment and storage medium - Google Patents
Data aggregation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN109460412A CN109460412A CN201811352375.9A CN201811352375A CN109460412A CN 109460412 A CN109460412 A CN 109460412A CN 201811352375 A CN201811352375 A CN 201811352375A CN 109460412 A CN109460412 A CN 109460412A
- Authority
- CN
- China
- Prior art keywords
- data
- polymerization
- event object
- grouping
- field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a kind of data aggregation method, device, equipment and storage mediums.Wherein, this method comprises: the configuration attribute according to source component parses pending data, at least two event objects are generated；According to the polymeric type, polymerization grouping field mark and polymerization calculated field mark configured in blocker chain, at least two event object is polymerize.Technical solution provided in an embodiment of the present invention, by effectively being polymerize to data, so that the number of data in storage to storage equipment is reduced, thereby reduces the memory space to storage equipment and calculates the requirement of power before data storage to storage equipment.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of data aggregation method, device, equipment and storage to be situated between
Matter.
Background technique
In order to improve the performance of mass data collection, the various mature and extraordinary magnanimity of effect have been produced at present
Data acquisition components, such as the open source Flume component of common Apache foundation.Flume be a kind of open source data acquisition,
Processing loads, and can acquire data from the various data sources such as file, write data into storage equipment such as Hbase after treatment
(Hadoop database, distributed data base), HDFS (Hadoop Distribute File System, distributed document
System), Apache Kafka (distributed post-subscription message system) etc..
Flume is by three source component (source), channel components (channel), destination component (sink) primary clusterings
Composition, source are responsible for acquiring and handle, and channel is responsible for the transmitting and buffering of data, and sink is responsible for the loading of data,
Source can be packaged into event object (event) from the collected every data of data source, event by data head (header) and
Data volume (body) is constituted, and collected each initial data is stored in body, other metadata are stored in header.
Each data that wherein source has been acquired entrusts to blocker (interceptor) to be handled, and Flume is certainly
Body provides the specific implementation of some interceptor, what these realizations were handled both for single data, mainly has and adds
Add content, modification content, abandon content etc. types are more typical and have morphlineInterceptor.Therefore it is ultimately written storage
The number of data of equipment is larger, deposits biggish memory space to occupy.
Summary of the invention
The embodiment of the present invention provides a kind of data aggregation method, device, equipment and storage medium, in data storage to storage
Before equipment, by effectively being polymerize to data, so that the number of data in storage to storage equipment is reduced, thereby reduce
The requirement of memory space and calculation power to storage equipment.
In a first aspect, the embodiment of the invention provides a kind of data aggregation methods, this method comprises:
Configuration attribute according to source component parses pending data, generates at least two event objects；
It is right according to the polymeric type, polymerization grouping field mark and polymerization calculated field mark configured in blocker chain
At least two event object is polymerize.
Second aspect, the embodiment of the invention also provides a kind of data aggregate device device, which includes:
Event object generation module parses pending data for the configuration attribute according to source component, generates extremely
Few two event objects；
Data aggregate module, for identifying and gathering according to the polymeric type, the polymerization grouping field that are configured in blocker chain
Total field identification of calculating polymerize at least two event object.
The third aspect, the embodiment of the invention also provides a kind of equipment, which includes:
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes any data aggregation method in first aspect.
Fourth aspect, the embodiment of the invention also provides a kind of storage mediums, are stored thereon with computer program, the program
Any data aggregation method in first aspect is realized when being executed by processor.
Data aggregation method, device, equipment and storage medium provided in an embodiment of the present invention, in the configuration according to source component
After attribute carries out parsing at least two event objects of generation to pending data, pass through according to the polymeric type configured in blocker chain
Type, polymerization grouping field mark and polymerization calculated field mark, polymerize at least two event objects of generation, in turn
So that the number of data in storage to storage equipment is reduced.In addition, compared with the prior art, the program is in data storage to storage
Before equipment, by effectively being polymerize to data, also further reduced the memory space to storage equipment and calculating wanting for power
It asks.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, of the invention other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is a kind of flow chart of the data aggregation method provided in the embodiment of the present invention one；
Fig. 2 is a kind of flow chart of the data aggregation method provided in the embodiment of the present invention two；
Fig. 3 is a kind of flow chart of the data aggregation method provided in the embodiment of the present invention three；
Fig. 4 is a kind of flow chart of the data aggregation method provided in the embodiment of the present invention four；
Fig. 5 is a kind of flow chart of the data aggregation method provided in the embodiment of the present invention five；
Fig. 6 is a kind of structural block diagram of the data aggregate device provided in the embodiment of the present invention six；
Fig. 7 is a kind of structural schematic diagram of the equipment provided in the embodiment of the present invention seven.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
State that the specific embodiments are only for explaining the present invention, rather than limitation of the invention.It also should be noted that for the ease of
Description, only some but not all contents related to the present invention are shown in the drawings.
Embodiment one
Fig. 1 is the flow chart of a kind of data aggregation method that the embodiment of the present invention one provides, the present embodiment be suitable for how
Data are effectively polymerize before data are stored in storage equipment, to reduce the memory space to storage equipment and calculate power
It is required that the case where, it is particularly suitable for the scene to the data aggregate based on Flume.This method can be provided by the embodiment of the present invention
Data aggregate device execute, the mode which can be used software and/or hardware realizes, is configured in and calculates in equipment.
Referring to Fig. 1, this method is specifically included:
S110, the configuration attribute according to source component parse pending data, generate at least two event objects.
In the present embodiment, pending data is the initial data acquired from data source；Data source is to provide needed for certain
Want the device or original media of data；The configuration attribute of source component refers to be arranged according to actual needs, in order to complete to treat place
Manage the acquisition of data and the attribute information that respective handling operation is set.The configuration attribute of source component may include data source class
Type, reading data catalogue and data parsing type etc..It optionally, can be according to actually required data cases dynamic adjustment source component
Configuration attribute in it is one or more.
Data source types are the attribute settings of data source, can include but is not limited to Spooling Directory
Source(spooldir)、AvroSource、ThriftSource、NetCat Source、Sequence Generator
The types such as Source and HTTP Source.Wherein, Spooling Directory Source is for monitoring in specified directory
Data change；AvroSource receives the flow of event from external Avro client for monitoring the port Avro, can also receive
The log information sent by the Avro client that Flume is provided；NetCat Source is used to monitor a designated port, and
Every a line of the data received is converted into an event object；Data parsing type (deserializer type) refers to use
In the type for being handled pending data or being parsed, can include but is not limited to LINE, Avro and
Different data parsing classes may be selected in the types such as BlobDeserializer, optionally, the data format difference according to data source
Type is defaulted as LINE.Wherein, every row that LINE type is used to enter text into is converted into an event object.
Specifically, the configuration attribute of source component can be set according to actual demand, it then can be according to the configuration category of source component
Data source types and reading data catalogue in property etc., acquire pending data from corresponding data source types, Jin Eryi
Pending data is parsed according to the data parsing type in the configuration attribute of source component, generates at least two event objects.
It illustratively, can also include: foundation before being parsed according to the configuration attribute of source component to pending data
Data source types in the configuration attribute of source component determine target data source；Pending data is acquired from target data source.Its
In, target data source refers to one or more data sources for acquiring required data.Specifically, can matching according to source component
The selection target data source from data source of the data source types in attribute is set, is then read according to data in the configuration attribute of source component
Catalogue is taken to read pending data from target data source.
It illustratively, can also include: foundation before being parsed according to the configuration attribute of source component to pending data
The configuration attribute of source component is arranged in actual demand.
S120, according to the polymeric type, polymerization grouping field mark and polymerization calculated field mark configured in blocker chain
Know, at least two event objects are polymerize.
Blocker chain is made of one or more blockers, and in the present embodiment, blocker chain is customized including at least one
Polymerization blocker such as preAggInterceptor, which can be used for simultaneously gathering multiple event objects
It closes, so that M event object be made to be polymerized to N (M > N) a event object after the processing of blocker chain, and then reduces data strip
Number.
In addition, the data type of data source determines the type of pending data collected, if collected to be processed
Data are unstructured datas, optionally, can also be including a structure intercepts device such as in blocker chain
MorphlineInterceptor, for carrying out structuring processing to event object.For example, if pending data is unstructured
Data can first be inputted in morphlineInterceptor and be carried out after source component dissection process generates M event object
Processing, obtains the event object of M structuring, then inputs preAggInterceptor and is polymerize, obtains N (M > N) a thing
Part object.If pending data collected is structural data, through source component dissection process generate M event object after,
PreAggInterceptor can be directly inputted to be polymerize, N (M > N) a event object is obtained.It specifically can be according to blocker chain
Configuration item such as polymeric type, the polymerization grouping field of middle polymerization blocker identify and polymerize calculated field etc. to realize multiple things
The polymerization of part object.
In the present embodiment, polymeric type such as can be counting for indicating the mode that multiple event objects are polymerize
(COUNT), sum (SUM) or be averaging (AVG) etc.；Polymerization grouping field mark refers to for dividing multiple event objects
The field identification or benchmark of group；Polymerization calculated field mark is for stating to which field polymerization in multiple event objects.
It is identified according to the polymerization grouping field of polymerization blocker configuration in blocker chain at least two specifically, can be
Event object is grouped, and is then directed to each grouping, is calculated mark according to polymerization and is obtained from event object each in the group
The polymerization, which calculates, identifies associated field value, and identifies and close according to polymerization calculated field of the polymeric type to event object in the group
The field value of connection is polymerize, to realize the polymerization of at least two event objects.It is also possible to polymerize in foundation blocker chain
The polymerization grouping field mark of blocker configuration screens at least two event objects, rejects the event pair for being unsatisfactory for requiring
As；Then remaining event object is grouped according to polymerization grouping field mark, and then is based on packet configuration, according to polymerization
Type identifies associated field value to the polymerization calculated field of event object in each grouping and polymerize, to realize at least two
The polymerization of a event object.
It can also be and identify according to the polymerization grouping field mark and polymerization calculated field configured in blocker chain to extremely
Few two event objects are simply screened, and then calculate word according to polymeric type and polymerization to the event object after screening
Segment identification is polymerize, and then realizes the polymerization etc. of at least two event objects.
It is to count (COUNT) for polymeric type, then the polymeric type that can configure in direct basis blocker chain and poly-
It is total to calculate field identification, at least two event objects are polymerize.
Illustratively, word is calculated according to the polymeric type, polymerization grouping field mark and polymerization configured in blocker chain
Segment identification can also include: to be stored by specified channel components to target after polymerizeing at least two event objects
Data after equipment such as Hbase, Apache Kafka or HDFS transmission polymerization, so that target storage device carries out local deposit
Storage.
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, by according to configure in blocker chain polymeric type, polymerization grouping field mark and
It polymerize calculated field mark, at least two event objects of generation is polymerize, so that in storage to storage equipment
Number of data is reduced.In addition, compared with the prior art, the program before data storage to storage equipment, by data into
Row effectively polymerization also further reduced the memory space to storage equipment and calculate the requirement of power.
Embodiment two
Fig. 2 is a kind of flow chart of data aggregation method provided by Embodiment 2 of the present invention, and the present embodiment is in above-mentioned implementation
On the basis of example, provide a kind of to according to the polymeric type, polymerization grouping field mark and polymerization configured in blocker chain
The scheme that calculated field polymerize at least two event objects.Referring to fig. 2, this method specifically includes:
S210, the configuration attribute according to source component parse pending data, generate at least two event objects.
S220 is obtained from the data head of each event object according to the polymerization grouping field mark configured in blocker chain
It polymerize grouping field and identifies associated grouping field value.
It should be noted that the event object of structuring, the data in data head are with key-value pair (Key-Value) shape
Formula storage, Key represents field identification, and Value is the associated field value of field identification.In the data head of one event object,
One field identification uniquely corresponds to a field value；It is corresponding, it can determine associated grouping according to polymerization grouping field mark
Field value.Optionally, the quantity of polymerization grouping field mark is used to characterize the dimension of grouping, can be one, can also be multiple,
It can specifically be configured according to the actual situation.
As shown in table 1, the second row to the 6th row, the data head of each one event object of behavior, name in the first row,
Gender and achievement are respectively different field identifications, and the occurrence (the second row to the 6th row) under it is then associated with for field identification
Field value.Such as name 1 is first associated field value of this field identification of event object name.If polymerizeing grouping field
It is identified as gender, then according to this polymerization grouping field mark of gender, is successively searched from the data head of each event object, it can be with
It obtains the polymerization grouping field in each event object and identifies associated grouping field value, be followed successively by male, male, female, female and female.
Table 1
| Name | Gender | Achievement |
| Name 1 | Male | 89 |
| Name 2 | Male | 88 |
| Name 3 | Female | 98 |
| Name 4 | Female | 99 |
| Name 5 | Female | 90 |
S230 is grouped at least two event objects according to each grouping field value.
Specifically, according to the polymerization grouping field mark configured in blocker chain, from the data head of each event object
It, can be according to each grouping field value, by grouping field value phase after obtaining the associated grouping field value of polymerization grouping field mark
Same is classified as one kind, and then realizes the grouping of at least two event objects.
By taking table 1 and polymerization grouping field are identified as gender as an example, to how to carry out the division operations of multiple event objects into
Row explanation.Specifically, male, male, female, female and female are followed successively by according to the grouping field value that gender obtains each event object, according to male
Five event objects are grouped with female, two groups can be divided into.Wherein, the corresponding event object 1 of the second row and the third line are corresponding
Event object 2 form one group, the corresponding event object 5 of corresponding 3 to the 6th row of event object of fourth line forms one group.
Illustratively, if the quantity of polymerization grouping field mark is at least two, according to each grouping field value, at least two
A event object, which is grouped, to include:
A, using the Aggregate connectors configured in blocker chain, each polymerization grouping field mark for connecting each event object is closed
The grouping field value of connection obtains the polymerization grouping field value of each event object；
In the present embodiment, Aggregate connectors refer to associated for connecting two or more polymerization grouping field marks
Symbol between grouping field value, such as can be "-", "+", " & ", it can specifically be configured according to the actual situation.Polymerization point
Group field value is made of Aggregate connectors and at least two grouping field values, such as "-one class of male ".
Such as table 2, a field identification class is increased on the basis of table 1, if polymerization grouping field is identified as gender and class
Grade obtains each polymerization point according to each polymerization grouping field mark for each event object from the data head of the event object
Group field identifies associated grouping field value, then connects each grouping field value using Aggregate connectors, the thing can be obtained
The polymerization grouping field value of part object.For example, being illustrated by taking the corresponding event object 1 of the second row as an example.If Aggregate connectors
For "-", the associated grouping field value of gender is male, and the associated grouping field value of class is one class, then polymerizeing grouping field value is
"-one class of male ".
Optionally, respectively it polymerize the order of connection that grouping field identifies associated grouping field value, can exists according to each key-value pair
The storage order of event object data head is attached, and can also be attached according to the order of connection configured in blocker chain
Deng.
Table 2
B, according to each polymerization grouping field value, at least two event objects are grouped.
It is illustrated by taking table 2 as an example.If polymerization grouping field is identified as gender and class, event object 1 to event object 5
Polymerization grouping field value successively are as follows:-one class of male ,-two classes of male ,-two classes of female ,-one class of female and-one class of female；It is poly- according to above-mentioned 5
It closes grouping field value to be grouped event object 1 to event object 5, four groups can be divided into.Wherein, event object 1 is one group, thing
Part object 2 is one group, and event object 3 is one group, and event object 4 and event object 5 are one group.
S240, according to the polymeric type and polymerization calculated field mark configured in blocker chain, successively to the thing of each grouping
Part object is polymerize.
Specifically, after to the grouping of at least two event objects, it, can be according to polymerization calculated field mark for each grouping
The associated polymerization calculated field value of polymerization calculated field mark for obtaining each event object in the group is known, then according to polymeric type
Type polymerize each polymerization calculated field value, and then completes the polymerization of multiple event objects.
It is illustrated for example, being exemplified by Table 1.If polymeric type is to be averaging (AVG), polymerization calculated field is identified as into
Achievement is grouped 5 event objects, obtains one group and one group of female of male.For one group of male, can be identified according to polymerization calculated field,
It is 89 and 88 that the polymerization calculated field for obtaining each event object in the group, which identifies associated polymerization calculated field value,；Then using poly-
It closes type to be averaging, 89 and 88 is polymerize, obtain 88.5.Using identical operating procedure, one group of female is polymerize, is obtained
To 95.7.Referring specifically to table 3, it can be seen that, the present embodiment is by the converging operation of blocker chain by multiple event objects such as 5
It is a, 2 are polymerized to, number of data is reduced, thereby reduces the requirement to storage device memory and calculation power etc..
Table 3
| Male | 88.5 |
| Female | 95.7 |
The table 2 being polymerize is grouped for various dimensions, final polymerization result is as shown in table 4.
Table 4
| - one class of male | 89 |
| - two classes of male | 88 |
| - one class of female | 94.5 |
| - two classes of female | 98 |
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, by according to configure in blocker chain polymeric type, polymerization grouping field mark and
It polymerize calculated field mark, at least two event objects of generation is polymerize, so that in storage to storage equipment
Number of data is reduced.In addition, compared with the prior art, the program before data storage to storage equipment, by data into
Row effectively polymerization also further reduced the memory space to storage equipment and calculate the requirement of power.
Embodiment three
Fig. 3 is a kind of flow chart for data aggregation method that the embodiment of the present invention three provides, and the present embodiment is in above-mentioned implementation
On the basis of example, and provide a kind of to according to polymeric type, the polymerization grouping field mark and poly- configured in blocker chain
Close the scheme that calculated field polymerize at least two event objects.Referring to Fig. 3, this method is specifically included:
S310, the configuration attribute according to source component parse pending data, generate at least two event objects.
S320 is obtained from the data head of each event object according to the polymerization grouping field mark configured in blocker chain
It polymerize grouping field and identifies associated grouping field value.
S330 is obtained from the data head of each event object according to the polymerization calculated field mark configured in blocker chain
It polymerize calculated field and identifies associated polymerization calculated field value.
It is illustrated by taking the table 1 in embodiment two as an example, if polymerization calculated field is identified as achievement, the corresponding thing of the second row
In the corresponding event object 5 of 1 to the 6th row of part object, it is to have which, which identifies associated polymerization calculated field value,
The achievement number such as 89 of body.
S340, according to preset polymerization calculated field value screening conditions and/or grouping field value screening conditions, to
Event object to be polymerized is selected in few two event objects.
In the present embodiment, it polymerize calculated field value screening conditions and grouping field value screening conditions is preparatory in blocker chain
Configuration, it can be configured according to actual needs.For example, polymerizeing calculated field value screening conditions for the event object of data
It can be and reject the event object for being less than setting polymerization calculated field threshold value, can also be will not polymerize calculated field in setting
Event object rejecting in threshold value (upper limit value and lower limit value) etc.；Which point be grouping field value screening conditions for showing for
Group field value is grouped.For example, the table 2 in embodiment two, grouping field value screening conditions can for gender be woman into
Row grouping.
Event object to be polymerized refers to the event object for needing to carry out converging operation at least two event objects.Specifically
, it can be according to preset polymerization calculated field value screening conditions and/or grouping field value screening conditions, from least two
Event object to be polymerized is selected in event object.
Specifically, being illustrated by taking the table 1 in embodiment two as an example.If polymerizeing calculated field value screening conditions is to achievement
Event object more than or equal to 89 is polymerize.According to polymerization calculated field value screening conditions to event object 1 to event object 5
It is screened, it is final to determine that meeting screening conditions event object is event object 1, event object 3, event object 4 and event pair
As 5.Using the event object for the condition that meets as event object to be polymerized.
It is illustrated by taking the table 2 in embodiment two as an example.If polymerize calculated field value screening conditions to be greater than achievement etc.
Event object in 89 is polymerize, and grouping field value screening conditions are that gender is female, and the event object that class is one class carries out
Polymerization.Event object 1 to event object 5 is carried out according to polymerization calculated field value screening conditions and grouping field value screening conditions
Screening, it is final to determine that meeting screening conditions event object is event object 4 and event object 5.By event object 4 and event object
5 are used as event object to be polymerized.
S350 treats polymerization events object and is grouped according to each grouping field value.
It is illustrated by taking the table 1 in embodiment two as an example.Polymerization events object is treated according to male and female to be grouped, and can be divided
It is two groups.Wherein, the corresponding event object 1 of the second row forms one group, and corresponding 3 to the 6th row of event object of fourth line is corresponding
Event object 5 forms one group.
S360, according to the polymeric type and polymerization calculated field mark configured in blocker chain, successively to the thing of each grouping
Part object is polymerize.
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, lead to before polymerizeing at least two event objects, first according to preset poly-
It is total to calculate field value screening conditions and/or grouping field value screening conditions screen at least two event objects, determine to
Polymerization events object；Then again according to the polymeric type and polymerization calculated field mark configured in blocker chain, to be polymerized
Event object is polymerize, so that the number of data in storage to storage equipment is reduced, and according to actually carrying out polymerization behaviour
Make, increases the flexibility of polymerization.In addition, compared with the prior art, the program passes through before data storage to storage equipment
Data are effectively polymerize, also further reduced the memory space to storage equipment and calculate the requirement of power.
Example IV
Fig. 4 is a kind of flow chart for data aggregation method that the embodiment of the present invention four provides, and the present embodiment is in above-mentioned implementation
It is further to optimize on the basis of example.Referring to fig. 4, this method specifically includes:
S410, the configuration attribute according to source component parse pending data, generate at least two event objects.
S420, for each event object, the configuration item according to structure intercepts device in blocker chain is to the event object
Data volume carries out structuring processing, obtains each field value of the event object.
Specifically, generating M through source component dissection process if pending data collected is unstructured data
After event object, needs first to carry out structuring processing to the M event object using structure intercepts device and then input polymerization is blocked
Device is cut to be polymerize.Structure intercepts device is preferably morphlineInterceptor.
In the present embodiment, the configuration item of structure intercepts device may include carrying out structuring processing to the data volume of event object
Structured command.Since the data stored in the data volume of an event object are initial data, to the knot of data volume
Structureization processing is that the structuring to initial data is handled.It optionally, can be according to the data format of the data volume of event object
The corresponding structured command of flexible choice completes the Green Formula of data.
If structure intercepts device is morphlineInterceptor, structured command can be morphline order.
Morphline order may include split, extractAvroPaths or other can complete data structured
Morphline order.Wherein, split is used for the title specified data delimiter for text data, specify every column data, with
Make in data head of the initial data according to configuration structure to event object.ExtractAvroPaths is used to be directed to Avro format
Data specify relevant configuration so that in data head of the initial data according to configuration structure to event object.
Specifically, each event object is directed to, using in structure intercepts device such as morphlineInterceptor
Morphline order carries out structuring processing to the initial data in data volume, obtains each field identification pair in the data volume
The field value answered.
Each field identification and corresponding each field value are added in the data head of the event object by S430.
Specifically, being directed to each event object, each field identification and structuring that can include by the event object be obtained
Each field value be stored in the event object in the form of key-value pair (Key-Value) structural body in, be convenient for subsequent quick lookup
And subsequent quick carry out converging operation etc..
S440, according to the polymeric type, polymerization grouping field mark and polymerization calculated field mark configured in blocker chain
Know, to treated, at least two event objects polymerize.
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, by according to configure in blocker chain polymeric type, polymerization grouping field mark and
It polymerize calculated field mark, at least two event objects of generation is polymerize, so that in storage to storage equipment
Number of data is reduced.In addition, compared with the prior art, the program before data storage to storage equipment, by data into
Row effectively polymerization also further reduced the memory space to storage equipment and calculate the requirement of power.
Embodiment five
Fig. 5 is a kind of flow chart for data aggregation method that the embodiment of the present invention five provides, and the present embodiment is in above-mentioned implementation
On the basis of example, a kind of preferable example is provided.Referring to Fig. 5, this method is specifically included:
The configuration attribute of source component is arranged in S510.
Wherein, the data source types of source component be spooldir, data parse type be LINE, reading data catalogue be/
tmp/csv。
Specifically, this gives a kind of Flume source configuration example, it is as follows:
1. agent.sources.spooldir.type=spooldir
2. agent.sources.spooldir.deserializer=LINE
3. agent.sources.spooldir.spoolDir=/tmp/csv
The configuration item of structure intercepts device and polymerization blocker in blocker chain is arranged in S520.
In the present embodiment, structure intercepts device is morphlineInterceptor, and polymerization blocker is
preAggInterceptor.The configuration item of polymerization blocker includes: that polymeric type is identified as COUNT, polymerization calculated field
Field1 and Field2, Aggregate connectors be-.The morphline order of structure intercepts device is split.
4. agent.sources.spooldir.interceptors=morphlineInterceptor,
preAggInterceptor
5. agent.sources.spooldir.interceptors.morphlineInterceptor .type=org
.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
⑥agent.sources.spooldir.interceptors.morphlineInterceptor.morphline
File=/etc/flume-ng/conf/morphline_conf.js
⑦agent.sources.spooldir.interceptors.morphlineInterceptor.morphline
Id=morphline1
8. agent.sources.spooldir.interceptors.preAggInterceptor.ty pe=com.co
mpany.flume.interceptor.PreAggInterceptor$Builder
9. agent.sources.spooldir.interceptors.preAggInterceptor.he ader=
Field1,Field2
10. agent.sources.spooldir.interceptors.preAggInterceptor.ag gMethod=
COUNT
agent.sources.spooldir.interceptors.preAggInterceptor.connectChar
=-
It wherein, is 8. the building class of preAggInterceptor；5. being the structure of built-in morphlineInterceptor
Class is built, is 6. the morphlineFile reading position of built-in morphlineInterceptor, here is configuration content:
S530, the configuration attribute according to source component parse pending data, generate two event objects.
Specifically, 1. determining target data source according in the configuration attribute of source component in step S510, then foundation is matched
It sets in attribute and 3. reads pending data from the specified directory of target data source, and then using 2. treating in configuration attribute
Processing data are parsed, and two event objects are generated.
S540, for each event object, the configuration item according to structure intercepts device in blocker chain is to the event object
Data volume carries out structuring processing, obtains each field value of the event object.
Each field identification and corresponding each field value are added in the data head of the event object by S550.
S560, polymeric type, polymerization calculated field mark and polymerization according to polymerization blocker configuration in blocker chain
Connector polymerize two event objects.
Using the configuration item for polymerizeing blocker in step S520, two event objects are polymerize.
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, by the blocker chain that is made of polymerization blocker and structure intercepts device to generation at least
Two event objects are polymerize, so that the number of data in storage to storage equipment is reduced.In addition, compared to existing skill
Art, the program, by effectively being polymerize to data, also further reduced to storage before data storage to storage equipment
The memory space of equipment and the requirement for calculating power.
Embodiment six
Fig. 6 is a kind of structural block diagram for data aggregate device that the embodiment of the present invention six provides, which can be performed this hair
Data aggregation method provided by bright any embodiment has the corresponding functional module of execution method and beneficial effect.Such as Fig. 6 institute
Show, the apparatus may include:
Event object generation module 610 parses pending data for the configuration attribute according to source component, generates
At least two event objects；
Data aggregate module 620, for according to configure in blocker chain polymeric type, polymerization grouping field mark and
Polymerization calculated field mark polymerize at least two event objects.
Technical solution provided in an embodiment of the present invention parses pending data in the configuration attribute according to source component
After generating at least two event objects, by according to configure in blocker chain polymeric type, polymerization grouping field mark and
It polymerize calculated field mark, at least two event objects of generation is polymerize, so that in storage to storage equipment
Number of data is reduced.In addition, compared with the prior art, the program before data storage to storage equipment, by data into
Row effectively polymerization also further reduced the memory space to storage equipment and calculate the requirement of power.
Illustratively, data aggregate module 620 may include:
Grouping value acquiring unit, for being identified according to the polymerization grouping field configured in blocker chain, from each event object
Data head in obtain polymerization grouping field and identify associated grouping field value；
Grouped element, for being grouped at least two event objects according to each grouping field value；
Polymerized unit, for being identified according to the polymeric type and polymerization calculated field that are configured in blocker chain, successively to each
The event object of grouping is polymerize.
Illustratively, if the quantity of polymerization grouping field mark is at least two, grouped element can be specifically used for:
Using the Aggregate connectors configured in blocker chain, each polymerization grouping field mark association of each event object is connected
Grouping field value, obtain the polymerization grouping field value of each event object；
According to each polymerization grouping field value, at least two event objects are grouped.
Illustratively, grouped element can also be specifically used for:
According to the polymerization calculated field mark configured in blocker chain, polymerization meter is obtained from the data head of each event object
Calculate the associated polymerization calculated field value of field identification；
According to preset polymerization calculated field value screening conditions and/or grouping field value screening conditions, from least two
Event object to be polymerized is selected in a event object；
According to each grouping field value, treats polymerization events object and be grouped.
Illustratively, above-mentioned apparatus can also include: structurized module, which can be used for:
For each event object, the configuration item according to structure intercepts device in blocker chain is to the data volume of the event object
Structuring processing is carried out, each field value of the event object is obtained；
Each field identification and corresponding each field value are added in the data head of the event object.
Illustratively, above-mentioned apparatus can also include: data acquisition module, which can be used for:
Target data source is determined according to the data source types in the configuration attribute of source component；
Pending data is acquired from target data source.
Embodiment seven
Fig. 7 is a kind of structural schematic diagram for equipment that the embodiment of the present invention seven provides.Fig. 7, which is shown, to be suitable for being used to realizing this
The block diagram of the example devices 12 of invention embodiment.The equipment 12 that Fig. 7 is shown is only an example, should not be to of the invention real
The function and use scope for applying example bring any restrictions.
As shown in fig. 7, the equipment 12 is showed in the form of universal computing device.The component of the equipment 12 may include but not
Be limited to: one or more processor or processing unit 16, system storage 28 connect different system components (including system
Memory 28 and processing unit 16) bus 18.
Bus 18 indicates one of a few class bus structures or a variety of, including memory bus or Memory Controller,
Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.It lifts
For example, these architectures include but is not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)
Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Equipment 12 typically comprises a variety of computer system readable media.These media can be it is any can be by equipment 12
The usable medium of access, including volatile and non-volatile media, moveable and immovable medium.
System storage 28 may include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (RAM) 30 and/or cache memory 32.Equipment 12 may further include it is other it is removable/nonremovable,
Volatile/non-volatile computer system storage medium.Only as an example, storage system 34 can be used for reading and writing irremovable
, non-volatile magnetic media (Fig. 7 do not show, commonly referred to as " hard disk drive ").Although being not shown in Fig. 7, use can be provided
In the disc driver read and write to removable non-volatile magnetic disk (such as " floppy disk "), and to removable anonvolatile optical disk
The CD drive of (such as CD-ROM, DVD-ROM or other optical mediums) read-write.In these cases, each driver can
To be connected by one or more data media interfaces with bus 18.System storage 28 may include that at least one program produces
Product, the program product have one group of (for example, at least one) program module, these program modules are configured to perform of the invention each
The function of embodiment.
Program/utility 40 with one group of (at least one) program module 42 can store and store in such as system
In device 28, such program module 42 includes but is not limited to operating system, one or more application program, other program modules
And program data, it may include the realization of network environment in each of these examples or certain combination.Program module 42
Usually execute the function and/or method in embodiment described in the invention.
Equipment 12 can also be communicated with one or more external equipments 14 (such as keyboard, sensing equipment, display 24 etc.),
Can also enable a user to the equipment interacted with the equipment communication with one or more, and/or with enable the equipment 12 and one
A or a number of other any equipment (such as network interface card, modem etc.) communications for calculating equipment and being communicated.This communication
It can be carried out by input/output (I/O) interface 22.Also, equipment 12 can also by network adapter 20 and one or
Multiple networks (such as local area network (LAN), wide area network (WAN) and/or public network, such as internet) communication.As shown, net
Network adapter 20 is communicated by bus 18 with other modules of equipment 12.It should be understood that being set although not shown in the drawings, can combine
Standby 12 use other hardware and/or software modules, including but not limited to: microcode, device driver, redundant processing unit, outer
Portion's disk drive array, RAID system, tape drive and data backup storage system etc..
Processing unit 16 by the program that is stored in system storage 28 of operation, thereby executing various function application and
Data processing, such as realize data aggregation method provided by the embodiment of the present invention.
Embodiment eight
The embodiment of the present invention eight additionally provides a kind of computer readable storage medium, is stored thereon with computer program, should
Program can realize any data aggregation method in above-described embodiment when being executed by processor.
The computer storage medium of the embodiment of the present invention, can be using any of one or more computer-readable media
Combination.Computer-readable medium can be computer-readable signal media or computer readable storage medium.It is computer-readable
Storage medium can be for example but not limited to: electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or
Any above combination of person.The more specific example (non exhaustive list) of computer readable storage medium includes: with one
Or the electrical connections of multiple conducting wires, portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM),
The read-only storage (CD-ROM) of erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc, light are deposited
Memory device, magnetic memory device or above-mentioned any appropriate combination.In this document, computer readable storage medium can be with
To be any include or the tangible medium of storage program, the program can be commanded execution system, device or device use or
It is in connection.
Computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.
The program code for including on computer-readable medium can transmit with any suitable medium, including but not limited to:
Wirelessly, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereof
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
It further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
It fully executes, partly execute on the user computer on the user computer, being executed as an independent software package, portion
Divide and partially executes or executed on a remote computer or server completely on the remote computer on the user computer.?
It is related in the situation of remote computer, remote computer can include local area network (LAN) or wide area by the network of any kind
Net (WAN) is connected to subscriber computer, or, it may be connected to outer computer (such as using ISP come
It is connected by internet).
Above-described embodiment serial number is for illustration only, does not represent the advantages or disadvantages of the embodiments.
Will be appreciated by those skilled in the art that each module of the above invention or each step can use general meter
Device is calculated to realize, they can be concentrated on single computing device, or be distributed in network constituted by multiple computing devices
On, optionally, they can be realized with the program code that computer installation can be performed, so as to be stored in storage
It is performed by computing device in device, perhaps they are fabricated to each integrated circuit modules or will be more in them
A module or step are fabricated to single integrated circuit module to realize.In this way, the present invention is not limited to any specific hardware and
The combination of software.
All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with
The difference of other embodiments, the same or similar part between each embodiment may refer to each other.
The above description is only a preferred embodiment of the present invention, is not intended to restrict the invention, for those skilled in the art
For, the invention can have various changes and changes.All any modifications made within the spirit and principles of the present invention are equal
Replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of data aggregation method characterized by comprising
Configuration attribute according to source component parses pending data, generates at least two event objects；
According to the polymeric type, polymerization grouping field mark and polymerization calculated field mark configured in blocker chain, to described
At least two event objects are polymerize.
2. the method according to claim 1, wherein according to the polymeric type configured in blocker chain, polymerizeing and dividing
Group field mark and polymerization calculated field polymerize at least two event object, comprising:
According to the polymerization grouping field mark configured in blocker chain, the polymerization point is obtained from the data head of each event object
Group field identifies associated grouping field value；
According to each grouping field value, at least two event object is grouped；
According to the polymeric type and polymerization calculated field mark configured in blocker chain, successively the event object of each grouping is carried out
Polymerization.
3. according to the method described in claim 2, it is characterized in that, if the quantity of the polymerization grouping field mark is at least two
It is a, according to each grouping field value, at least two event object is grouped, comprising:
Using the Aggregate connectors configured in blocker chain, each polymerization grouping field for connecting each event object identifies associated point
Group field value obtains the polymerization grouping field value of each event object；
According to each polymerization grouping field value, at least two event object is grouped.
4. according to the method described in claim 2, it is characterized in that, according to each grouping field value, at least two event
Object is grouped, comprising:
According to the polymerization calculated field mark configured in blocker chain, the polymerization meter is obtained from the data head of each event object
Calculate the associated polymerization calculated field value of field identification；
According to preset polymerization calculated field value screening conditions and/or grouping field value screening conditions, from described at least two
Event object to be polymerized is selected in a event object；
According to each grouping field value, the event object to be polymerized is grouped.
5. the method according to claim 1, wherein according to the polymeric type configured in blocker chain, polymerizeing and dividing
Before group field mark and polymerization calculated field polymerize at least two event object, further includes:
For each event object, the configuration item according to structure intercepts device in blocker chain carries out the data volume of the event object
Structuring processing, obtains each field value of the event object；
Each field identification and corresponding each field value are added in the data head of the event object.
6. the method according to claim 1, wherein the configuration attribute according to source component carries out pending data
Before parsing, further includes:
Target data source is determined according to the data source types in the configuration attribute of source component；
The pending data is acquired from the target data source.
7. a kind of data aggregate device characterized by comprising
Event object generation module parses pending data for the configuration attribute according to source component, generates at least two
A event object；
Data aggregate module, based on according to the polymeric type, polymerization grouping field mark and polymerization configured in blocker chain
Field identification is calculated to polymerize at least two event object.
8. the device according to shown in claim 7, which is characterized in that the data aggregate module includes:
Grouping value acquiring unit, for being identified according to the polymerization grouping field configured in blocker chain, from the number of each event object
Associated grouping field value is identified according to the polymerization grouping field is obtained in head；
Grouped element, for being grouped at least two event object according to each grouping field value；
Polymerized unit, for being identified according to the polymeric type and polymerization calculated field that are configured in blocker chain, successively to each grouping
Event object polymerize.
9. a kind of equipment, which is characterized in that the equipment includes:
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
Now such as data aggregation method as claimed in any one of claims 1 to 6.
10. a kind of storage medium, is stored thereon with computer program, which is characterized in that the realization when program is executed by processor
Such as data aggregation method as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811352375.9A CN109460412A (en) | 2018-11-14 | 2018-11-14 | Data aggregation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811352375.9A CN109460412A (en) | 2018-11-14 | 2018-11-14 | Data aggregation method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109460412A true CN109460412A (en) | 2019-03-12 |
Family
ID=65610353
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811352375.9A Pending CN109460412A (en) | 2018-11-14 | 2018-11-14 | Data aggregation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109460412A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110334117A (en) * | 2019-07-17 | 2019-10-15 | 北京脉冲星科技有限公司 | A kind of real-time data processing method and device |
| CN110598072A (en) * | 2019-09-24 | 2019-12-20 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN111198882A (en) * | 2019-12-26 | 2020-05-26 | 东软集团股份有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN112115164A (en) * | 2019-06-19 | 2020-12-22 | 北京金山云网络技术有限公司 | Data processing method and device, data query method and device, and network equipment |
| CN112131258A (en) * | 2020-09-23 | 2020-12-25 | 创新奇智(重庆)科技有限公司 | Data splicing method, device and equipment and computer storage medium |
| CN112242965A (en) * | 2019-07-18 | 2021-01-19 | 特拉维夫迈络思科技有限公司 | Telemetry event aggregation |
| CN112416972A (en) * | 2020-09-25 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Real-time data stream processing method, device, equipment and readable storage medium |
| CN113094359A (en) * | 2021-02-24 | 2021-07-09 | 华南师范大学 | Examination data preprocessing method, system, device and medium based on decision rule |
| CN113821519A (en) * | 2020-11-26 | 2021-12-21 | 北京沃东天骏信息技术有限公司 | Data processing method and field-driven design architecture |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446194A (en) * | 2010-10-14 | 2012-05-09 | 金蝶软件（中国）有限公司 | Method and device for grouping and checking product data |
| CN103838574A (en) * | 2014-02-20 | 2014-06-04 | 浪潮集团山东通用软件有限公司 | General method for grouped summarizing of data tables |
| CN104199831A (en) * | 2014-07-31 | 2014-12-10 | 深圳市腾讯计算机系统有限公司 | Information processing method and device |
| CN104408179A (en) * | 2014-12-15 | 2015-03-11 | 北京国双科技有限公司 | Method and device for processing data from data table |
| CN105930379A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Method and system for collecting log data by means of interceptor |
| CN106250519A (en) * | 2016-08-04 | 2016-12-21 | 曙光信息产业(北京)有限公司 | Data query method and apparatus for parallel database |
| US20180074852A1 (en) * | 2016-09-14 | 2018-03-15 | Salesforce.Com, Inc. | Compact Task Deployment for Stream Processing Systems |
| CN108399231A (en) * | 2018-02-13 | 2018-08-14 | 中体彩科技发展有限公司 | A kind of collecting method and Flume data collection clients |
| CN108446391A (en) * | 2018-03-23 | 2018-08-24 | 万帮充电设备有限公司 | Processing method, device, electronic equipment and the computer-readable medium of data |
| CN108512911A (en) * | 2018-03-15 | 2018-09-07 | 成都优易数据有限公司 | A kind of distributed capture agency plant and its implementation based on Flume |
| CN108804459A (en) * | 2017-05-02 | 2018-11-13 | 杭州海康威视数字技术股份有限公司 | Data query method and device |
-
2018
- 2018-11-14 CN CN201811352375.9A patent/CN109460412A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446194A (en) * | 2010-10-14 | 2012-05-09 | 金蝶软件（中国）有限公司 | Method and device for grouping and checking product data |
| CN103838574A (en) * | 2014-02-20 | 2014-06-04 | 浪潮集团山东通用软件有限公司 | General method for grouped summarizing of data tables |
| CN104199831A (en) * | 2014-07-31 | 2014-12-10 | 深圳市腾讯计算机系统有限公司 | Information processing method and device |
| CN104408179A (en) * | 2014-12-15 | 2015-03-11 | 北京国双科技有限公司 | Method and device for processing data from data table |
| CN105930379A (en) * | 2016-04-14 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Method and system for collecting log data by means of interceptor |
| CN106250519A (en) * | 2016-08-04 | 2016-12-21 | 曙光信息产业(北京)有限公司 | Data query method and apparatus for parallel database |
| US20180074852A1 (en) * | 2016-09-14 | 2018-03-15 | Salesforce.Com, Inc. | Compact Task Deployment for Stream Processing Systems |
| CN108804459A (en) * | 2017-05-02 | 2018-11-13 | 杭州海康威视数字技术股份有限公司 | Data query method and device |
| CN108399231A (en) * | 2018-02-13 | 2018-08-14 | 中体彩科技发展有限公司 | A kind of collecting method and Flume data collection clients |
| CN108512911A (en) * | 2018-03-15 | 2018-09-07 | 成都优易数据有限公司 | A kind of distributed capture agency plant and its implementation based on Flume |
| CN108446391A (en) * | 2018-03-23 | 2018-08-24 | 万帮充电设备有限公司 | Processing method, device, electronic equipment and the computer-readable medium of data |
Non-Patent Citations (1)
| Title |
|---|
| 香槟榔头: ""分组聚合函数详细讲解"", 《百度文库》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115164A (en) * | 2019-06-19 | 2020-12-22 | 北京金山云网络技术有限公司 | Data processing method and device, data query method and device, and network equipment |
| CN110334117A (en) * | 2019-07-17 | 2019-10-15 | 北京脉冲星科技有限公司 | A kind of real-time data processing method and device |
| CN112242965A (en) * | 2019-07-18 | 2021-01-19 | 特拉维夫迈络思科技有限公司 | Telemetry event aggregation |
| CN110598072A (en) * | 2019-09-24 | 2019-12-20 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN110598072B (en) * | 2019-09-24 | 2022-03-01 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN111198882A (en) * | 2019-12-26 | 2020-05-26 | 东软集团股份有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN111198882B (en) * | 2019-12-26 | 2024-07-23 | 东软集团股份有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN112131258A (en) * | 2020-09-23 | 2020-12-25 | 创新奇智(重庆)科技有限公司 | Data splicing method, device and equipment and computer storage medium |
| CN112416972A (en) * | 2020-09-25 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Real-time data stream processing method, device, equipment and readable storage medium |
| CN113821519A (en) * | 2020-11-26 | 2021-12-21 | 北京沃东天骏信息技术有限公司 | Data processing method and field-driven design architecture |
| CN113094359A (en) * | 2021-02-24 | 2021-07-09 | 华南师范大学 | Examination data preprocessing method, system, device and medium based on decision rule |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109460412A (en) | Data aggregation method, device, equipment and storage medium | |
| US11314764B2 (en) | Automated scalable contextual data collection and extraction system | |
| US11546380B2 (en) | System and method for creation and implementation of data processing workflows using a distributed computational graph | |
| CN108491267B (en) | Method and apparatus for generating information | |
| US20210385251A1 (en) | System and methods for integrating datasets and automating transformation workflows using a distributed computational graph | |
| CN110008045A (en) | Polymerization, device, equipment and the storage medium of micro services | |
| WO2021023149A1 (en) | Method and apparatus for dynamically returning message | |
| CN108985954B (en) | Method for establishing association relation of each identifier and related equipment | |
| CN109214543B (en) | Data processing method and device | |
| CN110213380A (en) | A kind of information push method, device, computer equipment and storage medium | |
| US8447857B2 (en) | Transforming HTTP requests into web services trust messages for security processing | |
| CN104408193B (en) | The processing method and processing device of data visualization model | |
| US20210209141A1 (en) | System and method for thought object selection by custom filtering and computed diversification | |
| US11756444B2 (en) | Student message monitoring using natural language processing | |
| WO2019165456A1 (en) | Automated scalable contextual data collection and extraction system | |
| CN112052152A (en) | Simulation test method and device | |
| CN113408970B (en) | User information generation method and device, electronic equipment and computer readable medium | |
| CN115600011A (en) | Educational resource pairing method, system and storage medium based on distribution algorithm | |
| US20210092159A1 (en) | System for the prioritization and dynamic presentation of digital content | |
| CN114490718A (en) | Data output method, data output device, electronic equipment and computer readable medium | |
| CN105847877A (en) | Processing method and system for live broadcasting request | |
| CA3101794A1 (en) | System and method for thought object selection | |
| CN105245380A (en) | Message transmission mode identifying method and device | |
| CN110262856A (en) | A kind of application data acquisition method, device, terminal and storage medium | |
| CN116915870B (en) | Task creation request processing method, device, electronic equipment and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190312 |