CN109783512A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109783512A CN109783512A CN201811528348.2A CN201811528348A CN109783512A CN 109783512 A CN109783512 A CN 109783512A CN 201811528348 A CN201811528348 A CN 201811528348A CN 109783512 A CN109783512 A CN 109783512A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- business datum
- type
- piecemeal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000003860 storage Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 52
- 230000002123 temporal effect Effects 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 23
- 230000015654 memory Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 13
- 230000008676 import Effects 0.000 claims description 9
- 238000011161 development Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 241000208340 Araliaceae Species 0.000 description 4
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 4
- 235000003140 Panax quinquefolius Nutrition 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 4
- 238000005520 cutting process Methods 0.000 description 4
- 235000008434 ginseng Nutrition 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000006116 polymerization reaction Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application involves data processing fields, and disclose a kind of data processing method, device, computer equipment and storage medium, and wherein method includes: acquisition business datum, and the business datum is directed into Druid database；Predefined dimensional information and piecemeal temporal information are obtained, the processing of prepolymerization piecemeal is carried out to the business datum to obtain block data according to the dimensional information and piecemeal temporal information, and save the block data；If receiving data query instruction, the query argument in the data query instruction is obtained；The block data is inquired according to the query argument, and returns to corresponding query result.The method increase data query speed, by the pretreatment to data so that hundred million grades of data can be realized second grade inquiry, while improving data sheet development efficiency.
Description
Technical field
This application involves Internet technical field more particularly to a kind of data processing method, device, computer equipment and deposit
Storage media.
Background technique
Currently, internet insurance industry to the processing of the insurance data such as rule mainly using large-scale relevant database into
Row storage processing, then carries out the statistical analysis of real-time multidimensional, but the today risen suddenly and sharply in data volume, the processing side using database
Formula has existed many disadvantages.For example, the query analysis speed of database is not able to satisfy insurance in the case where big data quantity
The demand of regular traffic；Business datum report making is more professional；Insurance rule is various and complicated, and database analyzes and counts
It takes a long time, influences business delivery；Database data API is not fully suitable for multidimensional analysis, realizes that function is complex etc.
Deng.Therefore, it is necessary to provide a kind of data processing method to solve the above problems.
Summary of the invention
This application provides a kind of data processing method, device, computer equipment and storage mediums, it is intended to improve data
Inquiry velocity.
This application provides a kind of data processing methods comprising:
Business datum is obtained, the business datum is directed into Druid database；
Predefined dimensional information and piecemeal temporal information are obtained, according to the dimensional information and piecemeal temporal information to institute
It states business datum and carries out the processing of prepolymerization piecemeal to obtain block data, and save the block data；
If receiving data query instruction, the query argument in the data query instruction is obtained；
The block data is inquired according to the query argument, and returns to corresponding query result.
This application provides a kind of data processing equipments comprising:
It obtains import unit and the business datum is directed into Druid database for obtaining business datum；
Processing unit is obtained, for obtaining predefined dimensional information and piecemeal temporal information, according to the dimensional information
The processing of prepolymerization piecemeal is carried out to obtain block data to the business datum with piecemeal temporal information, and saves the block count
According to；
Parameter acquiring unit, if obtaining the inquiry ginseng in the data query instruction for receiving data query instruction
Number；
Return unit is inquired, for inquiring the block data according to the query argument, and returns to corresponding inquiry knot
Fruit.
Present invention also provides a kind of computer equipments comprising memory, processor and is stored on the memory
And the computer program that can be run on the processor, the processor realize provided by the present application when executing described program
The step of data processing method described in meaning one.
Present invention also provides a kind of computer storage mediums, wherein the computer storage medium is stored with computer journey
Sequence, the computer program make the processor execute number described in any embodiment provided by the present application when being executed by processor
The step of according to processing method.
The embodiment of the present application provides data processing method, device, computer equipment and storage medium.This method is by obtaining
Business datum is taken, the business datum is directed into Druid database；Obtain predefined dimensional information and piecemeal time letter
Breath carries out the processing of prepolymerization piecemeal to the business datum according to the dimensional information and piecemeal temporal information to obtain block count
According to, and save the block data；If receiving data query instruction, the query argument in the data query instruction is obtained；
The block data is inquired according to the query argument, and returns to corresponding query result.The method increase data query speed
Degree by the pretreatment to data so that hundred million grades of data can be realized second grade inquiry, while improving data sheet development efficiency.
Detailed description of the invention
Technical solution in ord to more clearly illustrate embodiments of the present application, below will be to needed in embodiment description
Attached drawing is briefly described, it should be apparent that, the accompanying drawings in the following description is some embodiments of the present application, general for this field
For logical technical staff, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of schematic flow diagram for data processing method that one embodiment of the application provides；
Fig. 2 is the sub-step schematic flow diagram of data processing method in Fig. 1；
Fig. 3 is a kind of schematic flow diagram for data processing method that another embodiment of the application provides；
Fig. 4 is the sub-step schematic flow diagram of data processing method in the application Fig. 3；
Fig. 5 is a kind of schematic block diagram for data processing equipment that one embodiment of the application provides；
Fig. 6 is a kind of schematic block diagram for data processing equipment that another embodiment of the application provides；
Fig. 7 is a kind of schematic block diagram for computer equipment that one embodiment of the application provides.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiment is some embodiments of the present application, instead of all the embodiments.Based on this Shen
Please in embodiment, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall in the protection scope of this application.
It should be appreciated that ought use in this specification and in the appended claims, term " includes " and "comprising" instruction
Described feature, entirety, step, operation, the presence of element and/or component, but one or more of the other feature, whole is not precluded
Body, step, operation, the presence or addition of element, component and/or its set.
It is also understood that mesh of the term used in this present specification merely for the sake of description specific embodiment
And be not intended to limit the application.As present specification and it is used in the attached claims, unless on
Other situations are hereafter clearly indicated, otherwise " one " of singular, "one" and "the" are intended to include plural form.
It will be further appreciated that the term "and/or" used in present specification and the appended claims is
Refer to any combination and all possible combinations of one or more of associated item listed, and including these combinations.
Referring to Fig. 1, Fig. 1 is a kind of schematic flow diagram for data processing method that one embodiment of the application provides.The number
It is applied in server according to processing method, as shown in Figure 1, the data processing method includes step S101~S104.
S101, business datum is obtained, the business datum is directed into Druid database.
Specifically, the business datum is insurance business data, and insurance business data may include producing danger, endowment insurance, life insurance
Danger and the corresponding related data of the insurance kinds such as vehicle insurance, the insurance business data are stored in the corresponding database of operation system, this
Database is traditional relevant database.
In one embodiment, the acquisition insurance business data, specifically includes: by distributed system from business datum pair
Business datum is obtained in the database answered.It wherein is deployed with Druid time series database in the distributed system, and will acquire
Business datum be directed into Druid database.
Wherein, Druid database is a MOLAP (Multidimensional On-Line Analysis
Processing, Multi-dimension on-line analytical process) database, framework is MMDB framework, while being also the system of a multinode.Together
When be also a memory database, the storage towards column.Simultaneously also support a variety of plug-in units, as Kafka plug-in unit, MySQL plug-in unit and
HDFS plug-in unit etc..
Wherein, Druid database is the system of a multinode, which includes synchronization node Zookeeper, maincenter section
Point Broker, real time node Realtime, history node Historical and coordinator node Coordinator.It is saved for synchronous
Point Zookeeper, each node is more or less coupled, and Zookeeper is responsible for synchronous effect wherein, each section
Put the related job that will not run business into strong one, it is only necessary to synchronous with Zookeeper.In a data writing process, there are off-line data and batch
Data.Hub node Broker is query node, externally provides REST interface, receives the inquiry from external client, and will
These inquiries are forwarded to Realtime and Historical node, take data from the two nodes, then return to node
Data are merged and return to client by Broker.Here Broker node plays the role of a forwarding and merges, and merges
Process needs defined memory, and recommended configuration memory is more relatively large.History node Historical node is non-real-time data
The place of processing storage and inquiry is carried out, Broker request is only responded.It locally looks for when inquiring data, is then deposited in depth now
It is searched in storage, Broker is returned to after finding, be not associated with other nodes.Service is provided under the management of Zookeeper,
And using Zookeeper monitoring signal loading or delete New Data Segment.
S102, predefined dimensional information and piecemeal temporal information are obtained, is believed according to the dimensional information and piecemeal time
Breath carries out the processing of prepolymerization piecemeal to the business datum to obtain block data, and saves the block data.
Wherein, when the predefined dimensional information and piecemeal temporal information are the customized dimension of user and deblocking
Between, dimensional information such as includes the information such as insurance kind, user, mechanism and channel；Piecemeal temporal information such as according to the moon, week, day or
The different cycles times such as hour carry out the corresponding temporal information of deblocking.
Wherein, the prepolymerization piecemeal processing mainly include the data that will be directed into Druid carry out polymerization duplicate removal processing and
Cutting processing is done according to the piecemeal time.
Specifically, polymerization duplicate removal is to merge removal repeated data to corresponding data according to dimensional information.According to the piecemeal time
Cutting processing is done, specifically, every data in Druid database includes a Timestamp (timestamp information), thus
Cutting can be done according to timestamp information, different data lines may be sliced into different Segment (data file space),
Data column are compressed simultaneously, Segment is the basic unit of Druid database purchase, carries out data with timestamp information
Piecemeal, the advantage of doing so is that can be to avoid whole scan when inquiry.
When being named in addition, also name format can be preset according to use to data block, for example increase the beginning using data source
Between and the end time be named, according to being inquired after name, which thereby enhance the inquiry velocity of data.
If S103, receiving data query instruction, the query argument in the data query instruction is obtained.
Specifically, judge whether that the data query for receiving terminal transmission instructs, data query instruction includes inquiry ginseng
Number, wherein the query argument includes the parameter information for query-relevant data, such as Data Identification, data time section or number
According to display type etc..If receiving corresponding data inquiry instruction, data query instruction is parsed to obtain the data
Query argument in inquiry instruction.
S104, the block data is inquired according to the query argument, and returns to corresponding query result.
Specifically, according to the parameter query piecemeal accordingly in data query instruction in Druid database
Data simultaneously obtain corresponding query result, and the query result is fed back to terminal, which includes corresponding data letter
Breath, for example can be corresponding tables of data, data form or intersection table etc., so as to user query.
In one embodiment, the query argument includes query type, and the query type includes two types, respectively
Top-N query type and Group by query type.As shown in Fig. 2, step S104 includes sub-step S104a to S104c.
Wherein, S104a, the query type in the identification query argument；S104b, according to query type and type queries
Preset corresponding relationship between rule determines the corresponding type queries rule of the query type recognized；S104c, according to determination
Type queries rule query described in block data and return to corresponding query result.
Specifically, user needs to check the substantially summary of data under certain condition, is generally inquired using Top-N, Top-N
Inquiry can reach second grade response.Response mode is the one dimension dragging of a dimension in terminal (front end applications), and server will be upper
Primary result is cached, and several dimensions are finally only inquired.Top-N inquires primary verification certificate dimension, exists when increasing dimension
Last buffered results are taken to add lower dimension in Redis, various dimensions can exponentially increase, and inquiry velocity is decreased obviously.
For example, the query type is being recognized as Top-N inquiry, presetting according between query type and type queries rule
Corresponding relationship, determine the corresponding type queries rule of the query type that recognizes, the type rule searching is using recursive
Mode, and more fine-grained caching is uniformly executed by thread pool；According to block data described in determining type queries rule query
And return to corresponding inquiry knot.For example dimension A+A1, dimension A+A2, dimension A+A1+ dimension are changed to by dimension A, dimension A+ dimension B
B+B1, can make full use of the lifting sequence of Druid in this way, and the time of cost possible multiple spot optimizes query result.
For example, the query type is being recognized for Group by inquiry, according between query type and type queries rule
Preset corresponding relationship determines the corresponding type queries rule of the query type recognized, due to the Group by query type
Corresponding is crosstab, and analysis people needs to see full dose data rather than summary data.Start to be exactly that how many dimension no matter looked into all
Assemble them into together, when more than 4-5 dimension will efficiency it is very low.Group by inquires corresponding type queries rule,
Be using multithreading, before constructed substantially according to the mode of Top-N, retain most latter two dimension and carry out Group by, A1+B1+C
Dimension has cache policy in inquiry, is cached simultaneously for small cluster using block (Block), can save network transmission in this way.
In the present embodiment, the business datum is directed into Druid data by obtaining business datum by the above method
Library；Predefined dimensional information and piecemeal temporal information are obtained, according to the dimensional information and piecemeal temporal information to the industry
Data of being engaged in carry out the processing of prepolymerization piecemeal to obtain block data, and save the block data；If receiving data query to refer to
It enables, obtains the query argument in the data query instruction；The block data is inquired according to the query argument, and returns to phase
The query result answered.The method increase data query speed, by the pretreatment to data so that hundred million grades of data can be realized
Second grade inquiry, while improving data sheet development efficiency.
Referring to Fig. 3, Fig. 3 is a kind of schematic flow diagram for data processing method that another embodiment of the application provides.It should
Data processing method is applied in server, as shown in figure 3, the data processing method includes step S201~S205.
S201, it determines default deployment rule, Druid database is deployed in distributed system according to the default deployment rule
System；
Specifically, the system of Druid database multinode, different nodes need hardware resource different, such as some section
Point is to account for very much memory, can be multiple by the node deployment, and each node does not communicate mutually, synchronous also with Zookeeper,
Information decoupling is come.Coordinator plays the part of the role of a manager, and the data payload of negative Historical node group is equal
Weighing apparatus, it is ensured that data are available, reproducible, and in " best " configuration.Pass through the metadata from My SQL reading data segment simultaneously
Which Historical node is information determine using Zookeeper to determine which data segment should be loaded in the cluster
In the presence of, and create Zookeeper entry and tell that Historical node loads and deletion New Data Segment, the node can be one
A, multiple nodes, which conducts an election, to be generated host node (Leader), mode of remaining node as backup.
Real time node Realtime is real time shooting data, is responsible for monitoring input traffic and allows it in internal Druid
System immediately obtains.If you do not need to real-time loading data can remove the node, he requests only in response to Broker will
Data return to Broker.If Realtime and Historical node returns to same data simultaneously, Broker be will be considered that
Historical node data is believable, if it is constant that data, which enter depth to store Druid default data,.The node sheet
Body meeting storing data, if it exceeds data can be passed to depth storage by a period of time window, depth storage is served data to
Historical node.
Therefore, it is necessary to according to Druid database, customer demand and business datum characteristic to Druid database carry out portion
Administration, i.e., default deployment rule.Druid database is disposed in a distributed system according to the default deployment rule.
Wherein, the default deployment rule, such as are as follows: it is desirable that Broker node is The more the better, Coordinator node two
A, other nodes of Realtime are also The more the better.Aspect of performance can also do the conversion of different performance.In terms of tuning for
Broker consumes the biggish node of memory, using 20G-30G heap memory, Historica node in addition to memory is there are also hard disk consumption,
The IO of release hard disk is removed using more memories, it is relatively small that Coordinator node consumes memory, it is only necessary to meet the requirements i.e.
Can, Historica node and the separation of Realtime node, Coordinator and Broker are separated, and Nginx is added on Broker
Tool does load balancing, which thereby enhances concurrent High Availabitity.
S202, business datum is obtained, the business datum is directed into Druid database.
Specifically, since Druid database supports that Kafka plug-in unit can be in Druid number in order to improve data processing speed
According to disposing Kafka plug-in unit in library, Kafka be a kind of low delay, height handle up, distributed distribution subscription message system, it can be with
Handle large-scale real-time stream.
Specifically, it is based on Kafka plug-in unit, step S202 includes the steps that data acquisition imports, as shown in figure 4, the data
It obtains steps for importing and specifically includes sub-step S202a and S202b.
Wherein, S202a, business datum obtained by Kafka, and the business datum is stored in Kafka message queue
In；S202b, the business datum is read from the Kafka message queue in real time and is stored in Druid database.
Specifically, the corresponding database of insurance business system is obtained by the Kafka plug-in unit being deployed in Druid database
In business datum, and the business datum that gets saves in the message queue of Kafka plug-in unit；In real time from the Kafka message
The business datum is read in queue, and the business datum is stored in Druid database.
S203, predefined dimensional information and piecemeal temporal information are obtained, is believed according to the dimensional information and piecemeal time
Breath carries out the processing of prepolymerization piecemeal to the business datum to obtain block data, and saves the block data.
Wherein, when the predefined dimensional information and piecemeal temporal information are the customized dimension of user and deblocking
Between, dimensional information such as includes the information such as insurance kind, user, mechanism and channel；Piecemeal temporal information such as according to the moon, week, day or
The different cycles times such as hour carry out the corresponding temporal information of deblocking.
Specifically, prepolymerization piecemeal processing includes mainly that the data that will be directed into Druid carry out polymerization duplicate removal processing
Cutting processing is done with according to the piecemeal time, and data are stored in Druid database by treated.
If S204, receiving data query instruction, the query argument in the data query instruction is obtained.
Specifically, judge whether that the data query for receiving terminal transmission instructs, data query instruction includes inquiry ginseng
Number, wherein the query argument includes the parameter information for query-relevant data, such as Data Identification, data time section or number
According to display type etc..If receiving corresponding data inquiry instruction, data query instruction is parsed to obtain the data
Query argument in inquiry instruction.
S205, the block data is inquired according to the query argument, and returns to corresponding query result.
Specifically, according to the parameter query piecemeal accordingly in data query instruction in Druid database
Data simultaneously obtain corresponding query result, and the query result is fed back to terminal, which includes corresponding data letter
Breath, for example can be corresponding tables of data, data form or intersection table etc., so as to user query.
Referring to Fig. 5, Fig. 5 is a kind of schematic block diagram of data processing equipment provided by the embodiments of the present application.Such as Fig. 5 institute
Show, correspond to above data processing method, the application also provides a kind of data processing equipment.The data processing equipment includes being used for
The unit of above-mentioned data processing method is executed, which can be configured in server.
Wherein, as shown in figure 5, data processing equipment 400 includes: to obtain import unit 401, obtain processing unit 402, ginseng
Number acquiring unit 403 and inquiry return unit 404.
It obtains import unit 401 and the business datum is directed into Druid database for obtaining business datum.
Processing unit 402 is obtained, for obtaining predefined dimensional information and piecemeal temporal information, is believed according to the dimension
Breath and piecemeal temporal information carry out the processing of prepolymerization piecemeal to the business datum to obtain block data, and save the piecemeal
Data.
Parameter acquiring unit 403, if obtaining the inquiry in the data query instruction for receiving data query instruction
Parameter.
Return unit 404 is inquired, for inquiring the block data according to the query argument, and returns to corresponding inquiry
As a result.
In one embodiment, the query argument includes query type；The inquiry return unit 404 includes: that type is known
Small pin for the case unit 4041, rule determine subelement 4042 and result queries subelement 4043.
Wherein, type identification subelement 4041, for identification query type in the query argument；Rule determines that son is single
Member 4042, for determining the query type recognized according to preset corresponding relationship between query type and type queries rule
Corresponding type queries rule；Result queries subelement 4043, for the piecemeal according to determining type queries rule query
Data simultaneously return to corresponding query result.
Referring to Fig. 6, Fig. 6 is a kind of schematic block diagram for data processing equipment that another embodiment of the application provides.Such as
Shown in Fig. 6, correspond to above data processing method, the application also provides a kind of data processing equipment.The data processing equipment packet
The unit for executing above-mentioned data processing method is included, which can be configured in server.
Wherein, as shown in fig. 6, data processing equipment 500 comprises determining that deployment unit 501, obtains import unit 502, obtains
Take processing unit 503, parameter acquiring unit 504 and inquiry return unit 505.
It determines deployment unit 501, is used to determine default deployment rule, it is regular by Druid data according to the default deployment
Library is deployed in distributed system；
It obtains import unit 502 and the business datum is directed into Druid database for obtaining business datum；
Specifically, import unit 502 is obtained, comprising: obtain saving subunit 5021 and read storing sub-units 5022.
Wherein, saving subunit 5021 is obtained, for obtaining business datum by Kafka, and the business datum is protected
There are in Kafka message queue；Storing sub-units 5022 are read, described in reading from the Kafka message queue in real time
Business datum is simultaneously stored in Druid database.
Processing unit 503 is obtained, for obtaining predefined dimensional information and piecemeal temporal information, is believed according to the dimension
Breath and piecemeal temporal information carry out the processing of prepolymerization piecemeal to the business datum to obtain block data, and save the piecemeal
Data.
Parameter acquiring unit 504, if obtaining the inquiry in the data query instruction for receiving data query instruction
Parameter.
Return unit 505 is inquired, for inquiring the block data according to the query argument, and returns to corresponding inquiry
As a result.
It is apparent to those skilled in the art that for convenience of description and succinctly, the number of foregoing description
According to the specific work process of processing unit and unit, can refer to corresponding processes in the foregoing method embodiment, it is no longer superfluous herein
It states.
Above-mentioned apparatus can be implemented as a kind of form of computer program, and computer program can be in meter as shown in Figure 7
It calculates and is run on machine equipment.
Referring to Fig. 7, Fig. 7 is a kind of schematic block diagram of computer equipment provided by the embodiments of the present application.The computer
Equipment 700 can be server.
Referring to Fig. 7, which includes processor 720, memory and the net connected by system bus 710
Network interface 750, wherein memory may include non-volatile memory medium 730 and built-in storage 740.
The non-volatile memory medium 730 can storage program area 731 and computer program 732.The computer program 732
It is performed, processor 720 may make to execute any one data processing method.
The processor 720 supports the operation of entire computer equipment 700 for providing calculating and control ability.
The built-in storage 740 provides environment for the operation of the computer program 732 in non-volatile memory medium 730, should
When computer program 732 is executed by processor 720, processor 720 may make to execute any one data processing method.
The network interface 750 such as sends the task dispatching of distribution for carrying out network communication.Those skilled in the art can manage
It solves, structure shown in Fig. 7, only the block diagram of part-structure relevant to application scheme, is not constituted to the application side
The restriction for the computer equipment 700 that case is applied thereon, specific computer equipment 700 may include more than as shown in the figure
Or less component, perhaps combine certain components or with different component layouts.Wherein, the processor 720 is for transporting
Row program code stored in memory, to realize following steps:
Business datum is obtained, the business datum is directed into Druid database；
Predefined dimensional information and piecemeal temporal information are obtained, according to the dimensional information and piecemeal temporal information to institute
It states business datum and carries out the processing of prepolymerization piecemeal to obtain block data, and save the block data；
If receiving data query instruction, the query argument in the data query instruction is obtained；
The block data is inquired according to the query argument, and returns to corresponding query result.
In one embodiment, the query argument includes query type；The processor 720 is stored in storage for running
Program code realization in device is described according to the query argument inquiry block data, and returns to corresponding query result
When, it is implemented as follows step:
Identify the query type in the query argument；
According to preset corresponding relationship between query type and type queries rule, determine that the query type recognized is corresponding
Type queries rule；And
According to block data described in determining type queries rule query and return to corresponding query result.
In one embodiment, the query type includes: Top-N query type and Group by query type.
In one embodiment, the processor 720 obtains described in program code realization stored in memory for running
Business datum is taken, before the business datum is directed into Druid database, also realizes following steps:
It determines default deployment rule, Druid database is deployed in distributed system according to the default deployment rule.
In one embodiment, the processor 720 obtains described in program code realization stored in memory for running
Business datum is taken, when the business datum is directed into Druid database, is implemented as follows step:
Business datum is obtained by Kafka, and the business datum is stored in Kafka message queue；
The business datum is read from the Kafka message queue in real time and is stored in Druid database.
It should be appreciated that in the embodiment of the present application, processor 720 can be central processing unit (Central
Processing Unit, CPU), which can also be other general processors, digital signal processor (Digital
Signal Processor, DSP), specific integrated circuit (Application Specific Integrated Circuit,
ASIC), ready-made programmable gate array (Field-Programmable GateArray, FPGA) or other programmable logic devices
Part, discrete gate or transistor logic, discrete hardware components etc..Wherein, general processor can be microprocessor or
The processor is also possible to any conventional processor etc..
It will be understood by those skilled in the art that 700 structure of computer equipment shown in Fig. 7 is not constituted and is set to computer
Standby 700 restriction may include perhaps combining certain components or different component cloth than illustrating more or fewer components
It sets.
Those of ordinary skill in the art will appreciate that be realize above-described embodiment method in all or part of the process, be
Relevant hardware can be instructed to complete by computer program, computer program can be stored in a storage medium, this is deposited
Storage media is computer readable storage medium.In the embodiment of the present invention, which can be stored in computer system
It in storage medium, and is executed by least one processor in the computer system, includes the reality such as above-mentioned each method with realization
Apply the process step of example.
The computer readable storage medium can be magnetic disk, CD, USB flash disk, mobile hard disk, read-only memory (ROM, Read-
Only Memory), the various media that can store program code such as magnetic or disk.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware
With the interchangeability of software, each exemplary composition and step are generally described according to function in the above description.This
A little functions are implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Specially
Industry technical staff can use different methods to achieve the described function each specific application, but this realization is not
It is considered as beyond scope of the present application.
In several embodiments provided herein, it should be understood that disclosed data processing equipment and method, it can
To realize by another way.For example, data processing equipment embodiment described above is only schematical.For example,
The division of each unit, only a kind of logical function partition, there may be another division manner in actual implementation.Such as it is multiple
Unit or assembly can be combined or can be integrated into another system, or some features can be ignored or not executed.
Step in the embodiment of the present application method can be sequentially adjusted, merged and deleted according to actual needs.
Unit in the embodiment of the present application device can be combined, divided and deleted according to actual needs.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit
It is that each unit physically exists alone, is also possible to two or more units and is integrated in one unit.It is above-mentioned integrated
Unit both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and when sold or used as an independent product,
It can store in a computer readable storage medium.Based on this understanding, the technical solution of the application substantially or
Person says that all or part of the part that contributes to existing technology or the technical solution can body in the form of software products
Reveal and, which is stored in a storage medium, including some instructions are with so that a computer is set
Standby (can be personal computer, terminal or the network equipment etc.) execute each embodiment the method for the application whole or
Part steps.
The above, the only specific embodiment of the application, but the protection scope of the application is not limited thereto, it is any
Those familiar with the art within the technical scope of the present application, can readily occur in various equivalent modifications or replace
It changes, these modifications or substitutions should all cover within the scope of protection of this application.Therefore, the protection scope of the application should be with right
It is required that protection scope subject to.
Claims (10)
1. a kind of data processing method is applied to distributed system characterized by comprising
Business datum is obtained, the business datum is directed into Druid database；
Predefined dimensional information and piecemeal temporal information are obtained, according to the dimensional information and piecemeal temporal information to the industry
Data of being engaged in carry out the processing of prepolymerization piecemeal to obtain block data, and save the block data；
If receiving data query instruction, the query argument in the data query instruction is obtained；
The block data is inquired according to the query argument, and returns to corresponding query result.
2. data processing method according to claim 1, which is characterized in that the query argument includes query type；Institute
It states and the block data is inquired according to the query argument, and return to corresponding query result, comprising:
Identify the query type in the query argument；
According to preset corresponding relationship between query type and type queries rule, the corresponding class of query type recognized is determined
Type rule searching；And
According to block data described in determining type queries rule query and return to corresponding query result.
3. data processing method according to claim 2, which is characterized in that the query type includes: Top-N inquiry class
Type and Group by query type.
4. data processing method according to claim 1, which is characterized in that in the acquisition business datum, by the industry
Business data are directed into before Druid database, further includes:
It determines default deployment rule, Druid database is deployed in distributed system according to the default deployment rule.
5. data processing method according to claim 1, which is characterized in that the acquisition business datum, by the business
Data are directed into Druid database, comprising:
Business datum is obtained by Kafka, and the business datum is stored in Kafka message queue；And
The business datum is read from the Kafka message queue in real time and is stored in Druid database.
6. a kind of data processing equipment characterized by comprising
It obtains import unit and the business datum is directed into Druid database for obtaining business datum；
Processing unit is obtained, for obtaining predefined dimensional information and piecemeal temporal information, according to the dimensional information and is divided
Block temporal information carries out the processing of prepolymerization piecemeal to the business datum to obtain block data, and saves the block data；
Parameter acquiring unit, if obtaining the query argument in the data query instruction for receiving data query instruction；
Return unit is inquired, for inquiring the block data according to the query argument, and returns to corresponding query result.
7. data processing equipment according to claim 6, which is characterized in that the query argument includes query type；Institute
State inquiry return unit, comprising:
Type identification subelement, for identification query type in the query argument；
Rule determines subelement, for determining identification according to preset corresponding relationship between query type and type queries rule
The corresponding type queries rule of the query type arrived；
Result queries subelement for the block data according to determining type queries rule query and returns to corresponding inquiry
As a result.
8. data processing equipment according to claim 6, which is characterized in that the acquisition import unit, comprising:
Saving subunit is obtained, for obtaining business datum by Kafka, and the business datum is stored in Kafka message
In queue；
Storing sub-units are read, for reading the business datum from the Kafka message queue in real time and being stored in Druid
In database.
9. a kind of computer equipment, which is characterized in that including memory, processor and be stored on the memory and can be in institute
The computer program run on processor is stated, the processor is realized when executing the computer program as in claim 1 to 5
The step of any one the method.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer journey
Sequence, the computer program make the processor execute such as claim 1 to 5 any one the method when being executed by processor
The step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811528348.2A CN109783512A (en) | 2018-12-13 | 2018-12-13 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811528348.2A CN109783512A (en) | 2018-12-13 | 2018-12-13 | Data processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109783512A true CN109783512A (en) | 2019-05-21 |
Family
ID=66496194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811528348.2A Pending CN109783512A (en) | 2018-12-13 | 2018-12-13 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109783512A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457320A (en) * | 2019-08-06 | 2019-11-15 | 深圳萨摩耶互联网金融服务有限公司 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
| CN110489418A (en) * | 2019-07-26 | 2019-11-22 | 阿里巴巴集团控股有限公司 | A kind of data aggregation method and system |
| CN111061719A (en) * | 2019-12-26 | 2020-04-24 | 广州市百果园信息技术有限公司 | Data collection method, device, equipment and storage medium |
| CN111090672A (en) * | 2019-12-18 | 2020-05-01 | 北京云迹科技有限公司 | Data optimization method and device |
| CN112835991A (en) * | 2019-11-25 | 2021-05-25 | 北京达佳互联信息技术有限公司 | System, method, device and storage medium for monitoring data |
| CN112905593A (en) * | 2021-03-04 | 2021-06-04 | 天九共享网络科技集团有限公司 | Report generation method, device, medium and electronic equipment |
| CN112988809A (en) * | 2021-02-09 | 2021-06-18 | 中国联合网络通信集团有限公司 | Data query method, device, equipment and medium based on relational database |
| CN113010542A (en) * | 2021-03-12 | 2021-06-22 | 中国平安财产保险股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113905067A (en) * | 2021-09-28 | 2022-01-07 | 湖南大学 | Intelligent network vehicle state monitoring and analyzing system and method |
| CN114281895A (en) * | 2021-12-24 | 2022-04-05 | 成都索贝数码科技股份有限公司 | Multi-data center synchronization method supporting remote pulling |
| CN114661772A (en) * | 2022-05-25 | 2022-06-24 | 深圳希施玛数据科技有限公司 | Data processing method and related device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150032775A1 (en) * | 2013-07-26 | 2015-01-29 | Metamarkets Group Inc. | Segment data visibility and management in a distributed database of time stamped records |
| CN106327324A (en) * | 2016-08-23 | 2017-01-11 | 杭州同盾科技有限公司 | Network behavior characteristic rapid calculation method and system |
| CN107038162A (en) * | 2016-02-03 | 2017-08-11 | 滴滴（中国）科技有限公司 | Real time data querying method and system based on database journal |
| CN107832913A (en) * | 2017-10-11 | 2018-03-23 | 微梦创科网络科技（中国）有限公司 | The Forecasting Methodology and system to monitoring data trend based on deep learning |
| CN107944059A (en) * | 2017-12-29 | 2018-04-20 | 深圳市中润四方信息技术有限公司西安分公司 | A kind of user behavior analysis method and system based on stream calculation |
-
2018
- 2018-12-13 CN CN201811528348.2A patent/CN109783512A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150032775A1 (en) * | 2013-07-26 | 2015-01-29 | Metamarkets Group Inc. | Segment data visibility and management in a distributed database of time stamped records |
| CN107038162A (en) * | 2016-02-03 | 2017-08-11 | 滴滴（中国）科技有限公司 | Real time data querying method and system based on database journal |
| CN106327324A (en) * | 2016-08-23 | 2017-01-11 | 杭州同盾科技有限公司 | Network behavior characteristic rapid calculation method and system |
| CN107832913A (en) * | 2017-10-11 | 2018-03-23 | 微梦创科网络科技（中国）有限公司 | The Forecasting Methodology and system to monitoring data trend based on deep learning |
| CN107944059A (en) * | 2017-12-29 | 2018-04-20 | 深圳市中润四方信息技术有限公司西安分公司 | A kind of user behavior analysis method and system based on stream calculation |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110489418A (en) * | 2019-07-26 | 2019-11-22 | 阿里巴巴集团控股有限公司 | A kind of data aggregation method and system |
| CN110489418B (en) * | 2019-07-26 | 2023-09-22 | 创新先进技术有限公司 | Data aggregation method and system |
| CN110457320A (en) * | 2019-08-06 | 2019-11-15 | 深圳萨摩耶互联网金融服务有限公司 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
| CN112835991B (en) * | 2019-11-25 | 2023-11-21 | 北京达佳互联信息技术有限公司 | System, method, device and storage medium for monitoring data |
| CN112835991A (en) * | 2019-11-25 | 2021-05-25 | 北京达佳互联信息技术有限公司 | System, method, device and storage medium for monitoring data |
| CN111090672B (en) * | 2019-12-18 | 2023-08-22 | 北京云迹科技股份有限公司 | Data optimization method and device |
| CN111090672A (en) * | 2019-12-18 | 2020-05-01 | 北京云迹科技有限公司 | Data optimization method and device |
| CN111061719B (en) * | 2019-12-26 | 2023-08-29 | 广州市百果园信息技术有限公司 | Data collection method, device, equipment and storage medium |
| CN111061719A (en) * | 2019-12-26 | 2020-04-24 | 广州市百果园信息技术有限公司 | Data collection method, device, equipment and storage medium |
| CN112988809A (en) * | 2021-02-09 | 2021-06-18 | 中国联合网络通信集团有限公司 | Data query method, device, equipment and medium based on relational database |
| CN112988809B (en) * | 2021-02-09 | 2023-10-03 | 中国联合网络通信集团有限公司 | Data query method, device, equipment and medium based on relational database |
| CN112905593A (en) * | 2021-03-04 | 2021-06-04 | 天九共享网络科技集团有限公司 | Report generation method, device, medium and electronic equipment |
| CN112905593B (en) * | 2021-03-04 | 2024-02-02 | 天九共享网络科技集团有限公司 | Report generation method, report generation device, report generation medium and electronic equipment |
| CN113010542A (en) * | 2021-03-12 | 2021-06-22 | 中国平安财产保险股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113010542B (en) * | 2021-03-12 | 2023-09-19 | 中国平安财产保险股份有限公司 | Service data processing method, device, computer equipment and storage medium |
| CN113905067A (en) * | 2021-09-28 | 2022-01-07 | 湖南大学 | Intelligent network vehicle state monitoring and analyzing system and method |
| CN114281895A (en) * | 2021-12-24 | 2022-04-05 | 成都索贝数码科技股份有限公司 | Multi-data center synchronization method supporting remote pulling |
| CN114281895B (en) * | 2021-12-24 | 2023-12-08 | 成都索贝数码科技股份有限公司 | Multi-data center synchronization method supporting remote pulling |
| CN114661772A (en) * | 2022-05-25 | 2022-06-24 | 深圳希施玛数据科技有限公司 | Data processing method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783512A (en) | Data processing method, device, computer equipment and storage medium | |
| US9053160B2 (en) | Distributed, real-time online analytical processing (OLAP) | |
| EP2702510B1 (en) | Joining tables in a mapreduce procedure | |
| US9418101B2 (en) | Query optimization | |
| US8140591B2 (en) | Enabling workflow awareness within a business process management (BPM) system | |
| CN110851465B (en) | Data query method and system | |
| US10824614B2 (en) | Custom query parameters in a database system | |
| WO2020087082A1 (en) | Trace and span sampling and analysis for instrumented software | |
| US20140324917A1 (en) | Reclamation of empty pages in database tables | |
| CN107341033A (en) | A kind of data statistical approach, device, electronic equipment and storage medium | |
| CN109388657B (en) | Data processing method, device, computer equipment and storage medium | |
| CN111881221A (en) | Method, device and equipment for customer portrait in logistics service | |
| CN109800269A (en) | Data managing method, device, computer equipment and storage medium | |
| EP2880556A1 (en) | Supporting multi-tenancy in a federated data management system | |
| US10521440B2 (en) | High performance data profiler for big data | |
| US20210120092A1 (en) | Adaptive data fetching from network storage | |
| US20180302268A1 (en) | Systems and Methods for Real Time Streaming | |
| WO2023197864A1 (en) | Method and device for generating calling topological graph | |
| CN110321364B (en) | Transaction data query method, device and terminal of credit card management system | |
| CN115114359A (en) | User data processing method and device | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN109063210A (en) | Resource object querying method, device, equipment and the storage medium of storage system | |
| AU2012277965A1 (en) | Method and system for processing data for database modification | |
| CN112052259A (en) | Data processing method, device, equipment and computer storage medium | |
| CN116089037A (en) | Asynchronous task processing realization method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |