CN108182233A - A kind of distributed data abstracting method, device, computer equipment and storage medium - Google Patents
A kind of distributed data abstracting method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN108182233A CN108182233A CN201711445062.3A CN201711445062A CN108182233A CN 108182233 A CN108182233 A CN 108182233A CN 201711445062 A CN201711445062 A CN 201711445062A CN 108182233 A CN108182233 A CN 108182233A
- Authority
- CN
- China
- Prior art keywords
- data
- gathered
- storage
- receiver
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The embodiment of the invention discloses a kind of distributed data abstracting method, device, computer equipment and storage mediums.Wherein method includes：At least two independent data extraction modules obtain the data of at least two data sources, wherein, the data extraction module matches with corresponding data source；The data extraction module converts the data to preset format, and sets Data Identification to data, forms gathered data；The gathered data is sent to data receiver by the data extraction module；The data receiver receives the gathered data, and according to the Data Identification by the acquired data storage to preset memory locations.The embodiment of the present invention solves the problems, such as that single data extraction module cannot carry out data pick-up to different types of data source, improves the range of data pick-up, while multiple data extraction modules synchronize data pick-up to multiple data sources, improve data pick-up efficiency.
Description
Technical field
The present embodiments relate to computer technology more particularly to a kind of distributed data abstracting method, device, computers
Equipment and storage medium.
Background technology
The clinical decision of medical industry needs to integrate the patient data of multiple platforms or multi-data source, due to data source and
The inconsistency of data structure causes different data that cannot directly apply, and needs to carry out after-treatment.
At present, it is usually to be suitable for Database Systems, and different for data structure in database table to the extraction of data
Data are only capable of carrying out data pick-up, Wu Faquan by SQL (Structured Query Language, structured query language)
Face covers medical device data, while data pick-up efficiency is low, and stability is poor.
Invention content
The present invention provides a kind of distributed data abstracting method, device, computer equipment and storage medium, to realize to more
Kind data source carries out efficient data acquisition.
In a first aspect, an embodiment of the present invention provides a kind of distributed data abstracting method, this method includes：
At least two independent data extraction modules obtain the data of at least two data sources, wherein, the data pick-up
Module matches with corresponding data source；
The data extraction module converts the data to preset format, and sets Data Identification to data, and formation is adopted
Collect data；
The gathered data is sent to data receiver by the data extraction module；
The data receiver receives the gathered data, and according to the Data Identification by the acquired data storage extremely
Preset memory locations.
Second aspect, the embodiment of the present invention additionally provide a kind of distributed data draw-out device, which includes：
At least two independent data extraction modules, for obtaining the data of at least two data sources, wherein, the data
Abstraction module matches with corresponding data source, converts the data to preset format, and set Data Identification, shape to data
Into gathered data, the gathered data is sent to data receiver；
The data receiver, for receiving the gathered data, and according to the Data Identification by the gathered data
It stores to preset memory locations.
The third aspect, the embodiment of the present invention additionally provide a kind of computer equipment, including memory, processor and are stored in
On memory and the computer program that can run on a processor, the processor realize that the present invention such as appoints when performing described program
The distributed data abstracting method that embodiment of anticipating provides.
Fourth aspect, the embodiment of the present invention additionally provide a kind of computer readable storage medium, are stored thereon with computer
Program realizes the distributed data abstracting method that any embodiment of the present invention provides when the program is executed by processor.
The embodiment of the present invention adapts to the data extraction module of different data sources by setting at least two, independently obtains
The data of at least two data sources are sent to data receiver after converting data to unified form, and data receiver is to receiving
Data according to Data Identification carry out classification storage, different types of data source cannot be carried out by solving single data extraction module
The problem of data pick-up, improves the range of data pick-up, while multiple data extraction modules synchronize number to multiple data sources
According to extraction, data pick-up efficiency is improved, data extraction module is mutual indepedent, complementation interference, improves the stabilization of data pick-up
Property.
Description of the drawings
Fig. 1 is a kind of flow chart for distributed data abstracting method that the embodiment of the present invention one provides；
Fig. 2 is a kind of structure diagram of distributed data draw-out device provided by Embodiment 2 of the present invention；
Fig. 3 is the structure diagram for the computer equipment that the embodiment of the present invention three provides.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limitation of the invention.It also should be noted that in order to just
Part related to the present invention rather than entire infrastructure are illustrated only in description, attached drawing.
Embodiment one
Fig. 1 is the flow chart of a kind of distributed data abstracting method that the embodiment of the present invention one provides, and the present embodiment can fit
For multiple data sources to be carried out with the situation of distributed data acquisition, it is particularly suitable in medical industry Clinical Decision Support Systems
The distributed data acquisition of multi-data source, this method can be held by distributed data draw-out device provided in an embodiment of the present invention
Row, which can be used hapalonychia and/or the mode of hardware is realized.This method specifically includes：
S110, at least two independent data extraction modules obtain the data of at least two data sources, wherein, data pick-up
Module matches with corresponding data source.
Wherein, the equipment that data source refers to being capable of providing data can be the server for storing data illustratively,
It can also be the hardware device for being capable of detection data, such as lung ventilator, blood can be but not limited in medical industry hardware device
Pressure meter and patient monitor etc..Data extraction module is used to carry out data pick-up to data source.Illustratively, if data source is service
Device, then data extraction module can be that data pick-up is carried out in a manner that transmission data extracts request, and wherein data pick-up please
It for example can be HTTP (Hyper Text Transport Protocol, hypertext transfer protocol) message form to ask.If data
Source is hardware device, then data extraction module can be connect by the form of connecting line and connectivity port with hardware device, and
Carry out data pick-up.
In the present embodiment, each data extraction module adapts to a kind of data source, and each data extraction module it
Between independently of each other, the data pick-up simultaneously to a variety of different data sources progress rapidly and efficiently can be realized by way of distribution,
The type for the data source that can extract data is expanded, improves data pick-up efficiency.Meanwhile between each data extraction module mutually
It is independent, when arbitrary data abstraction module occurs abnormal, other data extraction modules will not be had an impact, improve data pumping
The stability taken.
In the present embodiment, data extraction module is determined according to the number amount and type of the data source of data to be extracted, optionally,
Each data source corresponds to a data extraction module.Optionally, if there are new data source, and there is no corresponding numbers
During according to abstraction module, the data extraction module being adapted to new data source is established, it is not necessary to modify entire data pick-up systems, are applicable in
Property is strong.
It should be noted that a data extraction module is adapted to a data source, collectable forms data or most evidences, show
Example property, if data source is sphygmomanometer, corresponding data extraction module can acquire the blood pressure data of patient；If data source is prison
Instrument is protected, then the collectable data of corresponding data extraction module include but not limited to heart rate, breathing, blood oxygen and the body temperature of patient
Deng.Optionally, same item data can be acquired by one or more data sources and be obtained.Illustratively, to blood pressure data, Ke Yishi
By carrying out data pick-up acquisition to sphygmomanometer, can also be by carrying out data pick-up acquisition to monitor.
S120, data extraction module convert data to preset format, and set Data Identification to data, form acquisition number
According to.
In the present embodiment, the data extracted from different data sources are converted to same form, improve the unification of data
Property, convenient for the identification of different data, storage and subsequent data application, wherein data application can be by extracting patient
Data generation clinical decision.Illustratively, preset format can be：(data type) data, such as data source is patient monitor,
Data can be extracted and include heart rate and body temperature, correspondingly, the data of preset format can be (HR) 90, (T) 37.
In the present embodiment, Data Identification is set to data, is distinguished using to data, wherein, Data Identification can be
Related at least one of data extraction module, data acquisition time or data type, optionally, Data Identification can be one
Character string.Gathered data includes the data and Data Identification of preset format.
Gathered data is sent to data receiver by S130, data extraction module.
Wherein, data receiver is used to receive and store the data of extraction.
S140, data receiver receive gathered data, and according to Data Identification by acquired data storage to default storage position
It puts.
In the present embodiment, the data source, data type or acquisition time of gathered data are can recognize that by Data Identification, it is optional
, data receiver carries out classification storage according to the data source or data type of gathered data to data.
Optionally, data receiver further includes after gathered data is received：Data receiver is adopted according to gathered data
Collect time and data type, judge that gathered data whether there is Data duplication；If so, the gathered data repeated is deleted.
In the present embodiment, due to that can acquire same item data by different data sources, data receiver is receiving acquisition
The repeated acquisition that whether there is same data type in same acquisition time is detected after data, if so, deleting duplicated data,
Avoid the problem that repeated data causes clinical decision inaccurate.
Optionally, step S140 includes：Data receiver receives gathered data, and obtains the receiving time of gathered data,
Gathered data is prestored according to receiving time and is stored in storage queue；
Data receiver identifies the Data Identification of each gathered data, according to the storage order of gathered data in storage queue and
Data Identification is by acquired data storage to preset memory locations.
Wherein, storage queue is used to be pre-stored gathered data, wherein, gathered data is received according to data receiver
Receiving time, stored successively in storage queue, and according to the principle of first in first out, according to the storage in storage queue
Sequence stores gathered data successively to preset memory locations, realizes the asynchronous storage to gathered data, solves while to more
A data source carries out the excessive caused data stalled store of data volume during data acquisition, improves the stabilization of data pick-up system
Property.
Optionally, the data receiver is sent out after gathered data is prestored is stored in storage queue to data extraction module
Reception information is sent, the data extraction module to be prompted to carry out the extraction of next data, improves data pick-up speed.
The technical solution of the present embodiment adapts to the data extraction module of different data sources by setting at least two, solely
The data of at least two data sources are on the spot obtained, data receiver, data receiver are sent to after converting data to unified form
It holds and classification storage is carried out according to Data Identification to the data of reception, solving single data extraction module cannot be to different type number
The problem of carrying out data pick-up according to source, improves the range of data pick-up, at the same multiple data extraction modules to multiple data sources into
Row synchrodata extracts, and improves data pick-up efficiency, and data extraction module is mutual indepedent, and complementation interference improves data pumping
The stability taken.
On the basis of above-described embodiment, data receiver further includes after gathered data is received：Data receiver is examined
Whether the form for surveying gathered data meets preset format；If it is not, then deleting gathered data, and sent out to corresponding data extraction module
Send form abnormal prompt information.
In the present embodiment, in data receiver after gathered data is received, if detecting the lattice of the gathered data of reception
Formula does not meet preset format, then deletes the gathered data, and generates form abnormal prompt information, and data extraction module is according to form
The adjustment of abnormal prompt information sends form.Form abnormal prompt information is additionally operable to be prompted to user, to prompt user's logarithm
It is overhauled according to abstraction module.
Optionally, data receiver further includes after gathered data is received：Data receiver is according to the number of gathered data
Data standard range is determined according to type；If data receiver detects that gathered data outside data standard range, deletes acquisition
Data, and generate data exception prompt message.
Wherein, for the medical data of human body, there are one data standard range, data standards for the data of each type
Range refers to the normal data range of human body, illustratively, ranging from 36.2 DEG C of the data standard of human oral cavity temperature~
37.2 DEG C, adult normal heart rate range is 60-100 beats/min.
In the present embodiment, the data standard range of each data type is stored in advance in data receiver, works as data receiver
After end receives gathered data, the data type of gathered data is identified, and obtain data standard range.Whether detect the gathered data
In the range of data standard, if so, by the acquired data storage to preset memory locations, if not, it is determined that the data exception,
The gathered data is deleted, generates data exception prompt message.
It is whether abnormal according to data exception prompt message detection data abstraction module and corresponding data source, if so, carrying
Show that user is overhauled, however, it is determined that detection data abstraction module and corresponding data source are in normal operating conditions, then sentence
Whether the gathered data of breaking is to acquire in real time, if so, determining that the corresponding user's body of the data source is abnormal, generates alarm signal
Breath.
In the present embodiment, gathered data is verified by data standard range, improves the accuracy of gathered data,
Avoid misleading of the wrong data to clinical decision.
In one embodiment, after gathered data is received, whether the form for detecting gathered data accords with data receiver
Preset format is closed, if it is not, then deleting gathered data, and form abnormal prompt information is sent to corresponding data extraction module；If
It is that data standard range is then determined according to the data type of gathered data.If detecting gathered data outside data standard range,
Gathered data is then deleted, and generates data exception prompt message；If detecting, gathered data, should in the range of data standard
Acquired data storage is to preset memory locations.
Embodiment two
Fig. 2 is a kind of result schematic diagram of distributed data draw-out device provided by Embodiment 2 of the present invention, and Fig. 2 is only one
The realization method of kind distributed data draw-out device, by taking two independent data extraction modules as an example, in other embodiments, number
It is determined according to demand according to the quantity of abstraction module, which specifically includes：
At least two independent data extraction modules 210, for obtaining the data of at least two data sources, wherein, data
Abstraction module matches with corresponding data source, converts data to preset format, and set Data Identification to data, formation is adopted
Collect data, gathered data is sent to data receiver；
Data receiver 220, for receiving gathered data, and according to Data Identification by acquired data storage to default storage
Position.
Optionally, data receiver 220 includes pre-storing unit and storage unit；Wherein,
Pre-storing unit for receiving gathered data, and obtains the receiving time of gathered data, will be adopted according to receiving time
Collection data pre-storage is stored in storage queue；
Storage unit, it is suitable according to the storage of gathered data in storage queue for identifying the Data Identification of each gathered data
Sequence and Data Identification are by acquired data storage to preset memory locations.
Optionally, data receiver 220 further includes re-scheduling unit, wherein,
Re-scheduling unit, for after gathered data is received, according to the acquisition time and data type of gathered data, judging
Gathered data whether there is Data duplication, if so, the gathered data repeated is deleted.
Optionally, data receiver 220 further includes format detecting unit and the first data delete unit；
Format detecting unit, for after gathered data is received, whether the form for detecting gathered data to meet default lattice
Formula；
First data delete unit, if the form for gathered data does not meet preset format, delete gathered data, and
Form abnormal prompt information is sent to corresponding data extraction module.
Optionally, data receiver 220 further includes data area determination unit and the second data delete unit；
Data area determination unit, for after gathered data is received, determining to count according to the data type of gathered data
According to critical field；
Second data delete unit, if for detecting that gathered data outside data standard range, deletes gathered data,
And generate data exception prompt message.
Distributed data draw-out device provided in an embodiment of the present invention can perform point that any embodiment of the present invention is provided
Cloth data pick-up method has and performs the corresponding function module of distributed data abstracting method and advantageous effect.
Embodiment three
Fig. 3 is the structure diagram of a kind of computing device that the embodiment of the present invention three provides.Fig. 3 shows real suitable for being used for
The block diagram of the exemplary computer device 12 of existing embodiment of the present invention.The computing device 12 that Fig. 3 is shown is only an example, no
The function and use scope for coping with the embodiment of the present invention bring any restrictions.
As shown in figure 3, computing device 12 may include the electronic equipment with calculation processing power, type may include but not
It is limited to terminal device and server device, wherein terminal device is such as can be mobile terminal, PC machine, and server device is for example
Can be server or computer cluster etc..The component of computing device 12 can include but is not limited to：One or more processing
Device or processing unit 16, system storage 28, connection different system component (including system storage 28 and processing unit 16)
Bus 18.
Bus 18 represents one or more in a few class bus structures, including memory bus or Memory Controller,
Peripheral bus, graphics acceleration port, processor or the local bus using the arbitrary bus structures in a variety of bus structures.It lifts
For example, these architectures include but not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)
Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Computing device 12 typically comprises a variety of computer system readable media.These media can any can be counted
The usable medium that equipment 12 accesses is calculated, including volatile and non-volatile medium, moveable and immovable medium.
System storage 28 can include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (RAM) 30 and/or cache memory 32.Computing device 12 may further include other removable/not removable
Dynamic, volatile/non-volatile computer system storage medium.Only as an example, storage system 34 can be used for read-write can not
Mobile, non-volatile magnetic media (Fig. 3 do not show, commonly referred to as " hard disk drive ").Although being not shown in Fig. 3, Ke Yiti
For for moving the disc driver of non-volatile magnetic disk (such as " floppy disk ") read-write and to moving non-volatile light
The CD drive of disk (such as CD-ROM, DVD-ROM or other optical mediums) read-write.In these cases, each driver
It can be connected by one or more data media interfaces with bus 18.Memory 28 can include at least one program and produce
Product, the program product have one group of (for example, at least one) program module, these program modules are configured to perform of the invention each
The function of embodiment.
Program/utility 40 with one group of (at least one) program module 42 can be stored in such as memory 28
In, such program module 42 includes --- but being not limited to --- operating system, one or more application program, other programs
Module and program data may include the realization of network environment in each or certain combination in these examples.Program mould
Block 42 usually performs function and/or method in embodiment described in the invention.
Computing device 12 can also be with one or more external equipments 14 (such as keyboard, sensing equipment, display 24 etc.)
Communication can also enable a user to the equipment interacted with the computing device 12 communication and/or with causing the meter with one or more
Calculate any equipment (such as network interface card, modem etc.) that equipment 12 can communicate with one or more of the other computing device
Communication.This communication can be carried out by input/output (I/O) interface 22.Also, computing device 12 can also be fitted by network
Orchestration 20 and one or more network (such as LAN (LAN), wide area network (WAN) and/or public network, such as internet)
Communication.As shown in the figure, network adapter 20 is communicated by bus 18 with other modules of computing device 12.Although it should be understood that
It is not shown in figure, computing device 12 can be combined and use other hardware and/or software module, including but not limited to：Microcode is set
Standby driver, redundant processing unit, external disk drive array, RAID system, tape drive and data backup storage system
System etc..
Processing unit 16 may include but be not limited to central processing unit (CPU) and/or image processor (GPU), pass through fortune
Row is stored in the program in system storage 28, so as to perform various functions application and data processing, such as realize the application
Any distributed data abstracting method that inventive embodiments provide：At least two independent data extraction modules obtain at least two
The data of a data source, wherein, the data extraction module matches with corresponding data source；The data extraction module is by institute
It states data and is converted to preset format, and Data Identification is set to data, form gathered data；The data extraction module is by described in
Gathered data is sent to data receiver；The data receiver receives the gathered data, and will according to the Data Identification
The acquired data storage is to preset memory locations.
Example IV
The embodiment of the present invention four provides a kind of computer readable storage medium, is stored thereon with computer program, the journey
The distributed data abstracting method provided such as the present application embodiment is provided when sequence is executed by processor：At least two is independent
Data extraction module obtains the data of at least two data sources, wherein, the data extraction module and corresponding data source phase
Match；The data extraction module converts the data to preset format, and sets Data Identification to data, forms acquisition number
According to；The gathered data is sent to data receiver by the data extraction module；The data receiver receives the acquisition
Data, and according to the Data Identification by the acquired data storage to preset memory locations.
The arbitrary combination of one or more computer-readable media may be used.Computer-readable medium can be calculated
Machine readable signal medium or computer readable storage medium.Computer readable storage medium for example can be --- but it is unlimited
In --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device or it is arbitrary more than combination.It calculates
The more specific example (non exhaustive list) of machine readable storage medium storing program for executing includes：Electrical connection with one or more conducting wires, just
It takes formula computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable type and may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In this document, computer readable storage medium can any include or store journey
The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.
Computer-readable signal media can include in a base band or as a carrier wave part propagation data-signal,
Wherein carry computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including --- but
It is not limited to --- electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be
Any computer-readable medium other than computer readable storage medium, which can send, propagate or
Transmission for by instruction execution system, device either device use or program in connection.
The program code included on computer-readable medium can be transmitted with any appropriate medium, including --- but it is unlimited
In --- wireless, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
It can write to perform the computer that operates of the present invention with one or more programming language or combinations
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
Further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
It fully performs, partly perform on the user computer on the user computer, the software package independent as one performs, portion
Divide and partly perform or perform on a remote computer or server completely on the remote computer on the user computer.
Be related in the situation of remote computer, remote computer can pass through the network of any kind --- including LAN (LAN) or
Wide area network (WAN)-be connected to subscriber computer or, it may be connected to outer computer (such as is carried using Internet service
Pass through Internet connection for quotient).
Note that it above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that
The present invention is not limited to specific embodiment described here, can carry out for a person skilled in the art various apparent variations,
It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above example to the present invention
It is described in further detail, but the present invention is not limited only to above example, without departing from the inventive concept, also
It can include other more equivalent embodiments, and the scope of the present invention is determined by scope of the appended claims.
Claims (10)
1. a kind of distributed data abstracting method, which is characterized in that including：
At least two independent data extraction modules obtain the data of at least two data sources, wherein, the data extraction module
Match with corresponding data source；
The data extraction module converts the data to preset format, and sets Data Identification to the data, and formation is adopted
Collect data；
The gathered data is sent to data receiver by the data extraction module；
The data receiver receives the gathered data, and according to the Data Identification by the acquired data storage to default
Storage location.
2. according to the method described in claim 1, it is characterized in that, the data receiver receives the gathered data, and root
According to the Data Identification by the acquired data storage to preset memory locations, including：
The data receiver receives the gathered data, and obtains the receiving time of the gathered data, according to the reception
The gathered data is prestored and is stored in storage queue by the time；
The data receiver identifies the Data Identification of each gathered data, suitable according to the storage of gathered data in the storage queue
Sequence and the Data Identification are by the acquired data storage to preset memory locations.
3. according to the method described in claim 2, it is characterized in that, the data receiver receive the gathered data it
Afterwards, it further includes：
The data receiver judges whether the gathered data deposits according to the acquisition time and data type of the gathered data
In Data duplication；
If so, the gathered data repeated is deleted.
4. method according to claim 1 or 2, which is characterized in that the data receiver is receiving the gathered data
Later, it further includes：
Whether the form that the data receiver detects the gathered data meets preset format；
If it is not, then deleting the gathered data, and form abnormal prompt information is sent to corresponding data extraction module.
5. method according to claim 1 or 2, which is characterized in that the data receiver is receiving the gathered data
Later, it further includes：
The data receiver determines data standard range according to the data type of the gathered data；
If the data receiver detects that the gathered data outside the data standard range, deletes the acquisition number
According to, and generate data exception prompt message.
6. a kind of distributed data draw-out device, which is characterized in that including：
At least two independent data extraction modules, for obtaining the data of at least two data sources, wherein, the data pick-up
Module matches with corresponding data source, converts the data to preset format, and set Data Identification to data, formation is adopted
Collect data, the gathered data is sent to data receiver；
The data receiver, for receiving the gathered data, and according to the Data Identification by the acquired data storage
To preset memory locations.
7. device according to claim 6, which is characterized in that the data receiver includes pre-storing unit and storage is single
Member；Wherein,
The pre-storing unit for receiving the gathered data, and obtains the receiving time of the gathered data, according to described
The gathered data is prestored and is stored in storage queue by receiving time；
The storage unit for identifying the Data Identification of each gathered data, is deposited according to gathered data in the storage queue
Storage sequence and the Data Identification are by the acquired data storage to preset memory locations.
8. device according to claim 7, which is characterized in that the data receiver further includes re-scheduling unit, wherein,
The re-scheduling unit, for after the gathered data is received, according to the acquisition time and data of the gathered data
Type judges the gathered data with the presence or absence of Data duplication, if so, the gathered data repeated is deleted.
9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor
Calculation machine program, which is characterized in that the processor realizes the side as described in any in claim 1-5 when performing described program
Method.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The method as described in any in claim 1-5 is realized during execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711445062.3A CN108182233A (en) | 2017-12-27 | 2017-12-27 | A kind of distributed data abstracting method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711445062.3A CN108182233A (en) | 2017-12-27 | 2017-12-27 | A kind of distributed data abstracting method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108182233A true CN108182233A (en) | 2018-06-19 |
Family
ID=62547696
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711445062.3A Pending CN108182233A (en) | 2017-12-27 | 2017-12-27 | A kind of distributed data abstracting method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108182233A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109063063A (en) * | 2018-07-20 | 2018-12-21 | 泰华智慧产业集团股份有限公司 | Data processing method and device based on multi-source data |
| CN109657167A (en) * | 2018-11-29 | 2019-04-19 | 彩讯科技股份有限公司 | Collecting method, device, server and storage medium |
| CN109684393A (en) * | 2018-12-11 | 2019-04-26 | 中科恒运股份有限公司 | Collecting method, computer readable storage medium and terminal device |
| CN110598072A (en) * | 2019-09-24 | 2019-12-20 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN110716938A (en) * | 2019-10-15 | 2020-01-21 | 北京明略软件系统有限公司 | Data aggregation method and device, storage medium and electronic device |
| CN110750563A (en) * | 2018-07-20 | 2020-02-04 | 北京京东尚科信息技术有限公司 | Multi-model data processing method, system, device, electronic equipment and storage medium |
| CN110928681A (en) * | 2019-11-11 | 2020-03-27 | 北京明略软件系统有限公司 | Data processing method and device, storage medium and electronic device |
| CN111680108A (en) * | 2019-03-11 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN112395333A (en) * | 2020-11-20 | 2021-02-23 | 北京百度网讯科技有限公司 | Method and device for checking data exception, electronic equipment and storage medium |
| CN114791840A (en) * | 2021-01-26 | 2022-07-26 | 武汉斗鱼网络科技有限公司 | Data assembling method and device, electronic equipment and medium |
| CN116582772A (en) * | 2023-03-29 | 2023-08-11 | 四川辰鳗科技有限公司 | Electric energy data acquisition method, system, electronic equipment and medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049538A (en) * | 2012-12-25 | 2013-04-17 | 华中科技大学 | Method and system for activity information aggregation searching and interaction based on location services |
| CN103186652A (en) * | 2011-12-28 | 2013-07-03 | 英业达股份有限公司 | Distributed data de-duplication system and method thereof |
| US20140207728A1 (en) * | 2011-05-24 | 2014-07-24 | Red Lambda, Inc. | Systems for Storing Data Streams in a Distributed Environment |
| CN104090896A (en) * | 2013-12-19 | 2014-10-08 | 深圳市腾讯计算机系统有限公司 | Method, device and system for importing data |
| CN104504006A (en) * | 2014-12-11 | 2015-04-08 | 厦门市美亚柏科信息股份有限公司 | Method and system for acquiring and analyzing data on news client |
| CN104866528A (en) * | 2015-04-24 | 2015-08-26 | 广东小天才科技有限公司 | Multi-platform data acquisition method and system |
| CN106372185A (en) * | 2016-08-31 | 2017-02-01 | 广东京奥信息科技有限公司 | Data preprocessing method for heterogeneous data sources |
| CN106446091A (en) * | 2016-09-13 | 2017-02-22 | 北京协力筑成金融信息服务股份有限公司 | Preprocessing method and device for multi-source time series data |
| CN106503274A (en) * | 2016-12-22 | 2017-03-15 | 北京览群智数据科技有限责任公司 | A kind of Data Integration and searching method and server |
| CN106649720A (en) * | 2016-12-22 | 2017-05-10 | 北京览群智数据科技有限责任公司 | Data processing method and server |
| CN106682099A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Data storage method and device |
| CN106846065A (en) * | 2017-02-07 | 2017-06-13 | 咪咕互动娱乐有限公司 | A kind of data processing method and device |
| CN106933913A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | Data processing method and device |
| CN107247799A (en) * | 2017-06-27 | 2017-10-13 | 北京天机数测数据科技有限公司 | Data processing method, system and its modeling method of compatible a variety of big data storages |
| CN107451223A (en) * | 2017-07-17 | 2017-12-08 | 广州特道信息科技有限公司 | The big data acquisition system and method for a kind of high concurrent parallel computation |
-
2017
- 2017-12-27 CN CN201711445062.3A patent/CN108182233A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140207728A1 (en) * | 2011-05-24 | 2014-07-24 | Red Lambda, Inc. | Systems for Storing Data Streams in a Distributed Environment |
| CN103186652A (en) * | 2011-12-28 | 2013-07-03 | 英业达股份有限公司 | Distributed data de-duplication system and method thereof |
| CN103049538A (en) * | 2012-12-25 | 2013-04-17 | 华中科技大学 | Method and system for activity information aggregation searching and interaction based on location services |
| CN104090896A (en) * | 2013-12-19 | 2014-10-08 | 深圳市腾讯计算机系统有限公司 | Method, device and system for importing data |
| CN104504006A (en) * | 2014-12-11 | 2015-04-08 | 厦门市美亚柏科信息股份有限公司 | Method and system for acquiring and analyzing data on news client |
| CN104866528A (en) * | 2015-04-24 | 2015-08-26 | 广东小天才科技有限公司 | Multi-platform data acquisition method and system |
| CN106933913A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | Data processing method and device |
| CN106372185A (en) * | 2016-08-31 | 2017-02-01 | 广东京奥信息科技有限公司 | Data preprocessing method for heterogeneous data sources |
| CN106446091A (en) * | 2016-09-13 | 2017-02-22 | 北京协力筑成金融信息服务股份有限公司 | Preprocessing method and device for multi-source time series data |
| CN106682099A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Data storage method and device |
| CN106503274A (en) * | 2016-12-22 | 2017-03-15 | 北京览群智数据科技有限责任公司 | A kind of Data Integration and searching method and server |
| CN106649720A (en) * | 2016-12-22 | 2017-05-10 | 北京览群智数据科技有限责任公司 | Data processing method and server |
| CN106846065A (en) * | 2017-02-07 | 2017-06-13 | 咪咕互动娱乐有限公司 | A kind of data processing method and device |
| CN107247799A (en) * | 2017-06-27 | 2017-10-13 | 北京天机数测数据科技有限公司 | Data processing method, system and its modeling method of compatible a variety of big data storages |
| CN107451223A (en) * | 2017-07-17 | 2017-12-08 | 广州特道信息科技有限公司 | The big data acquisition system and method for a kind of high concurrent parallel computation |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109063063A (en) * | 2018-07-20 | 2018-12-21 | 泰华智慧产业集团股份有限公司 | Data processing method and device based on multi-source data |
| CN110750563A (en) * | 2018-07-20 | 2020-02-04 | 北京京东尚科信息技术有限公司 | Multi-model data processing method, system, device, electronic equipment and storage medium |
| CN109063063B (en) * | 2018-07-20 | 2020-06-23 | 泰华智慧产业集团股份有限公司 | Data processing method and device based on multi-source data |
| CN109657167A (en) * | 2018-11-29 | 2019-04-19 | 彩讯科技股份有限公司 | Collecting method, device, server and storage medium |
| CN109657167B (en) * | 2018-11-29 | 2023-11-21 | 彩讯科技股份有限公司 | Data acquisition method, device, server and storage medium |
| CN109684393A (en) * | 2018-12-11 | 2019-04-26 | 中科恒运股份有限公司 | Collecting method, computer readable storage medium and terminal device |
| CN111680108A (en) * | 2019-03-11 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN111680108B (en) * | 2019-03-11 | 2023-11-03 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN110598072B (en) * | 2019-09-24 | 2022-03-01 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN110598072A (en) * | 2019-09-24 | 2019-12-20 | 恩亿科(北京)数据科技有限公司 | Feature data aggregation method and device |
| CN110716938A (en) * | 2019-10-15 | 2020-01-21 | 北京明略软件系统有限公司 | Data aggregation method and device, storage medium and electronic device |
| CN110928681A (en) * | 2019-11-11 | 2020-03-27 | 北京明略软件系统有限公司 | Data processing method and device, storage medium and electronic device |
| CN112395333B (en) * | 2020-11-20 | 2023-07-25 | 北京百度网讯科技有限公司 | Method, device, electronic equipment and storage medium for checking data abnormality |
| CN112395333A (en) * | 2020-11-20 | 2021-02-23 | 北京百度网讯科技有限公司 | Method and device for checking data exception, electronic equipment and storage medium |
| CN114791840A (en) * | 2021-01-26 | 2022-07-26 | 武汉斗鱼网络科技有限公司 | Data assembling method and device, electronic equipment and medium |
| CN114791840B (en) * | 2021-01-26 | 2023-09-22 | 武汉斗鱼网络科技有限公司 | Data assembling method, device, electronic equipment and medium |
| CN116582772A (en) * | 2023-03-29 | 2023-08-11 | 四川辰鳗科技有限公司 | Electric energy data acquisition method, system, electronic equipment and medium |
| CN116582772B (en) * | 2023-03-29 | 2024-01-16 | 四川辰鳗科技有限公司 | Electric energy data acquisition method, system, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108182233A (en) | A kind of distributed data abstracting method, device, computer equipment and storage medium | |
| CN108833131A (en) | System, method, equipment and the computer storage medium of distributed data base cloud service | |
| US10818013B2 (en) | Systems and methods for processing data extracted from frames captured from video signals | |
| CN109887234B (en) | Method and device for preventing children from getting lost, electronic equipment and storage medium | |
| CN110363067A (en) | Auth method and device, electronic equipment and storage medium | |
| CN110163257A (en) | Method, apparatus, equipment and the computer storage medium of drawing-out structure information | |
| US20140121559A1 (en) | Detecting cognitive impairment indicators | |
| CN109543560A (en) | Dividing method, device, equipment and the computer storage medium of personage in a kind of video | |
| US20130211854A1 (en) | Decoding, Model and Presentation System | |
| CN107958215A (en) | A kind of antifraud recognition methods, device, server and storage medium | |
| CN109960932A (en) | File test method, device and terminal device | |
| WO2021017286A1 (en) | Facial recognition method and apparatus, electronic device and non-volatile computer readable storage medium | |
| CN109933269A (en) | Method, equipment and the computer storage medium that small routine is recommended | |
| EP4086791A1 (en) | Response device, response system, response method, and recording medium | |
| CN110377694A (en) | Text is marked to the method, apparatus, equipment and computer storage medium of logical relation | |
| CN110060441A (en) | Method and apparatus for terminal anti-theft | |
| CN109871383A (en) | A kind of data processing method, device, storage medium and electronic equipment | |
| US11741986B2 (en) | System and method for passive subject specific monitoring | |
| CN110388933A (en) | Interest point search method, device, server and storage medium | |
| WO2008054162A1 (en) | Method, apparatus, and system for diagnosing health status of mobile terminal users | |
| CN109450669A (en) | A kind of method of abnormal alarm, equipment and computer storage medium | |
| CN109933254A (en) | Show method, equipment and the computer storage medium of reading content | |
| CN110188602A (en) | Face identification method and device in video | |
| TWI687206B (en) | Method and device for marking adventitious sound | |
| CN109545319B (en) | Prescription alarm method based on knowledge relation analysis and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180619 |
|
| RJ01 | Rejection of invention patent application after publication |