CN109101330A - Data capture method, device and system - Google Patents
Data capture method, device and system Download PDFInfo
- Publication number
- CN109101330A CN109101330A CN201810887200.1A CN201810887200A CN109101330A CN 109101330 A CN109101330 A CN 109101330A CN 201810887200 A CN201810887200 A CN 201810887200A CN 109101330 A CN109101330 A CN 109101330A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- data acquisition
- type
- acquisition request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application discloses data capture method, device and system.The data capture method, comprising: receive the data acquisition request that transmitting terminal is sent；Data acquisition request is parsed to determine the type of received data acquisition request；The data acquisition task that the type of creation and data acquisition request matches；And data acquisition task is executed, and the processing result of task is obtained to destination feedback data.The implementation realizes the distribution of the task based on data acquisition request and scheduling, is conducive to the efficiency for promoting data acquisition.
Description
Technical field
The invention relates to fields of communication technology, and in particular to message transmission field more particularly to data recipient
Method, device and system.
Background technique
In the system for much needing to carry out multiterminal communication interaction, server-side usually can all play the angle of performer
Color.Server-side receives the data acquisition request of client, and the result obtained to client feedback data.
In this system of autonomous driving vehicle, each terminal will generate a large amount of runtime data in the process of running,
These data for example, vehicle driving state information, surrounding enviroment acquisition information and each algoritic module are (for example, pretreatment, sense
Know, position, map etc.) input and output I/O information.These data are quickly generated and are consumed during vehicle operation,
The correctness of these information be directly related to vehicle whether normal operation, each module whether work normally, arithmetic result whether just
Often etc..There is important role to the exploitation and debugging of automated driving system so how to obtain and analyze these data.
In existing automated driving system, do not have unified outbound data acquisition-authentication-distribution function, each mould still
Block is to meet respective business demand respectively to develop data processing tools.
Summary of the invention
The embodiment of the present application proposes data capture method, device and system.
In a first aspect, the embodiment of the present application provides a kind of data capture method, comprising: receive the data that transmitting terminal is sent
Acquisition request；Data acquisition request is parsed to determine the type of received data acquisition request；Creation and data acquisition request
The data acquisition task that matches of type；And data acquisition task is executed, and obtain task to destination feedback data
Processing result.
In some embodiments, the type of data acquisition request includes pushing type；The class of creation and data acquisition request
The task that type matches, comprising: the type in response to data acquisition request is to push type, creates multithreading task, wherein more
The number of threads of thread task is determined based on the quantity of the data type of data acquisition request institute request；Execution task, and
Task processing result is fed back to destination, comprising: the per thread in multithreading task is made to execute data corresponding with the thread
Acquisition task；It is performed both by completion in response to each thread in multithreading task, the processing of task is obtained to transmitting terminal feedback data
As a result.
In some embodiments, the data of data acquisition request institute request are carried out by following data storing steps
Storage: in response to receiving data storage request, the data type of data to be stored indicated by data storage request is determined；It will
Data to be stored is added to the tail of the queue of message queue being pre-created, corresponding with identified data type.
In some embodiments, data storing steps further include: in response to the quantity etc. for the data that message queue is stored
In the pre-set length of message queue, message queue head of the queue data are removed, and using the data removed as historical data
It stores into pre-set database.
In some embodiments, data acquisition task is executed, and obtains the processing result of task to destination feedback data,
It include: the storage location that requested data is determined based on the real-time of data acquisition request requested data；It is deposited from identified
Position acquisition data corresponding with data acquisition task are stored up, and obtain the processing result of task to destination feedback data.
Second aspect, the embodiment of the present application provide a kind of data-acquisition system, comprising: task scheduling layer, for receiving
The data acquisition request that transmitting terminal is sent parses data acquisition request to determine the type of received data acquisition request；Appoint
Business process layer, the data acquisition task for creating and the type of data acquisition request matches execute data acquisition task, and
The processing result of task is obtained to destination feedback data.
In some embodiments, task process layer is also used to: the type in response to data acquisition request is to push type, wound
Build multithreading task, wherein data type of the number of threads of multithreading task based on data acquisition request institute request
Quantity determines；Per thread in multithreading task executes data acquisition task corresponding with the thread；And in response to multi-thread
Each thread in journey task is performed both by completion, and the processing result of task is obtained to transmitting terminal feedback data.
In some embodiments, system further include: data distribution layer, in response to receiving data storage request, really
Determine the data type of data to be stored indicated by data storage request, and by data to be stored be added to it is being pre-created,
The tail of the queue of message queue corresponding with identified data type.
In some embodiments, data distribution layer is further used for: in response to the quantity for the data that message queue is stored
Equal to the pre-set length of message queue, message queue head of the queue data are removed, and using the data removed as history number
According to storing into pre-set database.
In some embodiments, task process layer is further used for: based on the real-time of data acquisition request requested data
Property determines the storage location of requested data；And count corresponding with data acquisition task is obtained from identified storage location
According to, and to destination feedback data obtain task processing result.
The third aspect, the embodiment of the present application provide a kind of data acquisition facility, comprising: receiving unit is configured to connect
Receive and dispatch the data acquisition request that sending end is sent；Resolution unit is configured to parse data acquisition request to determine a received number
According to the type of acquisition request；Creating unit, the data for being configured for creating and the type of data acquisition request matches obtain
Take task；And execution unit, it is configured to execute data acquisition task, and the processing of task is obtained to destination feedback data
As a result.
In some embodiments, creating unit is further configured to be promotion in response to the type of data acquisition request
Type creates multithreading task, wherein data of the number of threads of multithreading task based on data acquisition request institute request
The quantity of type determines；Execution unit is further configured to make the per thread in multithreading task to execute and the thread pair
The data acquisition task answered, and, it is performed both by completion in response to each thread in multithreading task, is obtained to transmitting terminal feedback data
Take the processing result of task.
In some embodiments, device further include: file distributing unit is configured in response to receive data storage and ask
It asks, determines the data type of data to be stored indicated by data storage request, and data to be stored is added to preparatory wound
The tail of the queue of message queue building, corresponding with identified data type.
In some embodiments, file distributing unit is further configured to: the data stored in response to message queue
Quantity be equal to the pre-set length of message queue, by message queue head of the queue data remove, and using the data removed as
History data store is into pre-set database.
In some embodiments, execution unit is further configured to: the reality based on data acquisition request requested data
When property determines the storage location of requested data；And count corresponding with data acquisition task is obtained from identified storage location
According to, and to destination feedback data obtain task processing result.
Fourth aspect, the embodiment of the present application provide a kind of electronic equipment, comprising: one or more processors；Storage dress
It sets, for storing one or more programs, when one or more programs are executed by one or more processors, so that one or more
A processor realizes the method as described in first aspect.
5th aspect, the embodiment of the present application provide a kind of computer readable storage medium, are stored thereon with computer journey
Sequence, wherein the method as described in first aspect is realized when program is executed by processor.
Data capture method provided by the embodiments of the present application, device and system obtain firstly, receiving the data that transmitting terminal is sent
Request is taken, then, parses data acquisition request to determine the type of received data acquisition request, then, creation and data
The data acquisition task that the type of acquisition request matches finally, executing data acquisition task, and is obtained to destination feedback data
The processing result for taking task realizes task distribution and scheduling based on data acquisition request, is conducive to promote data acquisition
Efficiency.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is that the data capture method of the application one embodiment can be applied to exemplary system architecture figure therein；
Fig. 2 is the flow chart according to one embodiment of the data capture method of the application；
Fig. 3 is the schematic diagram according to an application scenarios of the data capture method of the application；
Fig. 4 is the flow chart according to another embodiment of the data capture method of the application；
Fig. 5 is the structure chart according to one embodiment of the data-acquisition system of the application；
Fig. 6 is the structural schematic diagram according to one embodiment of the data acquisition facility of the application；
Fig. 7 is adapted for for realizing that the data acquisition of the embodiment of the present application changes the computer system of the electronic equipment of method
Structural schematic diagram.
Specific embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
Convenient for description, part relevant to related invention is illustrated only in attached drawing.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 is shown can be using the exemplary system of the embodiment of the data capture method or data acquisition facility of the application
System framework 100.
As shown in Figure 1, system architecture 100 may include headend equipment 101, network 102 and data processing engine 103.Net
Network 102 between headend equipment 101 and data processing engine 103 to provide the medium of network communication.Network 102 may include
Various connection types, such as wired, wireless communication link or fiber optic cables etc..
Headend equipment 101 can be the equipment that any request data processing engine 103 provides it data processing service.Phase
Ying Di, data processing engine 103 can be to received data (for example, data acquisition requests that headend equipment 101 is sent)
It is handled, and to destination feedback data processing result (for example, the desired data obtained of data acquisition request).
In addition, data processing engine 103 also has data storage function.For example, data processing engine 103 can receive outside
The operation data of portion's module acquisition, and these operation datas are stored into the memory space of data processing engine 103.Such one
Come, when receiving the data acquisition request of the transmission of headend equipment 101, data processing engine 103 can be obtained from memory space
Take corresponding data.
It should be noted that data capture method provided by the embodiment of the present application is generally executed by data processing engine,
Correspondingly, data acquisition facility is generally positioned in data processing engine.
It should be understood that the number of the headend equipment in Fig. 1 is only schematical.According to needs are realized, it can have and appoint
The headend equipment for number of anticipating.
With continued reference to Fig. 2, the process 200 of one embodiment of the data capture method according to the application is shown.The number
According to acquisition methods, comprising the following steps:
Step 201, the data acquisition request that transmitting terminal is sent is received.
In the present embodiment, the executing subject of the data capture method of the present embodiment is (for example, data engine shown in FIG. 1
103) number that transmitting terminal (for example, headend equipment 101 shown in FIG. 1) is sent can be received by wired or radio connection
According to acquisition request.It should be pointed out that above-mentioned radio connection can include but is not limited to 3G/4G connection, WiFi connection, indigo plant
Tooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection and other currently known or future open
The radio connection of hair.
In application scenes, data acquisition request can be sent to an authentication module first and reflected by transmitting terminal
Power, to judge whether the transmitting terminal has the permission for the data for obtaining data acquisition request institute request by authentication module.
If transmitting terminal has data acquisition permission, which can be forwarded to executing subject by authentication module, so as to hold
Row main body receives the data acquisition request.Correspondingly, if authentication module, which judges that transmitting terminal does not have, obtains data acquisition request institute
The permission of the data of request can then be fed back to transmitting terminal and be used to indicate the unsanctioned message of authentication, to refuse to send
The data acquisition request at end.So, the data acquisition request of the received ouch of executing subject is by authenticating and having phase
Data acquisition permission is answered, no data is efficiently avoided and obtains the acquisition of the module or electronic equipment of permission to data, thus
Improve the safety of system.
Step 202, data acquisition request is parsed to determine the type of received data acquisition request.
Herein, the type of data acquisition request can be used for the class of the desired data obtained of characterize data acquisition request
Type.
For example, data acquisition request can be divided according to the type of data to be obtained included in data acquisition request
Type.
In application scenes, if certain data acquisition request only some data of request, it is possible to by the number
According to acquisition request as the data acquisition request for pulling (pull) type.
Alternatively, in other application scenarios, if certain multiple data of data acquisition request request, it is possible to will
The data acquisition request is as the data acquisition request for pushing (push) type.
Step 203, the data acquisition task to match with the type of data acquisition request is created.
In some optional implementations, after receiving data acquisition request, executing subject can be to be received
The data acquisition request arrived creates a main thread, and is created according to the type of data acquisition request from thread.So,
It from the data storage location of executing subject local or other electronic equipments of connection can be communicated obtains data from thread
Data indicated by acquisition request.
Step 204, data acquisition task is executed, and obtains the processing result of task to destination feedback data.
Herein, executing subject from local data storage location or can communicate other electronic equipments of connection
Data indicated by data acquisition request are obtained, and the processing result of data acquisition task is fed back to destination.
It is understood that if executing subject is from local data storage location or communicates other electronics of connection
Equipment gets data indicated by data acquisition request, then the data that can be will acquire are anti-to destination as processing result
Feedback；Conversely, if executing subject is not successfully from local data storage location or communicates other electronic equipments of connection
Data indicated by data acquisition request are got, then can will be used to indicate the failed information for obtaining data as processing and tie
Fruit feeds back to destination.
In addition, herein, the destination for receiving the processing result of data acquisition task can be transmission data acquisition request
Transmitting terminal, be also possible to be different from send data acquisition request transmitting terminal other electronic equipments.Those skilled in the art
Destination corresponding to each data acquisition request can be set according to the demand of specific application scenarios, so that executing subject exists
After completing data acquisition task, the processing result of data acquisition task is sent to corresponding destination.
Data capture method provided in this embodiment, firstly, the data acquisition request that transmitting terminal is sent is received, then, solution
Analysis data acquisition request then creates the type with data acquisition request with the type for determining received data acquisition request
The data acquisition task to match finally, executing data acquisition task, and obtains the processing knot of task to destination feedback data
Fruit, when avoiding each headend equipment respectively request data engine obtaining the data needed for it, the calculation resources that may cause
Waste, realize based on data acquisition request unified task distribution and scheduling, be conducive to promoted data acquisition efficiency.
With continued reference to the schematic diagram 300 that Fig. 3, Fig. 3 are according to the application scenarios of the data capture method of the present embodiment.
The data capture method that Fig. 3 schematically shows the present embodiment is applied to a kind of application when automatic driving vehicle
Scene.
In application scenarios shown in Fig. 3, headend equipment 310 sends data acquisition request to data engine 320.
After data engine 320 receives data acquisition request (referring to shown in appended drawing reference 321), the data are obtained first
Request is taken to be parsed (referring to shown in appended drawing reference 322), so that it is determined that the type of data acquisition request.Then, data engine
The data acquisition task (referring to shown in appended drawing reference 323) to match with the type of data acquisition request can be created, and is executed
The data acquisition task (referring to shown in appended drawing reference 324) created, and finally by the processing result of data acquisition task to mesh
End 320 feed back.
It is understood that the headend equipment 310 for sending data acquisition request can with the destination 330 for receiving processing result
To be the same equipment, it is also possible to different equipment.In practical applications, data engine 320 can be according to data acquisition
The parsing result of request analysis come determine destination specifically why equipment.
In some optional implementations of the present embodiment, the executing subject number that 202 parsings obtain through the above steps
It can also be promotion (push) type according to the type of acquisition request.
In these optional implementations, task that the creation of step 203 and the type of data acquisition request match
It can be further by realizing in a manner of below:
Type in response to data acquisition request is to push type, creates multithreading task, wherein the line of multithreading task
Number of passes amount is determined based on the quantity of the data type of data acquisition request institute request.
In the application scenes of these optional implementations, for example, being held after receiving data acquisition request
Row main body can create a main thread for the received data acquisition request, and be asked according to the data acquisition of the push type
It is multiple from thread to create to seek the quantity of the data type of institute's request, is each used to obtain one of data from thread
The data of type.
In addition, herein, data type can be understood as the type of the data of request.For example, in some applied fields
Jing Zhong, in the headend equipment request a certain period, the collected ambient image of camera and radar data.So, environment map
Data as can be used as a type, and radar data then can be used as the data of another type.In these application scenarios
In, executing subject can create two from thread.
Correspondingly, in these optional implementations, the execution task of step 204, and fed back at task to destination
Reason result can be further by realizing in a manner of below:
The per thread in multithreading task is set to execute data acquisition task corresponding with the thread；And in response to more
Each thread in thread task is performed both by completion, and the processing result of task is obtained to transmitting terminal feedback data.
Still in the headend equipment request a certain period, for the collected ambient image of camera and radar data.
Executing subject can create two from thread, so that two ambient images and radar number obtained respectively in the period from thread
According to.
It so, can after being respectively completed the acquisition of ambient image and radar data in the period from thread for two
Acquired data are sent to main thread, and the two classes data are integrated by main thread and feed back to destination.
So, executing subject executes the data acquisition request of push type by creation multithreading task, can be with
Further promote the execution efficiency of data acquisition.
With further reference to Fig. 4, it illustrates the processes 400 of another embodiment of data capture method.The data acquisition
The process 400 of method, comprising the following steps:
Step 401, the data acquisition request that transmitting terminal is sent is received.
Step 402, data acquisition request is parsed to determine the type of received data acquisition request.
Step 403, the data acquisition task to match with the type of data acquisition request is created.
Step 404, data acquisition task is executed, and obtains the processing result of task to destination feedback data.
It should be noted that the step 401 of the present embodiment~step 404 can be respectively adopted in embodiment illustrated in fig. 2
The similar mode of step 201~step 204 executes, and details are not described herein.
Unlike embodiment shown in Fig. 2, in the data capture method of the present embodiment, data acquisition request is requested
The data of acquisition can be stored by following data storing steps:
Step 405, in response to receiving data storage request, data to be stored indicated by data storage request is determined
Data type.Herein, the type that the data type of data to be stored again it will be understood that as data to be stored.
Step 406, data to be stored is added to be pre-created, message corresponding with identified data type team
The tail of the queue of column.
For example, in pilotless automobile, the data of the types such as the available radar data of data engine, ambient image.
So, data engine can create message queue, so that each message queue stores the data of some type respectively.
In addition, data storing steps can further include in some optional implementations of the present embodiment: ringing
The pre-set length of message queue should be equal in the quantity for the data that message queue is stored, message queue head of the queue data are moved
It removes, and using the data removed as history data store into pre-set database.
In general, message queue carries out data access using the principle of first in, first out.When each number of message queue
When storing data according to position, if the data in message queue not being shifted or being abandoned, next data will be unable to be stored in
In the message queue.
It therefore,, can be with if each data bit of message queue stores data in these optional implementations
By the data conversion storage of message queue head of the queue into pre-set database, so that message queue still has memory space to receive down
One data to be stored.
Herein, pre-set database for example can be Redis database.Since Redis database has storage
Speed is fast but needs the characteristics of consuming memory source, in application scenes, can preset the storage of Redis database
Capacity.It, can will be in Redis database if the data volume stored in Redis database reaches the pre-set memory capacity
The data of storage are transferred to other databases (for example, MySQL database) and carry out rule storage.
In some optional implementations of the present embodiment, the execution data acquisition task of step 404, and to destination
The processing result that feedback data obtains task may further include:
The storage location that requested data is determined based on the real-time of data acquisition request requested data, from identified
Storage location obtains data corresponding with data acquisition task, and the processing result of task is obtained to destination feedback data.
Since in the method for the present embodiment, data to be stored is stored in message queue first, if message queue is filled with,
Data are successively transferred in database from the head of the queue of message queue again.Thus, in application scenes, if data acquisition is asked
Seeking the desired data obtained is real time data, then, which should be stored in corresponding message queue.At this point it is possible to logical
Access message queue is crossed, to obtain the real time data.And in other application scenarios, it is obtained if data acquisition request is desired
Data be historical data (for example, data with the historical juncture of current time interval preset period of time), then, the data from
It is transferred in database in corresponding message queue.In these application scenarios, it can be gone through by accessing database to obtain this
History data.
Figure 4, it is seen that compared with the corresponding embodiment of Fig. 2, the process of the data capture method of the present embodiment
400, the data storage procedure of executing subject is further highlighted, by safeguarding a message queue for Various types of data, so that
It, can be according to the data type indicated by data acquisition request, in corresponding message queue when receiving data acquisition request
Middle carry out data acquisition, further improves the efficiency of data acquisition.
It is shown in Figure 5, it is the schematic diagram 500 of one embodiment of the data-acquisition system of the application.
The data system of the present embodiment may include task scheduling layer 501 and task process layer 502.Task scheduling layer 501
It can be used for receiving the data acquisition request of transmitting terminal transmission, parse data acquisition request to determine received data acquisition request
Type.Task process layer 502 can be used for creating the data acquisition task to match with the type of data acquisition request, execute number
According to obtain task, and to destination feedback data obtain task processing result.
In some optional implementations, as shown in figure 5, task process layer 502 may include multiple processing nodes.?
It, can be by data acquisition request after task scheduling layer 501 receives data acquisition request in these optional implementations
It is distributed to idle processing node.
In some optional implementations, task process layer 502 can be also used for the class in response to data acquisition request
Type is to push type, creates multithreading task, wherein the number of threads of multithreading task, which is based on data acquisition request, requests to obtain
The quantity of the data type taken determines；Per thread in multithreading task executes data acquisition task corresponding with the thread；
And it is performed both by completion in response to each thread in multithreading task, the processing result of task is obtained to transmitting terminal feedback data.
In these optional implementations, if the type of data acquisition request is to push type, task process layer 502 can
With the multithreading task of creation, which may include main thread and the data with data acquisition request institute request
The quantity of type is accordingly from thread.Respectively execute the acquisition task of the data with certain data type respectively from thread, and will
Acquired data are sent to main thread.If all in multithreading task complete respective data acquisition task, master from thread
Thread can respectively will be combined from the data that thread obtains, and be fed back to destination.
In some optional implementations, as shown in figure 5, the data-acquisition system of the present embodiment can also be wrapped further
Include data distribution layer 503.Data distribution layer 503 can be used for determining data storage request in response to receiving data storage request
The data type of indicated data to be stored, and data to be stored is added to be pre-created and identified data
The tail of the queue of the corresponding message queue of type.
In these optional implementations, data distribution layer 503 can also be further used in response to message queue institute
The quantity of the data of storage is equal to the pre-set length of message queue, and message queue head of the queue data are removed, and will be removed
Data as history data store into pre-set database.
In these optional implementations, message queue, database (e.g., including but be not limited to Redis database
And/or MySQL database) it can be used as a part (appended drawing reference in such as Fig. 5 of data storage cell in data-acquisition system
Shown in 504).Alternatively, message queue and database also can be set with the communication connection of the data-acquisition system of the present embodiment
In other electronic equipments.
In some optional implementations, task process layer 502 in the data-acquisition system of the present embodiment can be with
It is further used for: determines the storage location of requested data based on the real-time of data acquisition request requested data；And from
Identified storage location obtains data corresponding with data acquisition task, and the processing of task is obtained to destination feedback data
As a result.
With further reference to Fig. 6, as the realization to method shown in above-mentioned each figure, this application provides a kind of data acquisition dresses
The one embodiment set, the Installation practice is corresponding with embodiment of the method shown in Fig. 2, which specifically can be applied to respectively
In kind electronic equipment.
As shown in fig. 6, the data acquisition facility 600 of the present embodiment includes receiving unit 601, resolution unit 602, creation list
Member 603 and execution unit 604.
Wherein:
Receiving unit 601 can be configured to receive the data acquisition request that transmitting terminal is sent.
Resolution unit 602 can be configured to parsing data acquisition request to determine the class of received data acquisition request
Type.
Creating unit 603 may be configured for creating the data acquisition times to match with the type of data acquisition request
Business.
Execution unit 604 is configured to execute data acquisition task, and the processing of task is obtained to destination feedback data
As a result.
In some optional implementations, creating unit 603 can also be further configured to: in response to data acquisition
The type of request is to push type, creates multithreading task, wherein the number of threads of multithreading task is based on data acquisition request
The quantity of the data type of institute's request determines.
In these optional implementations, execution unit 604 can be further configured to: be made every in multithreading task
A thread executes data acquisition task corresponding with the thread, and, it has been performed both by response to each thread in multithreading task
At to the processing result of transmitting terminal feedback data acquisition task.
In some optional implementations, the data acquisition facility of the present embodiment can also include file distributing unit
(not shown).
In these optional implementations, file distributing unit can be configured to ask in response to receiving data storage
It asks, determines the data type of data to be stored indicated by data storage request, and data to be stored is added to preparatory wound
The tail of the queue of message queue building, corresponding with identified data type.
In some optional implementations, file distributing unit can be also further configured to: in response to message queue
The quantity of the data stored is equal to the pre-set length of message queue, and message queue head of the queue data are removed, and will be moved
The data removed are as history data store into pre-set database.
In these optional implementations, execution unit 604 can be also further configured to: be based on data acquisition request
The real-time of requested data determines the storage location of requested data；And it obtains from identified storage location and is obtained with data
The corresponding data of task are taken, and obtain the processing result of task to destination feedback data.
Below with reference to Fig. 7, it illustrates the electronic equipments for the data capture method for being suitable for being used to realize the embodiment of the present application
Computer system 700 structural schematic diagram.Electronic equipment shown in Fig. 7 is only an example, should not be implemented to the application
The function and use scope of example bring any restrictions.
As shown in fig. 7, computer system 700 includes central processing unit (CPU) 701, it can be read-only according to being stored in
Program in memory (ROM) 702 or be loaded into the program in random access storage device (RAM) 703 from storage section 706 and
Execute various movements appropriate and processing.In RAM 703, also it is stored with system 700 and operates required various programs and data.
CPU 701, ROM 702 and RAM 703 are connected with each other by bus 704.Input/output (I/O) interface 705 is also connected to always
Line 704.
I/O interface 705 is connected to lower component: the storage section 706 including hard disk etc.；And including such as LAN card, tune
The communications portion 707 of the network interface card of modulator-demodulator etc..Communications portion 707 executes mailing address via the network of such as internet
Reason.Driver 708 is also connected to I/O interface 705 as needed.Detachable media 709, such as disk, CD, magneto-optic disk, half
Conductor memory etc. is mounted on as needed on driver 708, in order to as needed from the computer program read thereon
It is mounted into storage section 706.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed from network by communications portion 707, and/or from detachable media
709 are mounted.When the computer program is executed by central processing unit (CPU) 701, limited in execution the present processes
Above-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media or
Computer readable storage medium either the two any combination.Computer readable storage medium for example can be --- but
Be not limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.
The more specific example of computer readable storage medium can include but is not limited to: have one or more conducting wires electrical connection,
Portable computer diskette, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only deposit
Reservoir (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory
Part or above-mentioned any appropriate combination.In this application, computer readable storage medium, which can be, any include or stores
The tangible medium of program, the program can be commanded execution system, device or device use or in connection.And
In the application, computer-readable signal media may include in a base band or the data as the propagation of carrier wave a part are believed
Number, wherein carrying computer-readable program code.The data-signal of this propagation can take various forms, including but not
It is limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer
Any computer-readable medium other than readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit use
In by the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc., Huo Zheshang
Any appropriate combination stated.
The calculating of the operation for executing the application can be write with one or more programming languages or combinations thereof
Machine program code, described program design language include object oriented program language-such as Java, Smalltalk, C+
+, further include conventional procedural programming language-such as " C " language or similar programming language.Program code can
Fully to execute, partly execute on the user computer on the user computer, be executed as an independent software package,
Part executes on the remote computer or executes on a remote computer or server completely on the user computer for part.
In situations involving remote computers, remote computer can pass through the network of any kind --- including local area network (LAN)
Or wide area network (WAN)-is connected to subscriber computer, or, it may be connected to outer computer (such as utilize Internet service
Provider is connected by internet).
Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of the module, program segment or code include one or more use
The executable instruction of the logic function as defined in realizing.It should also be noted that in some implementations as replacements, being marked in box
The function of note can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are actually
It can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it to infuse
Meaning, the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart can be with holding
The dedicated hardware based system of functions or operations as defined in row is realized, or can use specialized hardware and computer instruction
Combination realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described unit also can be set in the processor, for example, can be described as: a kind of processor packet
Include receiving unit, resolution unit, creating unit and execution unit.Wherein, the title of these units not structure under certain conditions
The restriction of the pairs of unit itself, for example, receiving unit is also described as " receiving the data acquisition request that transmitting terminal is sent
Unit ".
As on the other hand, present invention also provides a kind of computer-readable medium, which be can be
Included in device described in above-described embodiment；It is also possible to individualism, and without in the supplying device.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the device, so that should
Device: the data acquisition request that transmitting terminal is sent is received；Data acquisition request is parsed to determine received data acquisition request
Type；The data acquisition task that the type of creation and data acquisition request matches；And data acquisition task is executed, and to
The processing result of destination feedback data acquisition task.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.Those skilled in the art
Member is it should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent feature
Any combination and the other technical solutions formed.Such as features described above has similar function with (but being not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (17)
1. a kind of data capture method, comprising:
Receive the data acquisition request that transmitting terminal is sent；
The data acquisition request is parsed to determine the type of received data acquisition request；
The data acquisition task that the type of creation and the data acquisition request matches；And
The data acquisition task is executed, and feeds back the processing result of the data acquisition task to destination.
2. according to the method described in claim 1, wherein, the type of the data acquisition request includes pushing type；
The task that the type of the creation and the data acquisition request matches, comprising:
Type in response to the data acquisition request is to push type, creates multithreading task, wherein the multithreading task
Number of threads based on the data the data type of acquisition request institute request quantity determine；
It is described to execute the task, and task processing result is fed back to destination, comprising:
The per thread in the multithreading task is set to execute data acquisition task corresponding with the thread；
It is performed both by completion in response to each thread in the multithreading task, feeds back the data acquisition task to the transmitting terminal
Processing result.
3. method according to claim 1 or 2, wherein the data of data acquisition request institute request are by such as
Under data storing steps stored:
In response to receiving data storage request, the data class of data to be stored indicated by the data storage request is determined
Type；
The data to be stored is added to the team of message queue being pre-created, corresponding with identified data type
Tail.
4. according to the method described in claim 3, wherein, the data storing steps further include:
It is equal to the pre-set length of the message queue in response to the quantity for the data that message queue is stored, by the message
Queue head of the queue data remove, and using the data removed as history data store into pre-set database.
5. it is described to execute the data acquisition task according to the method described in claim 4, wherein, and institute is fed back to destination
State the processing result of data acquisition task, comprising:
The real-time of acquisition request requested data determines the storage location of requested data based on the data；
Data corresponding with the data acquisition task are obtained from identified storage location, and feed back the data to destination
The processing result of acquisition task.
6. a kind of data-acquisition system, comprising:
Task scheduling layer parses the data acquisition request for receiving the data acquisition request of transmitting terminal transmission to determine
The type of received data acquisition request；
Task process layer, for creating the data acquisition task to match with the type of the data acquisition request, described in execution
Data acquisition task, and feed back to destination the processing result of the data acquisition task.
7. data-acquisition system according to claim 6, wherein the task process layer is also used to:
Type in response to the data acquisition request is to push type, creates multithreading task, wherein the multithreading task
Number of threads based on the data the data type of acquisition request institute request quantity determine；
Per thread in the multithreading task executes data acquisition task corresponding with the thread；And
It is performed both by completion in response to each thread in the multithreading task, feeds back the data acquisition task to the transmitting terminal
Processing result.
8. system according to claim 6 or 7, wherein the system also includes:
Data distribution layer, for determining indicated by the data storage request wait deposit in response to receiving data storage request
The data type of data is stored up, and the data to be stored is added to being pre-created, opposite with identified data type
The tail of the queue for the message queue answered.
9. system according to claim 8, wherein the data distribution layer is further used for:
It is equal to the pre-set length of the message queue in response to the quantity for the data that message queue is stored, by the message
Queue head of the queue data remove, and using the data removed as history data store into pre-set database.
10. system according to claim 9, wherein the task process layer is further used for:
The real-time of acquisition request requested data determines the storage location of requested data based on the data；And
Data corresponding with the data acquisition task are obtained from identified storage location, and feed back the data to destination
The processing result of acquisition task.
11. a kind of data acquisition facility, comprising:
Receiving unit is configured to receive the data acquisition request of transmitting terminal transmission；
Resolution unit is configured to parse the data acquisition request to determine the type of received data acquisition request；
Creating unit is configured for creating the data acquisition task to match with the type of the data acquisition request；With
And
Execution unit is configured to execute the data acquisition task, and the place of the data acquisition task is fed back to destination
Manage result.
12. data acquisition facility according to claim 11, in which:
The creating unit is further configured to the type in response to the data acquisition request to push type, creates more
Thread task, wherein the data class of the number of threads of multithreading task acquisition request institute request based on the data
The quantity of type determines；
The execution unit is further configured to make the per thread in the multithreading task to execute corresponding with the thread
Data acquisition task, and, it is performed both by completion in response to each thread in the multithreading task, feeds back institute to the transmitting terminal
State the processing result of data acquisition task.
13. device according to claim 11 or 12, wherein described device further include:
File distributing unit is configured in response to receive data storage request, determines indicated by the data storage request
Data to be stored data type, and by the data to be stored be added to it is being pre-created, with identified data class
The tail of the queue of the corresponding message queue of type.
14. device according to claim 13, wherein the file distributing unit is further configured to:
It is equal to the pre-set length of the message queue in response to the quantity for the data that message queue is stored, by the message
Queue head of the queue data remove, and using the data removed as history data store into pre-set database.
15. device according to claim 14, wherein the execution unit is further configured to:
The real-time of acquisition request requested data determines the storage location of requested data based on the data；And
Data corresponding with the data acquisition task are obtained from identified storage location, and feed back the data to destination
The processing result of acquisition task.
16. a kind of electronic equipment, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
Now such as method as claimed in any one of claims 1 to 5.
17. a kind of computer readable storage medium, is stored thereon with computer program, wherein described program is executed by processor
Shi Shixian method for example as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810887200.1A CN109101330A (en) | 2018-08-06 | 2018-08-06 | Data capture method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810887200.1A CN109101330A (en) | 2018-08-06 | 2018-08-06 | Data capture method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109101330A true CN109101330A (en) | 2018-12-28 |
Family
ID=64848905
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810887200.1A Pending CN109101330A (en) | 2018-08-06 | 2018-08-06 | Data capture method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109101330A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109840149A (en) * | 2019-02-14 | 2019-06-04 | 百度在线网络技术（北京）有限公司 | Method for scheduling task, device, equipment and storage medium |
| WO2020133545A1 (en) * | 2018-12-29 | 2020-07-02 | Beijing Voyager Technology Co., Ltd. | Systems and methods for data management |
| CN112685427A (en) * | 2021-01-25 | 2021-04-20 | 拉卡拉支付股份有限公司 | Data access method and device, electronic equipment and storage medium |
| CN113407527A (en) * | 2020-11-20 | 2021-09-17 | 广东美云智数科技有限公司 | Permission data acquisition method, management device and storage medium |
| WO2021223110A1 (en) * | 2020-05-06 | 2021-11-11 | 深圳市大疆创新科技有限公司 | Data processing apparatus and method, medium, computer device, and mobile device |
| EP3910505A4 (en) * | 2019-01-07 | 2022-08-17 | China Mobile Communication Co., Ltd. Research Institute | Data processing method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467412A (en) * | 2010-11-16 | 2012-05-23 | 金蝶软件（中国）有限公司 | Method, device and business system for processing operation request |
| US20150355892A1 (en) * | 2014-06-09 | 2015-12-10 | International Business Machines Corporation | Resources provisioning based on a set of discrete configurations |
| CN105791407A (en) * | 2016-03-11 | 2016-07-20 | 江苏物联网研究发展中心 | Multi-vehicle-mounted-terminal-based concurrent pipe controlled communication assembly |
| CN106506389A (en) * | 2016-10-19 | 2017-03-15 | 广州华多网络科技有限公司 | Network request asynchronous processing method and device |
| CN107038162A (en) * | 2016-02-03 | 2017-08-11 | 滴滴（中国）科技有限公司 | Real time data querying method and system based on database journal |
| CN107341056A (en) * | 2017-07-05 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of method and device of the thread distribution based on NFS |
| CN107943577A (en) * | 2016-10-12 | 2018-04-20 | 百度在线网络技术（北京）有限公司 | Method and apparatus for scheduler task |
| CN108171971A (en) * | 2017-12-18 | 2018-06-15 | 武汉烽火众智数字技术有限责任公司 | Vehicular real time monitoring method and system based on Spark Streaming |
-
2018
- 2018-08-06 CN CN201810887200.1A patent/CN109101330A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467412A (en) * | 2010-11-16 | 2012-05-23 | 金蝶软件（中国）有限公司 | Method, device and business system for processing operation request |
| US20150355892A1 (en) * | 2014-06-09 | 2015-12-10 | International Business Machines Corporation | Resources provisioning based on a set of discrete configurations |
| CN107038162A (en) * | 2016-02-03 | 2017-08-11 | 滴滴（中国）科技有限公司 | Real time data querying method and system based on database journal |
| CN105791407A (en) * | 2016-03-11 | 2016-07-20 | 江苏物联网研究发展中心 | Multi-vehicle-mounted-terminal-based concurrent pipe controlled communication assembly |
| CN107943577A (en) * | 2016-10-12 | 2018-04-20 | 百度在线网络技术（北京）有限公司 | Method and apparatus for scheduler task |
| CN106506389A (en) * | 2016-10-19 | 2017-03-15 | 广州华多网络科技有限公司 | Network request asynchronous processing method and device |
| CN107341056A (en) * | 2017-07-05 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of method and device of the thread distribution based on NFS |
| CN108171971A (en) * | 2017-12-18 | 2018-06-15 | 武汉烽火众智数字技术有限责任公司 | Vehicular real time monitoring method and system based on Spark Streaming |
Non-Patent Citations (1)
| Title |
|---|
| 通信学会CIC: "消息队列", 《百度百科》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111385340B (en) * | 2018-12-29 | 2021-09-24 | 北京航迹科技有限公司 | Data processing method and system |
| WO2020133545A1 (en) * | 2018-12-29 | 2020-07-02 | Beijing Voyager Technology Co., Ltd. | Systems and methods for data management |
| CN111385340A (en) * | 2018-12-29 | 2020-07-07 | 北京航迹科技有限公司 | Data processing method and system |
| US10899367B2 (en) | 2018-12-29 | 2021-01-26 | Beijing Voyager Technology Co., Ltd. | Systems and methods for data management |
| US11529974B2 (en) | 2018-12-29 | 2022-12-20 | Beijing Voyager Technology Co., Ltd. | Systems and methods for data management |
| EP3910505A4 (en) * | 2019-01-07 | 2022-08-17 | China Mobile Communication Co., Ltd. Research Institute | Data processing method and device |
| US11924070B2 (en) | 2019-01-07 | 2024-03-05 | China Mobile Communication Co., Ltd Research Institute | Data processing method and device |
| CN109840149A (en) * | 2019-02-14 | 2019-06-04 | 百度在线网络技术（北京）有限公司 | Method for scheduling task, device, equipment and storage medium |
| WO2021223110A1 (en) * | 2020-05-06 | 2021-11-11 | 深圳市大疆创新科技有限公司 | Data processing apparatus and method, medium, computer device, and mobile device |
| CN113407527A (en) * | 2020-11-20 | 2021-09-17 | 广东美云智数科技有限公司 | Permission data acquisition method, management device and storage medium |
| CN113407527B (en) * | 2020-11-20 | 2024-03-26 | 美云智数科技有限公司 | Authority data acquisition method, authority data management device and storage medium |
| CN112685427A (en) * | 2021-01-25 | 2021-04-20 | 拉卡拉支付股份有限公司 | Data access method and device, electronic equipment and storage medium |
| CN112685427B (en) * | 2021-01-25 | 2024-03-26 | 拉卡拉支付股份有限公司 | Data access method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109101330A (en) | Data capture method, device and system | |
| US11188380B2 (en) | Method and apparatus for processing task in smart device | |
| CN109634598A (en) | A kind of page display method, device, equipment and storage medium | |
| CN108182111A (en) | Task scheduling system, method and apparatus | |
| EP3806013A1 (en) | Method and apparatus for carrying shelf | |
| CN112051993A (en) | State machine template generation and task processing method, device, medium and equipment | |
| CN109697118A (en) | Streaming computing task management method, device, electronic equipment and storage medium | |
| US11507419B2 (en) | Method,electronic device and computer program product for scheduling computer resources in a task processing environment | |
| CN108628881A (en) | Method of data synchronization and device | |
| CN111352711B (en) | Multi-computing engine scheduling method, device, equipment and storage medium | |
| CN110431597A (en) | System and method for optimizing online on-demand service | |
| CN109885624A (en) | Data processing method, device, computer equipment and storage medium | |
| KR102211613B1 (en) | Asynchronous processing of sequential data blocks | |
| CN109284180A (en) | A kind of method for scheduling task, device, electronic equipment and storage medium | |
| CN109783197A (en) | Dispatching method and device for program runtime environment | |
| CN113886485A (en) | Data processing method, device, electronic equipment, system and storage medium | |
| CN109033312A (en) | Method and apparatus for obtaining information | |
| CN110515741A (en) | A kind of degradation processing method and device based on local task queue | |
| CN108021462B (en) | Method and apparatus for calling cloud service | |
| CN114661752A (en) | Method, device and system for scheduling plan of distributed database | |
| CN109978380A (en) | A kind of workflow processing method and device | |
| CN109343856A (en) | The generation method and device of custom algorithm component | |
| CN109978431A (en) | Dispatching method, device, server and the storage medium of logistics vehicles | |
| US8781798B2 (en) | Systems and methods for exploring and utilizing solutions to cyber-physical issues in a sandbox | |
| CN116821187A (en) | Database-based data processing method and device, medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181228 |
|
| RJ01 | Rejection of invention patent application after publication |