CN111881216A - Data acquisition method and device based on shared template - Google Patents
Data acquisition method and device based on shared template Download PDFInfo
- Publication number
- CN111881216A CN111881216A CN202010767768.7A CN202010767768A CN111881216A CN 111881216 A CN111881216 A CN 111881216A CN 202010767768 A CN202010767768 A CN 202010767768A CN 111881216 A CN111881216 A CN 111881216A
- Authority
- CN
- China
- Prior art keywords
- data processing
- template
- data
- shared
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 283
- 230000004927 fusion Effects 0.000 claims abstract description 12
- 230000004044 response Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 abstract description 11
- 238000013507 mapping Methods 0.000 abstract description 9
- 238000011161 development Methods 0.000 abstract description 6
- 238000004891 communication Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 230000018109 developmental process Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data acquisition method, a data acquisition device, a data acquisition medium and electronic equipment based on a shared template. The method comprises the steps of obtaining a shared data processing template and template parameters through data processing request information, and obtaining a data processing result from a fusion type distributed database through the shared data processing template and the template parameters. The data processing of similar logic is realized by configuring the generated shared data processing template, so that a plurality of services can share the same shared data processing template, the code redundancy is reduced, the maintenance cost is reduced, and the data processing is flexible and efficient. Meanwhile, the table association of the distributed service database is realized by utilizing the fusion type distributed database, so that the problems that the table association can be realized only by developing a large number of codes and related data is required to be acquired through memory mapping in the micro service are solved. Under the condition of not influencing the performance of the distributed service database, the code development amount is reduced, the service system logic is clear, and the maintenance is easy.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a data acquisition method, apparatus, medium, and electronic device based on a shared template.
Background
Data processing functions are often required in software system development, such as: order data, transaction record data, customer consultation data, etc. The current common practice is to provide a data processing interface in the original service system, and usually adopts a read-write separation method: and calling the write library to realize data write-in and modification related operations, and calling the read library to acquire data and query related operations. For a business system with a huge number of users, if a large number of users perform read-write operations on a database in the same time period, (for example, in the same time period, business data is exported and/or data is retrieved), the pressure of the database is increased dramatically, and the performance of the system is reduced. Moreover, if each business system is provided with a similar function (for example, a business report), a large amount of redundant codes exist in the system, the system resources are occupied, the scale of the system is increased, and the system is abnormally difficult to maintain.
For the business system adopting the micro-service, the business system is divided into a plurality of micro-services and a plurality of business databases according to the business template. If the report data required by one micro service depends on the data of a plurality of service databases, the micro service cannot directly realize the table association operation in the common sense, and only can request the interface of another micro service through the interface to call the corresponding associated data and acquire the related data through memory mapping. This approach is not only slow, but also requires significant development and may result in interleaving of system code, increasing maintenance costs.
Disclosure of Invention
The present disclosure is directed to a method, an apparatus, a medium, and an electronic device for data acquisition based on a shared template, which are capable of solving at least one of the above-mentioned technical problems. The specific scheme is as follows:
according to a specific implementation manner of the present disclosure, in a first aspect, the present disclosure provides a data acquisition method based on a shared template, including:
receiving a data processing request message;
analyzing the data processing request message to acquire data processing request information;
acquiring a shared data processing template and template parameters of the shared data processing template based on the data processing request information;
generating a data processing instruction facing the fusion type distributed database according to the shared data processing template and the request data; wherein the converged distributed database comprises data of a plurality of distributed traffic databases;
and generating a processing result file comprising a data processing result based on the data processing instruction.
According to a second aspect, the present disclosure provides a data acquisition apparatus based on a shared template, including:
a receiving request message unit for receiving a data processing request message;
the analysis request message unit is used for analyzing the data processing request message and acquiring data processing request information;
the shared data processing template obtaining unit is used for obtaining a shared data processing template and template parameters of the shared data processing template based on the data processing request information;
a data processing instruction generating unit, configured to generate a data processing instruction for the fusion-oriented distributed database according to the shared data processing template and the request data; wherein the converged distributed database comprises data of a plurality of distributed traffic databases;
and the processing result file generating unit is used for generating a processing result file comprising a data processing result based on the data processing instruction.
According to a third aspect, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the shared template-based data acquisition method according to any of the first aspects.
According to a fourth aspect thereof, the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the shared template-based data acquisition method according to any one of the first aspect.
Compared with the prior art, the scheme of the embodiment of the disclosure at least has the following beneficial effects:
the method comprises the steps of obtaining a shared data processing template and template parameters through data processing request information, and obtaining a data processing result from a fusion type distributed database through the shared data processing template and the template parameters. The data processing of similar logic is realized by configuring the generated shared data processing template, so that a plurality of businesses can share the same shared data processing template, the embarrassment that one business corresponds to one set of codes is avoided, the code redundancy is reduced, the maintenance cost is reduced, and the data processing is flexible and efficient. Meanwhile, the table association of the distributed service database is realized by utilizing the fusion type distributed database, so that the problems that the table association can be realized only by developing a large number of codes and related data is required to be acquired through memory mapping in the micro service are solved. Under the condition of not influencing the performance of the distributed service database, the code development amount is reduced, so that the service system has clear logic, single responsibility and easy maintenance. And a queue mechanism is adopted to ensure that tasks in the system can be executed in sequence, so that the system slowness caused by simultaneous execution of multiple tasks is avoided. The pressure of concurrent data processing on the database is effectively relieved.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale. In the drawings:
FIG. 1 shows a flow diagram of a shared template-based data acquisition method according to an embodiment of the present disclosure;
FIG. 2 illustrates a block diagram of elements of a shared template-based data acquisition apparatus, according to an embodiment of the present disclosure;
fig. 3 shows an electronic device connection structure schematic according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
Alternative embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
The first embodiment provided by the present disclosure, that is, an embodiment of a data acquisition method based on a shared template.
The embodiments of the present disclosure are described in detail below with reference to fig. 1.
Before the server side provides the data processing service, data configuration is needed so that the data processing service can realize data processing with similar logic. The method specifically comprises the following steps:
and S100-1, configuring a shared data processing template and storing the shared data processing template in the template data set.
The shared data processing template is a processing template configured in advance based on a data processing mode, for example, a SQL instruction template configured in advance based on a relational database.
In order to enable the shared data processing template to realize data processing of similar logic, at least one template parameter is set in the shared data processing template, and the client can obtain different data processing results from the same shared data processing template through different template parameters; different data processing results can also be obtained from different shared data processing templates through the same set of request data. The configured shared data processing template is saved in the template dataset. Sharing the data processing template makes data processing flexible and efficient.
And S100-2, setting a preset file format.
The embodiment of the present disclosure stores the data processing result in the processing result file, and therefore, a preset file format needs to be configured. The preset file format comprises file type and file header format information. For example, file types, include: a text file type (i.e., txt file type), a comma separated value file type (i.e., CVS file type), or a spreadsheet file type (i.e., xlsx file type). The file header of the processing result file stores file header information based on the file header format information, and the file header information is used for analyzing data in the processing result file.
As shown in fig. 1, in step S101, a data processing request message is received.
The disclosed embodiment is based on the design idea of middleware, and a large amount of data processing with similar logic is taken as an independent service, such as a microservice.
A message refers to a way of interacting and communicating between software objects. Typically, a request message is sent by a client to a server for requesting the server to provide a service associated with the requested content.
The service provides an interface for data processing for the client, and the client sends a data processing request message to the server through the data processing interface.
Step S102, analyzing the data processing request message and acquiring the data processing request message.
Typically the request message comprises at least a request data header and a request data body. The request data body comprises data processing request information. The parsing information of the request message can be obtained from the request header, and the request data body can be obtained through the parsing information. And acquiring data processing request information according to the preset analysis format of the request data body.
Step S103, acquiring a shared data processing template and a template parameter of the shared data processing template based on the data processing request information.
The data processing request information includes information associated with a shared data processing template and request data associated with template parameters of the shared data processing template.
The shared data processing template realizes data processing of similar logic, so that a plurality of businesses can share the same shared data processing template, the embarrassment that one business corresponds to one set of codes is avoided, the code redundancy is reduced, and the maintenance cost is reduced.
Optionally, if the data processing request information includes a shared data processing template identifier and request data, the obtaining, based on the data processing request information, a shared data processing template and template parameters of the shared data processing template includes the following steps:
step S103-1, retrieving a template data set based on the shared data processing template identifier, and acquiring the shared data processing template.
And the data processing request message sent by the client comprises the shared data processing template identification and the request data. The shared data processing template identification is a unique identification.
The template data set stores pre-configured shared data processing templates, and each shared data processing template has a mapping relation with the shared data processing template identification.
And step S103-2, acquiring the template parameters based on the request data.
The data processing results of the shared data processing template are different due to different template parameters of a plurality of services. Therefore, data processing of similar logic services is realized by sharing the data processing template.
And step S104, generating a data processing instruction facing the fusion type distributed database according to the shared data processing template and the request data.
Wherein the converged distributed database includes data of a plurality of distributed traffic databases. For example, the converged distributed database is a TiDB, which can asynchronously synchronize data of multiple distributed service databases into the TiDB, thereby associating data of multiple distributed service databases to implement table.
The embodiment of the disclosure realizes the table association of the distributed service database by using the fusion distributed database, and solves the problems that the micro service needs to develop a large amount of codes to realize the table association and needs to acquire related data through memory mapping. Under the condition of not influencing the performance of the distributed service database, the code development amount is reduced, so that the service system has clear logic, single responsibility and easy maintenance.
Step S105, generating a processing result file including a data processing result based on the data processing instruction.
For a business system with a huge number of users, if a large number of users perform read-write operations on a database in the same time period, (for example, in the same time period, business data is exported and/or data is retrieved), the pressure of the database is increased dramatically, and the performance of the system is reduced. Optionally, the generating a processing result file including a data processing result based on the data processing instruction includes the following steps:
and step S105-1, generating a data processing task based on the data processing instruction, and storing the data processing task into a data processing task queue.
The queue, namely the downloading task buffer, has the characteristics of decoupling, peak elimination, pressure resistance and the like. Such as MQ queue middleware. The queue can ensure that the tasks in the queue can be executed in sequence, and the slow system caused by simultaneous execution of multiple tasks is avoided.
And step S105-2, acquiring the data processing task from the data processing task queue.
And step S105-3, executing the data processing task and generating a processing result file comprising a data processing result.
The embodiment of the disclosure adopts a queue mechanism, and effectively relieves the pressure of concurrent data processing on the database.
Optionally, the generating a processing result file including a data processing result includes the following steps:
and step S105-3-1, generating the processing result file with a preset file format.
The preset file format comprises file type and file header format information. For example, file types, include: a text file type (i.e., txt file type), a comma separated value file type (i.e., CVS file type), or a spreadsheet file type (i.e., xlsx file type). The file header of the processing result file stores file header information based on the file header format information, and the file header information is used for analyzing data in the processing result file.
And step S106, storing the processing result file into a distributed file system.
Distributed file systems refer to file systems that manage physical storage resources that are not necessarily directly connected to local nodes, but rather are connected to network nodes through a computer network. For example, FastDFS, is an open source lightweight distributed file system that includes the functions of: file storage, file synchronization, file access (file upload, file download), etc.
Step S107, obtaining the access path of the processing result file in the distributed file system.
The access path is a path for the client to obtain the processing result file from the distributed file system.
And step S108, generating and returning a data processing response message based on the access path.
The response message is a result message generated on demand of the request message. Usually, a client sends a request message, a server processes data according to the request of the request message, and returns the processing result to the client in the form of a response message.
For a business system with huge user quantity, in order to transmit a data processing response message to a client terminal which makes a data processing request message. The data processing request information comprises a unique identifier of a request terminal. The requesting terminal unique identification may be information associated with the client communication information. For example, the requesting terminal unique identification is the IP address of the client that sent the data processing request message.
The unique identifier of the requesting terminal can also be a unique code which has a mapping relation with the communication information of the client terminal. The unique code and the client communication information are configured in advance and stored in the server, and the client communication information is obtained through retrieval when needed.
Therefore, the generating a data processing response message based on the access path includes the following steps:
and step S107-1, generating a data processing response message based on the access path and the unique identifier of the request terminal.
The response message includes at least a response header and a response body. For example, if the unique identifier of the request terminal is an IP address, the IP address is added to the corresponding position of the response data header; and the access path is added into the response data body according to a preset format.
For a large business system, the time for one data processing may be relatively long. Optionally, after the data processing response message is returned, the method further includes the following steps:
and S108, acquiring the user social contact information of the corresponding terminal based on the unique identifier of the request terminal.
The user social contact information is the contact information of the client user who sends the data processing request message. Such as a user's email, WeChat, and/or staple message. The user social contact information and the unique identifier of the request terminal have a mapping relation. The user social contact information and the unique identifier of the request terminal are configured in advance and stored in the server, and the user social contact information is obtained through retrieval of the unique identifier of the request terminal.
And step S109, sending prompt information of data processing completion to the user based on the user social contact information.
The embodiment of the disclosure acquires the shared data processing template and the template parameter through the data processing request information, and acquires the data processing result from the fusion type distributed database through the shared data processing template and the template parameter. The data processing of similar logic is realized by configuring the generated shared data processing template, so that a plurality of businesses can share the same shared data processing template, the embarrassment that one business corresponds to one set of codes is avoided, the code redundancy is reduced, the maintenance cost is reduced, and the data processing is flexible and efficient. Meanwhile, the table association of the distributed service database is realized by utilizing the fusion type distributed database, so that the problems that the table association can be realized only by developing a large number of codes and related data is required to be acquired through memory mapping in the micro service are solved. Under the condition of not influencing the performance of the distributed service database, the code development amount is reduced, so that the service system has clear logic, single responsibility and easy maintenance. And a queue mechanism is adopted to ensure that tasks in the system can be executed in sequence, so that the system slowness caused by simultaneous execution of multiple tasks is avoided. The pressure of concurrent data processing on the database is effectively relieved.
Corresponding to the first embodiment provided by the present disclosure, the present disclosure also provides a second embodiment, that is, a data acquisition apparatus based on a shared template. Since the second embodiment is basically similar to the first embodiment, the description is simple, and the relevant portions should be referred to the corresponding description of the first embodiment. The device embodiments described below are merely illustrative.
Fig. 2 shows an embodiment of a data acquisition device based on a shared template provided by the present disclosure.
As shown in fig. 2, the present disclosure provides a data acquisition apparatus 2 based on a shared template, including:
a receive request message unit 201, configured to receive a data processing request message;
an analysis request message unit 202, configured to analyze the data processing request message, and obtain data processing request information;
an acquire shared data processing template unit 203, configured to acquire a shared data processing template and template parameters of the shared data processing template based on the data processing request information;
a data processing instruction generating unit 204, configured to generate a data processing instruction for the fusion-oriented distributed database according to the shared data processing template and the request data; wherein the converged distributed database comprises data of a plurality of distributed traffic databases;
a generate processing result file unit 205 for generating a processing result file including a data processing result based on the data processing instruction.
Optionally, the data processing request information includes a shared data processing template identifier and request data;
in the obtain shared data processing template unit 203, the method includes:
an obtain shared data processing template subunit configured to retrieve a template dataset based on the shared data processing template identifier, and obtain the shared data processing template;
and the template parameter acquiring subunit is used for acquiring the template parameters based on the request data.
Optionally, the generating processing result file unit 205 includes:
a store-in-task queue subunit, configured to generate a data processing task based on the data processing instruction, and store the data processing task in a data processing task queue;
the task acquiring subunit is used for acquiring the data processing task from the data processing task queue;
and the file generation subunit is used for executing the data processing task and generating a processing result file comprising a data processing result.
Optionally, the generating file subunit includes:
and the preset format file generating subunit is used for generating the processing result file in the preset file format.
Optionally, the data acquiring apparatus 2 further includes:
a file saving unit, configured to save the processing result file in a distributed file system;
an access path acquiring unit, configured to acquire an access path of the processing result file in the distributed file system;
and the return response message unit is used for generating and returning a data processing response message based on the access path.
Optionally, the data processing request information includes a unique identifier of the requesting terminal;
in the return response message unit, the method includes:
and the response message generating subunit is used for generating a data processing response message based on the access path and the unique identifier of the request terminal.
Optionally, the data acquiring apparatus 2 further includes: a presentation unit;
in the cue unit, comprising:
the sub-unit for obtaining the user social contact information is used for obtaining the user social contact information of the corresponding terminal based on the unique identifier of the request terminal after the response message of the returned data processing;
and the prompt information subunit is used for sending prompt information of finishing data processing to the user based on the social contact information of the user.
Optionally, the data acquiring apparatus 2 further includes:
and a shared data processing template configuration unit, configured to configure the shared data processing template and the template parameter before the data processing request message is acquired, and store the shared data processing template and the template parameter in the template data set.
The embodiment of the disclosure acquires the shared data processing template and the template parameter through the data processing request information, and acquires the data processing result from the fusion type distributed database through the shared data processing template and the template parameter. The data processing of similar logic is realized by configuring the generated shared data processing template, so that a plurality of businesses can share the same shared data processing template, the embarrassment that one business corresponds to one set of codes is avoided, the code redundancy is reduced, the maintenance cost is reduced, and the data processing is flexible and efficient. Meanwhile, the table association of the distributed service database is realized by utilizing the fusion type distributed database, so that the problems that the table association can be realized only by developing a large number of codes and related data is required to be acquired through memory mapping in the micro service are solved. Under the condition of not influencing the performance of the distributed service database, the code development amount is reduced, so that the service system has clear logic, single responsibility and easy maintenance. And a queue mechanism is adopted to ensure that tasks in the system can be executed in sequence, so that the system slowness caused by simultaneous execution of multiple tasks is avoided. The pressure of concurrent data processing on the database is effectively relieved.
The embodiment of the present disclosure provides a third embodiment, that is, an electronic device, where the electronic device is used in a data acquisition method based on a shared template, and the electronic device includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the one processor to cause the at least one processor to perform the shared template-based data acquisition method of the first embodiment.
The fourth embodiment provides a computer storage medium, which stores computer-executable instructions that can execute the data acquisition method based on the shared template as described in the first embodiment.
Referring now to FIG. 3, shown is a schematic diagram of an electronic device suitable for use in implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 3, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)302 or a program loaded from a storage device 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device to communicate wirelessly or by wire with other devices to exchange data. While fig. 3 illustrates an electronic device having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 309, or installed from the storage means 308, or installed from the ROM 302. The computer program, when executed by the processing device 301, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText transfer protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims (15)
1. A data acquisition method based on a shared template is characterized by comprising the following steps:
receiving a data processing request message;
analyzing the data processing request message to acquire data processing request information;
acquiring a shared data processing template and template parameters of the shared data processing template based on the data processing request information;
generating a data processing instruction facing the fusion type distributed database according to the shared data processing template and the request data; wherein the converged distributed database comprises data of a plurality of distributed traffic databases;
and generating a processing result file comprising a data processing result based on the data processing instruction.
2. The data acquisition method according to claim 1,
the data processing request information comprises a shared data processing template identifier and request data;
the obtaining of the shared data processing template and the template parameters of the shared data processing template based on the data processing request information includes:
retrieving a template dataset based on the shared data processing template identifier, obtaining the shared data processing template;
and acquiring the template parameter based on the request data.
3. The data acquisition method according to claim 1, wherein the generating a processing result file including a data processing result based on the data processing instruction includes:
generating a data processing task based on the data processing instruction, and storing the data processing task into a data processing task queue;
acquiring the data processing task from the data processing task queue;
and executing the data processing task to generate a processing result file comprising a data processing result.
4. The data acquisition method according to claim 1, wherein the generating a processing result file including a data processing result includes:
and generating the processing result file in a preset file format.
5. The data acquisition method of claim 1, further comprising:
storing the processing result file into a distributed file system;
acquiring an access path of the processing result file in the distributed file system;
and generating and returning a data processing response message based on the access path.
6. The data acquisition method according to claim 1,
the data processing request information comprises a unique identifier of a request terminal;
the generating a data processing response message based on the access path includes:
and generating a data processing response message based on the access path and the unique identifier of the request terminal.
7. The data acquisition method according to claim 6, further comprising, after the returning the data processing response message:
acquiring user social contact information of a corresponding terminal based on the unique identifier of the request terminal;
and sending prompt information for finishing data processing to the user based on the social contact information of the user.
8. The data acquisition method according to claim 2, further comprising, before said receiving a data processing request message:
and configuring the shared data processing template and the template parameters, and storing the shared data processing template and the template parameters into the template data set.
9. A data acquisition device based on a shared template, comprising:
a receiving request message unit for receiving a data processing request message;
the analysis request message unit is used for analyzing the data processing request message and acquiring data processing request information;
the shared data processing template obtaining unit is used for obtaining a shared data processing template and template parameters of the shared data processing template based on the data processing request information;
a data processing instruction generating unit, configured to generate a data processing instruction for the fusion-oriented distributed database according to the shared data processing template and the request data; wherein the converged distributed database comprises data of a plurality of distributed traffic databases;
and the processing result file generating unit is used for generating a processing result file comprising a data processing result based on the data processing instruction.
10. The data acquisition device according to claim 9,
the data processing request information comprises a shared data processing template identifier and request data;
in the unit of processing template for acquiring shared data, the method comprises:
an obtain shared data processing template subunit configured to retrieve a template dataset based on the shared data processing template identifier, and obtain the shared data processing template;
and the template parameter acquiring subunit is used for acquiring the template parameters based on the request data.
11. The data acquisition apparatus according to claim 9, wherein in said generating processing result file unit, the following is included:
a store-in-task queue subunit, configured to generate a data processing task based on the data processing instruction, and store the data processing task in a data processing task queue;
the task acquiring subunit is used for acquiring the data processing task from the data processing task queue;
and the file generation subunit is used for executing the data processing task and generating a processing result file comprising a data processing result.
12. The data acquisition apparatus according to claim 11, wherein in the generate file subunit, the method includes:
and the preset format file generating subunit is used for generating the processing result file in the preset file format.
13. The data acquisition device according to claim 9, characterized in that the data acquisition device further comprises:
a file saving unit, configured to save the processing result file in a distributed file system;
an access path acquiring unit, configured to acquire an access path of the processing result file in the distributed file system;
and the return response message unit is used for generating and returning a data processing response message based on the access path.
14. A computer-readable storage medium on which a computer program is stored, the program, when being executed by a processor, implementing the data acquisition method according to any one of claims 1 to 8.
15. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a data acquisition method as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010767768.7A CN111881216A (en) | 2020-08-03 | 2020-08-03 | Data acquisition method and device based on shared template |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010767768.7A CN111881216A (en) | 2020-08-03 | 2020-08-03 | Data acquisition method and device based on shared template |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111881216A true CN111881216A (en) | 2020-11-03 |
Family
ID=73204422
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010767768.7A Pending CN111881216A (en) | 2020-08-03 | 2020-08-03 | Data acquisition method and device based on shared template |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881216A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650777A (en) * | 2020-12-23 | 2021-04-13 | 深圳前海微众银行股份有限公司 | Data warehouse manufacturing method and device, terminal equipment and computer storage medium |
| CN113485763A (en) * | 2021-07-02 | 2021-10-08 | 中国建设银行股份有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN113626194A (en) * | 2021-08-10 | 2021-11-09 | 上海微盟企业发展有限公司 | Report file generation method, device, equipment and readable storage medium |
| CN113837320A (en) * | 2021-10-29 | 2021-12-24 | 中国建设银行股份有限公司 | Model migration method and system based on data classification |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105721224A (en) * | 2016-03-31 | 2016-06-29 | 网易（杭州）网络有限公司 | Method, device and system for processing uniform resource locator |
| CN107229628A (en) * | 2016-03-23 | 2017-10-03 | 中兴通讯股份有限公司 | The method and device of distributed data base pretreatment |
| CN109522357A (en) * | 2018-11-28 | 2019-03-26 | 北京锐安科技有限公司 | A kind of data processing method, device, server and storage medium |
| CN109614409A (en) * | 2018-10-29 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Business data processing method, device and server |
| CN109766387A (en) * | 2018-12-20 | 2019-05-17 | 平安信托有限责任公司 | Data processing document generating method, device, computer equipment and storage medium |
-
2020
- 2020-08-03 CN CN202010767768.7A patent/CN111881216A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229628A (en) * | 2016-03-23 | 2017-10-03 | 中兴通讯股份有限公司 | The method and device of distributed data base pretreatment |
| CN105721224A (en) * | 2016-03-31 | 2016-06-29 | 网易（杭州）网络有限公司 | Method, device and system for processing uniform resource locator |
| CN109614409A (en) * | 2018-10-29 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Business data processing method, device and server |
| CN109522357A (en) * | 2018-11-28 | 2019-03-26 | 北京锐安科技有限公司 | A kind of data processing method, device, server and storage medium |
| CN109766387A (en) * | 2018-12-20 | 2019-05-17 | 平安信托有限责任公司 | Data processing document generating method, device, computer equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650777A (en) * | 2020-12-23 | 2021-04-13 | 深圳前海微众银行股份有限公司 | Data warehouse manufacturing method and device, terminal equipment and computer storage medium |
| CN113485763A (en) * | 2021-07-02 | 2021-10-08 | 中国建设银行股份有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN113626194A (en) * | 2021-08-10 | 2021-11-09 | 上海微盟企业发展有限公司 | Report file generation method, device, equipment and readable storage medium |
| CN113837320A (en) * | 2021-10-29 | 2021-12-24 | 中国建设银行股份有限公司 | Model migration method and system based on data classification |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111881216A (en) | Data acquisition method and device based on shared template | |
| CN111309747A (en) | Data synchronization method, system and device | |
| CN111163336B (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN111813685A (en) | Automatic testing method and device | |
| CN114047986A (en) | Form generation method, form data processing method and device and electronic equipment | |
| CN112099982A (en) | Collapse information positioning method, device, medium and electronic equipment | |
| CN111241137A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111338834A (en) | Data storage method and device | |
| CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN114116247A (en) | Redis-based message processing method, device, system, server and medium | |
| CN111309366B (en) | Method, device, medium and electronic equipment for managing registration core | |
| CN111355784B (en) | Method, device, medium and electronic equipment for processing request information | |
| CN110727694A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115454666A (en) | Data synchronization method and device among message queue clusters | |
| CN114785770A (en) | Mirror layer file sending method and device, electronic equipment and computer readable medium | |
| CN110086880B (en) | Method and apparatus for creating instances of network requests | |
| CN112732457A (en) | Image transmission method, image transmission device, electronic equipment and computer readable medium | |
| CN111309367A (en) | Method, device, medium and electronic equipment for managing service discovery | |
| CN111787043A (en) | Data request method and device | |
| CN111414161A (en) | Method, device, medium and electronic equipment for generating ID L file | |
| CN111444457A (en) | Data distribution method and device, storage medium and electronic equipment | |
| CN113778387B (en) | Method and device for generating code | |
| CN112948108B (en) | Request processing method and device and electronic equipment | |
| CN116820354B (en) | Data storage method, data storage device and data storage system | |
| CN113342837B (en) | Data transmission method, device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |