CN110569308A - Data file assembling method, device, equipment and storage medium - Google Patents
Data file assembling method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110569308A CN110569308A CN201910854278.8A CN201910854278A CN110569308A CN 110569308 A CN110569308 A CN 110569308A CN 201910854278 A CN201910854278 A CN 201910854278A CN 110569308 A CN110569308 A CN 110569308A
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- queue
- thread
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
the embodiment of the invention discloses a data file assembling method, a data file assembling device, data file assembling equipment and a storage medium. The method comprises the following steps: reading preset data from a database through a first thread; writing the read preset data into a preset blocking queue through a first thread; reading queue data from a preset blocking queue through a second thread; and writing the read queue data into a data file through a second thread. The embodiment of the invention realizes the efficient assembly of the data file and reduces the data flow overhead of continuously opening and closing the data flow in the single-thread processing.
Description
Technical Field
the present invention relates to data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for assembling a data file.
Background
The large data interaction between the two service systems firstly assembles the data into a file, and transmits the file in a file form to ensure the integrity of the data.
In the prior art, a business system generates a plurality of data files every day, and the processing method is that data in a database is read in batches, then the data are processed, and finally the data in the batch are written into the files, and then the data in the database are processed in batches according to the above method until the data in the last batch are processed.
however, the method for assembling the data file has the following defects: firstly, a single thread is used for reading data and then writing the data into a data file, and the single thread processing is time-consuming and causes poor timeliness. Secondly, the single thread executes the write file in batches, and each time the write file is executed, the overhead of opening and closing the data stream is generated, so that the performance is low.
Disclosure of Invention
Embodiments of the present invention provide a data file assembly method, apparatus, device, and storage medium, which implement efficient assembly of data files and reduce data stream overhead.
in a first aspect, an embodiment of the present invention provides a data file assembly method, including:
Reading preset data from a database through a first thread;
writing the read preset data into a preset blocking queue through the first thread;
reading queue data from the preset blocking queue through a second thread;
and writing the read queue data into a data file through the second thread.
In a second aspect, an embodiment of the present invention further provides a data file assembling apparatus, including:
the preset data reading module is used for reading preset data from the database through a first thread;
a blocking queue data writing module, configured to write the read preset data into a preset blocking queue through the first thread;
The queue data reading module is used for reading queue data from the preset blocking queue through a second thread;
and the data writing module is used for writing the read queue data into a data file through the second thread.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
One or more processors;
A memory for storing one or more programs,
When executed by the one or more processors, cause the one or more processors to implement a data file assembly method as provided by any embodiment of the invention.
In a fourth aspect, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used to perform a data file assembly method as provided in any of the embodiments of the present invention.
The embodiment of the invention reads preset data from a database through a first thread and writes the preset data read by the first thread into a preset blocking queue; and then reading data from the preset blocking queue through the second thread, and writing the queue data of the second thread reading band into a data file. By the aid of the branch thread processing, the data files are efficiently assembled, and data flow overhead of continuously opening and closing data flows in single thread processing is reduced.
Drawings
FIG. 1 is a flow chart of a data file assembling method according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a data file assembling method according to a second embodiment of the present invention;
FIG. 3 is a detailed flowchart of a first thread and a second thread according to a second embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a data file assembling apparatus according to a third embodiment of the present invention;
Fig. 5 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
example one
Fig. 1 is a flowchart of a data file assembling method in an embodiment of the present invention, where the embodiment is applicable to a data processing situation in a system, the method may be executed by a data file assembling device, the device may be implemented in a software and/or hardware manner, and is integrated in a device for executing the method, and the device for executing the method in the embodiment may be any intelligent terminal such as a computer, a tablet computer, and/or a mobile phone. Specifically, referring to fig. 1, the method specifically includes the following steps:
And S110, reading preset data from the database through the first thread.
Specifically, a thread is a processor of one task, and one thread represents one processor, wherein the task processed by the thread may be a data read operation, a data write operation, or a data calculation operation. A database is a repository used to store data, e.g., MySQL, SQL Server, Oracle, Sybase, DB2, etc. The preset data is data which is stored in the database and needs to be read, namely data which is required to complete a certain task and is stored in the database, for example, in order to investigate the average height of a male in a certain place, the preset data is data which is stored in the database and is related to the height of the male; or in order to count the age and gender distribution of the people with lung cancer in a certain area, the preset data is the data of the people with lung cancer in the certain area.
Optionally, before the preset data is read from the database by the first thread, a preset blocking queue is established. It should be noted that the preset blocking queue is a container for storing preset data, the preset blocking queue is an ordered container, and the preset data needs to follow the principle of first-in first-out in the preset blocking queue, that is, the data that enters the preset blocking queue first needs to be output from the preset blocking queue first.
And S120, writing the read preset data into a preset blocking queue through the first thread.
Specifically, the first thread writes the preset data read from the database into the preset blocking queue, so as to realize faster reading of the preset data.
Optionally, after the first thread writes all the preset data into the preset blocking queue, the first thread writes a preset ending identifier into the preset blocking queue, where the preset ending identifier is used to identify that the preset data is written completely. Optionally, the preset ending flag is a constant different from the preset data, and may be defined by the user, for example, if the preset data is the local temperature of the year, the preset ending flag may be set to 100, because the natural weather temperature may not exceed 100 degrees celsius, the setting of the preset ending flag to 100 may be obviously distinguished from the preset data, and after the first thread writes 100 into the preset blocking queue, the first thread will end the operation of writing the data into the preset blocking queue.
and S130, reading queue data from the preset blocking queue through the second thread.
Specifically, after the first thread writes preset data and a preset ending identifier into a preset blocking queue, the second thread reads queue data from the preset blocking queue, wherein the queue data includes all the preset data and the preset ending identifier written into the preset blocking queue by the first thread.
And S140, writing the read queue data into a data file through a second thread.
Specifically, the second thread converts the read queue data into a data stream, and writes the data stream into a data file, wherein the data file can realize large-scale data exchange between the two systems, and data transmission is performed in the form of the data file, so that the integrity of the data can be ensured, and the phenomenon of data loss can be avoided.
optionally, when the preset ending identifier is read from the preset blocking queue, closing the data stream for writing the queue data into the data file.
it should be noted that, in the embodiment of the present invention, the second thread does not write queue data into the data file after reading all queue data in the preset blocking queue, but writes the queue data into the data file while reading the queue data, and stops converting the queue data into a data stream when reading the preset end identifier, that is, stops writing the queue data into the data file. The preset blocking queue follows the first-in first-out principle, and the preset ending mark is written into the preset blocking queue at last, so that the condition that the preset data written into the data file is incomplete does not occur.
According to the technical scheme of the embodiment, the preset data are read from the database through the first thread, and the preset data read by the first thread are written into the preset blocking queue; and then reading data from the preset blocking queue through the second thread, and writing the queue data of the second thread reading band into a data file. Through the branch thread processing, the preset data are efficiently written into the data file, and the data flow overhead of continuously opening and closing the data flow in the single thread processing is reduced.
Example two
fig. 2 is a flowchart of a data file assembling method in a second embodiment of the present invention, which is optimized in this embodiment, and in this embodiment, before reading queue data from a preset blocking queue through a second thread, the method further includes:
And performing preset processing on the queue data in the preset blocking queue through a third thread.
correspondingly, as shown in fig. 2, the method specifically includes the following steps:
and S210, reading preset data from the database through a first thread.
and S220, writing the read preset data into a preset blocking queue through the first thread.
and S230, performing preset processing on queue data in the preset blocking queue through a third thread.
And performing preset processing on the queue data in the preset blocking queue through a third thread before reading the queue data from the preset blocking queue through the second thread. Specifically, the third thread processes the preset data in the preset blocking queue, where the processing mode may be data operation on the preset data or format conversion on the preset data.
the queue data is preset through the third thread, so that a system for receiving the data file can accurately identify the data in the data file, and errors that the data file cannot be analyzed cannot occur.
And S240, reading queue data from the preset blocking queue through the second thread.
And S250, writing the read queue data into a data file through a second thread.
For better understanding of the embodiment of the present invention, fig. 3 is a specific flowchart of the first thread and the second thread in the embodiment, specifically, the first thread reads preset data from a database and determines whether there is preset data, if there is preset data, the preset data is written into a preset blocking queue, and if there is no preset data, that is, all the preset data has been read, the preset end identifier is written into the preset blocking queue. The second thread reads queue data from the preset blocking queue, judges whether the read queue data is a preset ending identifier or not, and closes the data flow if the read queue data is the preset ending identifier; and if the queue data is not the preset ending identifier, writing the queue data into the data file.
in this embodiment, on the basis of the foregoing embodiments, the third thread is added to perform the preset processing on the queue data, and on the basis that the preset data is written into the data file, the system receiving the data file can accurately identify the data in the data file, and the problem that the data file cannot be analyzed does not occur.
EXAMPLE III
fig. 4 is a schematic structural diagram of a data file assembling apparatus according to a third embodiment of the present invention, which may be implemented in software and/or hardware. As shown in fig. 4, the apparatus includes: a preset data read module 410, a block queue data write module 420, a queue data read module 430, and a data write module 440.
the preset data reading module 410 is configured to read preset data from a database through a first thread;
A block queue data writing module 420, configured to write the read preset data into a preset block queue through a first thread;
A queue data reading module 430, configured to read queue data from a preset blocking queue through a second thread;
and a data writing module 440, configured to write the read queue data into a data file through the second thread.
Optionally, the apparatus further comprises: and the preset blocking queue establishing module is used for establishing a preset blocking queue.
And the preset ending identifier writing module is used for writing the read preset data into the preset blocking queue through the first thread, and writing the preset ending identifier into the preset blocking queue through the first thread when the first thread finishes writing the preset data into the preset blocking queue.
And the data flow closing module is used for closing the data flow for writing the queue data into the data file when the preset ending identifier is read from the preset blocking queue after the queue data is read from the preset blocking queue through the second thread.
Optionally, the apparatus further includes, before the queue data reading module 430: and the queue data presetting processing module is used for presetting the queue data in the preset blocking queue through a third thread.
The data file assembling device provided by the embodiment of the invention can execute the data file assembling method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the executing method.
example four
fig. 5 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention, as shown in fig. 5, the apparatus includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of the processors 510 in the device may be one or more, and one processor 510 is taken as an example in fig. 5; the processor 510, the memory 520, the input device 530 and the output device 540 of the apparatus may be connected by a bus or other means, as exemplified by the bus connection in fig. 5.
The memory 520, which is a computer-readable storage medium, can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data file assembling method in the embodiment of the present invention (for example, the default data reading module 410, the block queue data writing module 420, the queue data reading module 430, and the data writing module 440). The processor 510 executes various functional applications of the device and data processing by executing software programs, instructions and modules stored in the memory 520, that is, implements the data file assembly method described above.
The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 520 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 520 may further include memory located remotely from processor 510, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
the input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the apparatus. The output device 540 may include a display device such as a display screen.
EXAMPLE five
an embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a data file assembly method, including:
reading preset data from a database through a first thread;
Writing the read preset data into a preset blocking queue through a first thread;
Reading queue data from a preset blocking queue through a second thread;
and writing the read queue data into a data file through a second thread.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the data file assembly method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the data file assembling apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (12)
1. A method of assembling a data file, comprising:
Reading preset data from a database through a first thread;
Writing the read preset data into a preset blocking queue through the first thread;
reading queue data from the preset blocking queue through a second thread;
And writing the read queue data into a data file through the second thread.
2. the method for assembling data file according to claim 1, further comprising, before the reading preset data from the database by the first thread:
And establishing the preset blocking queue.
3. The method according to claim 1, further comprising, after the writing the read preset data into a preset blocking queue by the first thread, the step of:
and when the first thread finishes writing the preset data into the preset blocking queue, writing a preset ending identifier into the preset blocking queue through the first thread.
4. The method of claim 3, further comprising, after said reading queue data from said predetermined blocked queue by said second thread:
And when the preset ending mark is read from the preset blocking queue, closing the data flow for writing the queue data into the data file.
5. A method for assembling a data file according to claim 3 or 4, wherein said end flag is a preset constant.
6. The method of any of claims 1-4, wherein prior to said reading queue data from said pre-set blocked queue by said second thread, further comprising:
and performing preset processing on the queue data in the preset blocking queue through a third thread.
7. a data file assembling apparatus, comprising:
The preset data reading module is used for reading preset data from the database through a first thread;
A blocking queue data writing module, configured to write the read preset data into a preset blocking queue through the first thread;
The queue data reading module is used for reading queue data from the preset blocking queue through a second thread;
And the data writing module is used for writing the read queue data into a data file through the second thread.
8. The data file assembly device of claim 7, further comprising:
And the preset ending identifier writing module is used for writing the preset ending identifier into the preset blocking queue through the first thread when the first thread finishes writing the preset data into the preset blocking queue after the read preset data is written into the preset blocking queue through the first thread.
9. the data file assembly device of claim 8, further comprising:
and the data flow closing module is used for closing the data flow for writing the queue data into the data file when the preset ending identifier is read from the preset blocking queue after the queue data is read from the preset blocking queue through the second thread.
10. the data file assembly device of any one of claims 7-9, further comprising:
And the data processing module is used for performing preset processing on the queue data in the preset blocking queue through a third thread before the queue data is read from the preset blocking queue through the second thread.
11. an apparatus, characterized in that the apparatus comprises:
One or more processors;
A memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data file assembly method as claimed in any one of claims 1-6.
12. A storage medium containing computer-executable instructions for performing the data file assembly method of any one of claims 1-6 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910854278.8A CN110569308A (en) | 2019-09-10 | 2019-09-10 | Data file assembling method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910854278.8A CN110569308A (en) | 2019-09-10 | 2019-09-10 | Data file assembling method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569308A true CN110569308A (en) | 2019-12-13 |
Family
ID=68779034
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910854278.8A Pending CN110569308A (en) | 2019-09-10 | 2019-09-10 | Data file assembling method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569308A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111831625A (en) * | 2020-07-14 | 2020-10-27 | 深圳力维智联技术有限公司 | Data migration method, data migration device and readable storage medium |
| CN113672876A (en) * | 2021-10-21 | 2021-11-19 | 南京拓界信息技术有限公司 | OTG-based method and device for quickly obtaining evidence of mobile phone |
| CN114253714A (en) * | 2021-12-07 | 2022-03-29 | 四川新网银行股份有限公司 | Method, system and equipment for processing multithreading data based on blocking queue |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615490A (en) * | 2015-02-05 | 2015-05-13 | 浪潮集团有限公司 | Method and device for data conversion |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
-
2019
- 2019-09-10 CN CN201910854278.8A patent/CN110569308A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615490A (en) * | 2015-02-05 | 2015-05-13 | 浪潮集团有限公司 | Method and device for data conversion |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111831625A (en) * | 2020-07-14 | 2020-10-27 | 深圳力维智联技术有限公司 | Data migration method, data migration device and readable storage medium |
| CN111831625B (en) * | 2020-07-14 | 2024-03-12 | 深圳力维智联技术有限公司 | Data migration method, data migration device, and readable storage medium |
| CN113672876A (en) * | 2021-10-21 | 2021-11-19 | 南京拓界信息技术有限公司 | OTG-based method and device for quickly obtaining evidence of mobile phone |
| CN113672876B (en) * | 2021-10-21 | 2022-02-01 | 南京拓界信息技术有限公司 | OTG-based method and device for quickly obtaining evidence of mobile phone |
| CN114253714A (en) * | 2021-12-07 | 2022-03-29 | 四川新网银行股份有限公司 | Method, system and equipment for processing multithreading data based on blocking queue |
| CN114253714B (en) * | 2021-12-07 | 2024-04-12 | 四川新网银行股份有限公司 | Method, system and equipment for processing multithreaded data based on blocking queue |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| EP3497593B1 (en) | Summarized data storage management system for streaming data | |
| US10606806B2 (en) | Method and apparatus for storing time series data | |
| CN110569308A (en) | Data file assembling method, device, equipment and storage medium | |
| CN110825694A (en) | Data processing method, device, equipment and storage medium | |
| CN110688828A (en) | File processing method and device, file processing system and computer equipment | |
| US10225357B2 (en) | Compact data structures for push notifications | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| CN113282611A (en) | Method and device for synchronizing stream data, computer equipment and storage medium | |
| CN112613964A (en) | Account checking method, account checking device, account checking equipment and storage medium | |
| CN114925101A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN110633318A (en) | Data extraction processing method, device, equipment and storage medium | |
| CN113918659A (en) | Data operation method and device, storage medium and electronic equipment | |
| CN112506490A (en) | Interface generation method and device, electronic equipment and storage medium | |
| CN115269063A (en) | Process creation method, system, device and medium | |
| CN113626516A (en) | Data increment synchronization method and system | |
| CN111625524B (en) | Data processing method, device, equipment and storage medium | |
| CN109740027B (en) | Data exchange method, device, server and storage medium | |
| CN113778977A (en) | Data processing method and data processing device | |
| CN114185890B (en) | Database retrieval method and device, storage medium and electronic equipment | |
| CN110674224A (en) | Entity data processing method, device, equipment and computer readable storage medium | |
| CN114584616B (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN110727654B (en) | Data extraction method and device for distributed system, server and storage medium | |
| CN111163088B (en) | Message processing method, system and device and electronic equipment | |
| CN113127222B (en) | Data transmission method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |