CN111145028A - Distributed text pre-check method and device - Google Patents
Distributed text pre-check method and device Download PDFInfo
- Publication number
- CN111145028A CN111145028A CN201911418178.7A CN201911418178A CN111145028A CN 111145028 A CN111145028 A CN 111145028A CN 201911418178 A CN201911418178 A CN 201911418178A CN 111145028 A CN111145028 A CN 111145028A
- Authority
- CN
- China
- Prior art keywords
- text
- processed
- data
- distributed
- cache table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 239000012634 fragment Substances 0.000 claims abstract description 66
- 238000007689 inspection Methods 0.000 claims abstract description 43
- 238000003860 storage Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012545 processing Methods 0.000 abstract description 27
- 230000008569 process Effects 0.000 abstract description 23
- 238000001514 detection method Methods 0.000 abstract description 16
- 238000010586 diagram Methods 0.000 description 15
- 238000013467 fragmentation Methods 0.000 description 7
- 238000006062 fragmentation reaction Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed text pre-check method and a device, wherein the method comprises the following steps: storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed; acquiring data to be processed from the cache table according to fragments; performing text pre-inspection on the data to be processed based on a pre-inspection rule; and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result. The distributed pre-detection is realized by storing the text fragments to be processed into the cache table and then acquiring the data to be processed from the cache table by a plurality of threads or a plurality of servers according to the fragments, the time consumed by the text pre-detection is short, and the problem of repeated processing does not exist when a plurality of machines process the same text at the same time.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a distributed text pre-inspection method and device.
Background
In banking or financial industry systems, some transactions are triggered by uploaded text (e.g., corporate salary), and currently such transactions are mainly completed by the host. The host computer needs to perform preliminary examination on the text item by item, confirm that some key fields (such as account codes, organization codes and the like) are correctly filled, and then perform online transaction on the text item by item. Such transactions are characterized by text that is typically large. The text pre-inspection consumes long time, and the problem of repeated processing exists when a plurality of machines process the same text at the same time.
Disclosure of Invention
The present invention provides a distributed text preview method and apparatus, an electronic device, and a computer-readable storage medium, which can solve at least some of the problems in the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme:
in a first aspect, a distributed text preview method is provided, including:
storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
acquiring data to be processed from the cache table according to fragments;
performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result.
Further, the data state which is not subjected to text preview in the cache table is marked as unprocessed, and the data state which is subjected to text preview is marked as processed;
the acquiring the data to be processed from the cache table according to the fragments comprises the following steps:
streaming scans unprocessed data in the cache table to capture unprocessed fragments;
if unprocessed fragments are captured, acquiring to-be-processed data contained in the fragments;
and if the unprocessed fragment is not captured, rescanning the cache table.
Further, the distributed text preview method further comprises the following steps:
and pulling the text to be processed to the local by the service server by adopting a distributed lock adding mode.
Further, after storing the text fragment to be processed in the cache table, the method further includes:
and releasing the lock added on the text to be processed.
Further, the distributed text preview method further comprises the following steps:
and obtaining an online transaction result from the host and feeding back the online transaction result to the service terminal.
In a second aspect, a distributed text preview device is provided, which includes:
the text fragment caching module is used for storing the text fragments to be processed into a caching table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
the to-be-processed data acquisition module acquires the to-be-processed data from the cache table according to the fragments;
the text pre-inspection module is used for performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and the pre-check result sending module is used for sending the data to be processed and the corresponding text pre-check result to the host so that the host can carry out online transaction on the data to be processed according to the text pre-check result.
Further, the data state which is not subjected to text preview in the cache table is marked as unprocessed, and the data state which is subjected to text preview is marked as processed;
the module for acquiring the data to be processed comprises:
the stream type scanning unit is used for stream type scanning the unprocessed data in the cache table to capture unprocessed fragments;
the data acquisition unit is used for acquiring to-be-processed data contained in the fragments if unprocessed fragments are captured;
and the rescanning unit rescans the cache table if the unprocessed fragment is not grabbed.
Further, the distributed text preview device further comprises:
and the data locking and pulling module is used for pulling the text to be processed to the local by the service server in a distributed locking mode.
Further, the distributed text preview device further comprises:
and the lock releasing module is used for releasing the lock added on the text to be processed.
Further, the distributed text preview device further comprises:
and the transaction result feedback module is used for acquiring the online transaction result from the host and feeding the online transaction result back to the service terminal.
In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the distributed text preview method when executing the program.
In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the distributed text preview method described above.
The invention provides a distributed text preview method and device, electronic equipment and a computer readable storage medium, wherein the method comprises the following steps: storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed; acquiring data to be processed from the cache table according to fragments; performing text pre-inspection on the data to be processed based on a pre-inspection rule; and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result. The distributed pre-detection is realized by storing the text fragments to be processed into the cache table and then acquiring the data to be processed from the cache table by a plurality of threads or a plurality of servers according to the fragments, the time consumed by the text pre-detection is short, and the problem of repeated processing does not exist when a plurality of machines process the same text at the same time.
In order to make the aforementioned and other objects, features and advantages of the invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. In the drawings:
fig. 1 is a system architecture diagram for implementing a distributed text preview method according to an embodiment of the present invention;
FIG. 2 is a first flowchart illustrating a distributed text preview method according to an embodiment of the present invention;
fig. 3 shows the specific steps of step S200 in fig. 2;
FIG. 4 is a flowchart illustrating a distributed text preview method according to an embodiment of the present invention;
FIG. 5 is a third flowchart illustrating a distributed text preview method according to an embodiment of the present invention;
FIG. 6 is a block diagram of a distributed text preview device in an embodiment of the present invention;
fig. 7 is a block diagram of an electronic device according to an embodiment of the invention.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of this application and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
At present, banks or large financial platforms widely use IBM host systems to carry out batch online transactions, and the problems that in the prior art, text pre-inspection consumes long time and multiple machines process the same text at the same time are repeatedly processed exist.
In order to at least partially solve the technical problems in the prior art, embodiments of the present invention provide a distributed text pre-check method, where a text fragment to be processed is stored in a cache table, and then data to be processed is obtained from the cache table by multiple threads or multiple servers according to the fragment, so as to implement distributed pre-check, where the text pre-check consumes a short time, and multiple machines do not have a problem of repeated processing when processing the same text at the same time.
Interpretation of terms:
a host computer: the use of a dedicated processor instruction set, operating system and application software provides extremely high I/O capability, stability and security.
MIPS, million Instructions Per Second, which deals with millions of Instructions Per Second, a basis for host manufacturers to charge banks for using hosts.
Fig. 1 is a system architecture diagram for implementing a distributed text preview method according to an embodiment of the present invention; as shown in fig. 1, the system architecture includes: service terminal 1, shared disk 2, X86 distributed system 3, shared disk 4, host 5, and database 6.
The shared disk 2 is disposed between the service terminal 1 and the X86 distributed system 3, and is used for transferring data between the service terminal 1 and the X86 distributed system. The service terminal 1 stores the text to be processed into the shared disk 2, and the X86 distributed system 3 acquires the text to be processed from the shared disk 2 and performs distributed pre-inspection on the text to be processed; the shared disk 4 is provided between the X86 distributed system 3 and the host 5 for transferring data therebetween. The X86 distributed system 3 sends the text to be processed and the corresponding distributed pre-check result to the host 5, so that the host 5 performs online transaction on the data to be processed according to the text pre-check result; the host 5 stores the online transaction result to the shared disk 4, after the X86 distributed system 3 obtains the online transaction result from the shared disk 4, the online transaction result is stored to the shared disk 2, and the service terminal 1 obtains the online transaction result from the shared disk 2. Namely: the X86 distributed system interacts with hosts by sharing FTP paths and traffic terminals.
In addition, the X86 distributed system 3 is also connected to a database 6, and the database 6 is used for recording the text to be processed and the distributed preview result, the online transaction result, and the like.
According to the architecture diagram, the data flow is as follows:
1. the service terminal pushes the text to be processed to a shared disk 2 of an X86 distributed system;
2, scanning a new text by the X86 distributed system, and storing the text content in a database cache table;
3, scanning a cache table by an X86 distributed system, performing text pre-inspection, forming a text with a pre-inspection result, and putting the text into a shared disk 4 of a host;
4. the host 5 performs the transaction, and transmits the text with the transaction result to the X86 distribution system, and then the X86 distribution system transmits to the service terminal.
It is worth noting that since the text pre-check is executed by the X86 distributed system and the online transaction is executed by the host, the cheap resources of the X86 system can be effectively utilized, the host MIPS consumption is reduced, the stability and performance of the transaction can be ensured, and the customer account can be prevented from being confused.
Namely: the system architecture provided by the embodiment of the invention adopts a working mechanism of a double system (X86 distributed system + host): the dual systems together complete the bulk text transaction. The X86 distributed system is only responsible for completing the pre-check work of the text, and after the content of each line of text is pre-checked, the pre-check result is recorded in the corresponding column to inform the host. And after the pre-check is finished, pushing the host for transaction, and selecting the text passing the pre-check by the host for transaction according to the existing pre-check result. Due to the good characteristics of the host, the stability of the correct line of financial transaction can be ensured, the MIPS of the host is reduced by only using the X86 distributed system to perform text pre-inspection, the final guarantee is performed on the pre-inspection output result of the X86 distributed system by using the host, and the text pre-inspection is ensured to be omitted.
Specifically, the host is used for making final guarantee on the pre-detection result: and recording the result of the distributed pre-detection in the text, if the result is qualified data, performing transaction by the host, if the result is unqualified, not processing by the host, and if the pre-detection result is unknown due to instability of the X86 distributed system, performing the pre-detection method by the host, and performing transaction after re-performing the pre-detection on the data. Therefore, omission caused by the stability problem of the distributed system can be avoided in the pre-detection.
The X86 distributed system provides preview services in the form of micro services, thereby increasing the expandability of the services and providing different types of preview micro services for different types of texts;
it is understood that the service terminal 1 may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, an intelligent wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
The devices may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (remote procedure Call Protocol) used above the above Protocol, a REST Protocol (Representational state transfer Protocol), and the like.
The flow of the X86 distributed system 3 performing distributed pre-check on the text to be processed is shown in fig. 2, and the distributed text pre-check method may include:
step S100: storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
after the text to be processed is pulled to the local, the text is backed up locally, and the data in the text is imported into a cache table of the database line by line. When the text is imported, the text is fragmented, and the fragment number is recorded in a certain column of the cache table.
Because the text is large, if only one machine is used for processing one text, the time is necessarily long, and a plurality of machines are used for simultaneously processing the same text in the embodiment of the invention. Because transactions triggered by each line of content in the text are independent of each other, the content of the text is divided into a plurality of pieces by adopting a preset fragmentation strategy (for example, the text is divided into n pieces in a way of taking the line number for n, and the text can be specifically selected according to service requirements), and each fragment is taken as a task by a certain transaction server in a distributed system for pre-check processing by a batch framework (for example, elastic-job, spring batch).
In order to ensure reliability, a batch framework can be introduced into a scheduling center to ensure uniform task distribution, so that batch parallel processing is realized, the pre-detection efficiency is improved, and the scheduling center can reassign uncompleted tasks to other servers to complete if a server fails in certain fragmentation processing.
Specifically, reasonable fragmentation is carried out on larger text data by using a cache table, a plurality of X86 servers are used for processing texts together, and the concept of batch streaming processing is adopted to complete text pre-inspection.
Step S200: acquiring data to be processed from the cache table according to fragments;
and each thread or each server acquires data to be processed from the cache table according to fragments.
The machine multithreading multitask in the distributed cluster completes the whole process, and the server starts a plurality of processes and respectively completes a plurality of tasks which are independent of each other. These tasks are independent of each other and constitute the whole process flow as a whole. Thereby saving hardware resources and facilitating later maintenance.
Step S300: performing text pre-inspection on the data to be processed based on a pre-inspection rule;
wherein the pre-check rule is preset by a developer. The pre-check is performed only if some key fields are correctly filled, such as account codes, organization codes and the like.
Step S400: and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result.
If the pre-check result is qualified data, the host machine carries out transaction, if the pre-check result is unqualified, the host machine does not process the data, if the pre-check result is unknown, the host machine executes the pre-check method, and after the data is re-checked, the transaction is carried out. Therefore, omission caused by the stability problem of the distributed system can be avoided in the pre-detection.
The distributed text pre-inspection method provided by the embodiment of the invention realizes distributed pre-inspection by storing the text to be processed in the cache table in a fragmentation mode and then acquiring the data to be processed from the cache table by a plurality of threads or a plurality of servers according to the fragmentation mode, and has the advantages of short text pre-inspection time and no repeated processing problem when a plurality of machines process the same text at the same time.
In an alternative embodiment, referring to fig. 3, this step S200 may include the following:
step S210: streaming scans unprocessed data in the cache table to capture unprocessed fragments;
and the data state which is not subjected to text preview in the cache table is marked as unprocessed, and the data state which is subjected to text preview is marked as processed.
In addition, the scan interval may be self-configurable.
Step S220: judging whether unprocessed fragments are captured or not;
if yes, go to step S230; if not, go to step S240.
Step S230: acquiring to-be-processed data contained in the fragments;
step S240: the cache table is rescanned.
The server for executing the task scans unprocessed fragments in the cache table in a streaming mode, if the unprocessed fragments are captured, the preview micro service is called for previewing, the state of the data in the cache table is updated after the processing is finished, and the unprocessed fragments are changed into processed fragments. If not, the cache table is rescanned.
In addition, after all the data in the cache table are processed, the server can export the processed table to a text and place the text into a text folder shared with the host, so that the host can process the table according to the pre-detection result.
In an alternative embodiment, referring to fig. 4, the distributed text preview method may further include the following:
step S10: and pulling the text to be processed to the local by the service server by adopting a distributed lock adding mode.
The process of locking and pulling the text by the single X86 server is as follows: and a distributed lock (which can be based on redis, a database, zookeeper and the like) is added to pull the text to be processed to the local distributed cluster servers, so that one text is only processed and pre-checked once by one server in the distributed cluster, and the text cannot be processed for a plurality of times to cause repetition.
Specifically, the server starts a task (task one) to acquire the text to be processed in the text folder shared by the front end and the distributed system. During acquisition, each text is pulled by only one server due to the distributed lock, and repeated pulling processing cannot be caused.
In a further embodiment, with continued reference to fig. 4, the distributed text preview method may further include the following:
step S150: and releasing the lock added on the text to be processed.
Specifically, after the text to be processed is cached in the cache table, the lock added to the text to be processed is released.
In an alternative embodiment, referring to fig. 5, the distributed text preview method may further include the following:
step S500: and obtaining an online transaction result from the host and feeding back the online transaction result to the service terminal.
Specifically, after the online transaction result is obtained, the host returns a text backup and pushes the text backup to the service terminal, so that the service terminal can know the transaction processing condition in time.
In addition, it is worth explaining that the embodiment of the present invention adopts multi-thread multi-task to complete the pre-check work: the server in the distributed cluster starts a plurality of threads to complete the task in a segmented manner, the boundaries of the multiple tasks are clear, and if one task fails, the front end does not need to be informed to do business from the beginning, and the task can be restarted from a certain task.
For example, after a main thread is started, three sub-threads may be started for implementing three different tasks, task one: and pulling the text to be processed to the local by the service server in a distributed locking mode, specifically, firstly locking the text to be processed, acquiring the text, fragmenting the text data, caching the text data into a cache table, then releasing the lock, then pulling another text to be processed again, and repeating the steps. And a second task: performing pre-inspection, specifically, acquiring unprocessed data from a cache table according to fragments, calling a pre-inspection text micro-service for pre-inspection, uploading a result document to a host shared disk, returning, and performing cycle according to the steps; and a third task: and acquiring the text after the transaction from the host shared disk, feeding the text with the transaction result back to the service terminal, acquiring the text after the transaction from the host shared disk again, and circulating according to the steps to realize multithreading multi-task pre-inspection.
In addition, the method may further include:
auxiliary steps are as follows: after the X86 distributed system pulls the front text, the text needs to be backed up to other folders and deleted, so as not to pull repeatedly.
The good step is as follows: the X86 distributed system records the text into the database cache table, and after the pre-check is completed, the cache table needs to be emptied.
In summary, the distributed text preview method provided in the embodiments of the present invention can greatly save host resources and save overhead, the preview texts are performed in batch by the X86 distributed system, efficiency is ensured, and the host is used to process unclear preview texts on the distributed system, so as to ensure that the preview is not missed.
Based on the same inventive concept, the embodiment of the present application further provides a distributed text preview device, which can be used to implement the method described in the above embodiment, as described in the following embodiments. Because the principle of the distributed text preview device for solving the problem is similar to that of the method, the implementation of the distributed text preview device can refer to the implementation of the method, and repeated details are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 6 is a block diagram of a distributed text preview device in the embodiment of the present invention. As shown in fig. 6, the distributed text preview device specifically includes: the system comprises a text fragment caching module 10, a to-be-processed data acquisition module 20, a text preview module 30 and a preview result sending module 40.
The text fragment cache module 10 stores the text fragments to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
the to-be-processed data acquisition module 20 acquires the to-be-processed data from the cache table according to the fragments;
the text pre-check module 30 performs text pre-check on the data to be processed based on a pre-check rule;
the pre-check result sending module 40 sends the data to be processed and the corresponding text pre-check result to the host, so that the host performs online transaction on the data to be processed according to the text pre-check result.
The distributed text pre-detection device provided by the embodiment of the invention realizes distributed pre-detection by storing the text to be processed in the cache table in a fragmentation mode and then acquiring the data to be processed from the cache table by a plurality of threads or a plurality of servers according to the fragmentation mode, and has the advantages of short text pre-detection time and no repeated processing problem when a plurality of machines process the same text at the same time.
In an optional embodiment, the data state of the cache table which is not subjected to text preview is marked as unprocessed, and the data state of the cache table which is subjected to text preview is marked as processed;
the to-be-processed data acquisition module 20 includes: the device comprises a streaming scanning unit, a data acquisition unit and a rescanning unit.
Stream type scanning unit stream type scans unprocessed data in the cache table to capture unprocessed fragments;
if an unprocessed fragment is captured by the data acquisition unit, acquiring to-be-processed data contained in the fragment;
and if the rescanning unit does not grab the unprocessed fragment, rescanning the cache table.
In an optional embodiment, the distributed text preview device further includes: and the data locking and pulling module is used for pulling the text to be processed to the local by the service server in a distributed locking mode.
In an optional embodiment, the distributed text preview device further includes: and the lock releasing module is used for releasing the lock added on the text to be processed.
In an optional embodiment, the distributed text preview device further includes: and the transaction result feedback module is used for acquiring the online transaction result from the host and feeding the online transaction result back to the service terminal.
The apparatuses, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or implemented by a product with certain functions. A typical implementation device is an electronic device, which may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
In a typical example, the electronic device specifically includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program:
storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
acquiring data to be processed from the cache table according to fragments;
performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result.
As can be seen from the above description, the electronic device provided in the embodiment of the present invention may be used for distributed pre-inspection of a text, where the text to be processed is segmented and stored in a cache table, and then the data to be processed is obtained from the cache table by multiple threads or multiple servers according to the segmentation, so as to implement distributed pre-inspection, where the time consumed by the text pre-inspection is short, and there is no problem of repeated processing when multiple machines process the same text at the same time.
Referring now to FIG. 7, shown is a schematic diagram of an electronic device 600 suitable for use in implementing embodiments of the present application.
As shown in fig. 7, the electronic apparatus 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate works and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM)) 603. In the RAM603, various programs and data necessary for the operation of the system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted as necessary on the storage section 608.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the invention includes a computer-readable storage medium having a computer program stored thereon, which when executed by a processor, performs the steps of:
storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
acquiring data to be processed from the cache table according to fragments;
performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result.
As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present invention may be used for distributed pre-inspection of a text, where the text to be processed is fragmented and stored in a cache table, and then data to be processed is obtained from the cache table by multiple threads or multiple servers according to the fragments, so as to implement distributed pre-inspection, where the time consumed by the text pre-inspection is short, and there is no problem of repeated processing when multiple machines process the same text at the same time.
In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (12)
1. A distributed text preview method is characterized by comprising the following steps:
storing a text fragment to be processed into a cache table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
acquiring data to be processed from the cache table according to fragments;
performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and sending the data to be processed and the corresponding text pre-check result to the host computer so that the host computer performs online transaction on the data to be processed according to the text pre-check result.
2. The distributed text preview method of claim 1, wherein the data status of the cache table that is not subjected to text preview is marked as unprocessed, and the data status of the cache table that is subjected to text preview is marked as processed;
the acquiring the data to be processed from the cache table according to the fragments comprises the following steps:
streaming scans unprocessed data in the cache table to capture unprocessed fragments;
if unprocessed fragments are captured, acquiring to-be-processed data contained in the fragments;
and if the unprocessed fragment is not captured, rescanning the cache table.
3. The distributed text preview method of claim 1, further comprising:
and pulling the text to be processed to the local by the service server by adopting a distributed lock adding mode.
4. The distributed text preview method of claim 3, wherein after storing the text fragment to be processed in the cache table, the method further comprises:
and releasing the lock added on the text to be processed.
5. The distributed text preview method of claim 1, further comprising:
and obtaining an online transaction result from the host and feeding back the online transaction result to the service terminal.
6. A distributed text preview device, comprising:
the text fragment caching module is used for storing the text fragments to be processed into a caching table, wherein the text to be processed comprises a plurality of pieces of data to be processed;
the to-be-processed data acquisition module acquires the to-be-processed data from the cache table according to the fragments;
the text pre-inspection module is used for performing text pre-inspection on the data to be processed based on a pre-inspection rule;
and the pre-check result sending module is used for sending the data to be processed and the corresponding text pre-check result to the host so that the host can carry out online transaction on the data to be processed according to the text pre-check result.
7. The distributed text preview device of claim 6, wherein the data status of the cache table that is not subjected to text preview is marked as unprocessed, and the data status of the cache table that is subjected to text preview is marked as processed;
the module for acquiring the data to be processed comprises:
the stream type scanning unit is used for stream type scanning the unprocessed data in the cache table to capture unprocessed fragments;
the data acquisition unit is used for acquiring to-be-processed data contained in the fragments if unprocessed fragments are captured;
and the rescanning unit rescans the cache table if the unprocessed fragment is not grabbed.
8. The distributed text preview device of claim 6, further comprising:
and the data locking and pulling module is used for pulling the text to be processed to the local by the service server in a distributed locking mode.
9. The distributed text preview device of claim 8, further comprising:
and the lock releasing module is used for releasing the lock added on the text to be processed.
10. The distributed text preview device of claim 6, further comprising:
and the transaction result feedback module is used for acquiring the online transaction result from the host and feeding the online transaction result back to the service terminal.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the distributed text preview method of any of claims 1 to 5 are implemented when the program is executed by the processor.
12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the distributed text preview method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911418178.7A CN111145028A (en) | 2019-12-31 | 2019-12-31 | Distributed text pre-check method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911418178.7A CN111145028A (en) | 2019-12-31 | 2019-12-31 | Distributed text pre-check method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111145028A true CN111145028A (en) | 2020-05-12 |
Family
ID=70522837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911418178.7A Pending CN111145028A (en) | 2019-12-31 | 2019-12-31 | Distributed text pre-check method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145028A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737274A (en) * | 2020-06-19 | 2020-10-02 | 中国工商银行股份有限公司 | Transaction data processing method and device and server |
| CN111752961A (en) * | 2020-06-29 | 2020-10-09 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107508901A (en) * | 2017-09-04 | 2017-12-22 | 北京京东尚科信息技术有限公司 | Distributed data processing method, apparatus, server and system |
| CN107679140A (en) * | 2017-09-25 | 2018-02-09 | 中国银行股份有限公司 | Document handling method and system |
| CN107688500A (en) * | 2017-07-26 | 2018-02-13 | 阿里巴巴集团控股有限公司 | A kind of distributed task scheduling processing method, device, system and equipment |
| US20190286900A1 (en) * | 2018-03-16 | 2019-09-19 | Bank Of America Corporation | Dynamic Duplicate Detection |
| CN110457384A (en) * | 2019-08-15 | 2019-11-15 | 中国银行股份有限公司 | Transaction data processing method and device |
-
2019
- 2019-12-31 CN CN201911418178.7A patent/CN111145028A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107688500A (en) * | 2017-07-26 | 2018-02-13 | 阿里巴巴集团控股有限公司 | A kind of distributed task scheduling processing method, device, system and equipment |
| CN107508901A (en) * | 2017-09-04 | 2017-12-22 | 北京京东尚科信息技术有限公司 | Distributed data processing method, apparatus, server and system |
| CN107679140A (en) * | 2017-09-25 | 2018-02-09 | 中国银行股份有限公司 | Document handling method and system |
| US20190286900A1 (en) * | 2018-03-16 | 2019-09-19 | Bank Of America Corporation | Dynamic Duplicate Detection |
| CN110457384A (en) * | 2019-08-15 | 2019-11-15 | 中国银行股份有限公司 | Transaction data processing method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737274A (en) * | 2020-06-19 | 2020-10-02 | 中国工商银行股份有限公司 | Transaction data processing method and device and server |
| CN111737274B (en) * | 2020-06-19 | 2023-08-08 | 中国工商银行股份有限公司 | Transaction data processing method, device and server |
| CN111752961A (en) * | 2020-06-29 | 2020-10-09 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| US20180365254A1 (en) | Method and apparatus for processing information flow data | |
| CN108280150B (en) | Distributed asynchronous service distribution method and system | |
| CN111784329B (en) | Service data processing method and device, storage medium and electronic device | |
| CN109032796B (en) | Data processing method and device | |
| US9055065B2 (en) | Managing participant order in distributed transactions | |
| CN113094434A (en) | Database synchronization method, system, device, electronic equipment and medium | |
| CN111145028A (en) | Distributed text pre-check method and device | |
| US20240028382A1 (en) | Method and Apparatus for Starting Secure Container | |
| CN110648216A (en) | Wind control method and device | |
| CN111831461A (en) | Method and device for processing business process | |
| CN115170321A (en) | Method and device for processing batch transaction data | |
| US12086107B2 (en) | File sharing method, apparatus, and system | |
| US9632837B2 (en) | Systems and methods for system consolidation | |
| CN112685104A (en) | Application program starting control method and device, electronic equipment and storage medium | |
| CN107644020B (en) | Data storage and calling method and device | |
| CN110807058A (en) | Method and system for exporting data | |
| CN114168607A (en) | Global serial number generation method, device, equipment, medium and product | |
| CN114418702A (en) | Method and device for confirming order state to be paid based on time window | |
| CN113486032A (en) | Date switching method and device, electronic equipment and computer readable medium | |
| CN111580940A (en) | Method and device for processing multiple inter-process processes of application program and electronic equipment | |
| US11677624B2 (en) | Configuration of a server in view of a number of clients connected to the server | |
| CN112037037B (en) | Method and device for matching remittance message and position message | |
| CN115484149B (en) | Network switching method, network switching device, electronic equipment and storage medium | |
| US20240152504A1 (en) | Data interaction method, apparatus, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200512 |
|
| RJ01 | Rejection of invention patent application after publication |