CN108446301B - Business file splitting and summarizing method, device and equipment - Google Patents
Business file splitting and summarizing method, device and equipment Download PDFInfo
- Publication number
- CN108446301B CN108446301B CN201810077847.8A CN201810077847A CN108446301B CN 108446301 B CN108446301 B CN 108446301B CN 201810077847 A CN201810077847 A CN 201810077847A CN 108446301 B CN108446301 B CN 108446301B
- Authority
- CN
- China
- Prior art keywords
- service
- file
- records
- splitting
- category
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Abstract
The embodiment of the specification discloses a method, a device and equipment for splitting and summarizing service files. The service system sends the service file to be processed to the file splitting system, the service file is split into a plurality of records and then called back to the service system, the service system adds classification marks to the records according to service logic and then returns to the file splitting system, so that the service records can be classified according to the classification marks and a summary file is generated.
Description
Technical Field
The present specification relates to the field of computer technologies, and in particular, to a method, an apparatus, and a device for splitting and summarizing a service file.
Background
In the process of service processing, a plurality of service records in a service file often need to be split and then summarized.
In the prior art, when a service system needs to split and summarize files, the service system usually splits and summarizes the service files in the service system itself, or after splitting the service files into single-line service records by using a file splitting system of a third party, the service system performs summarization processing.
Based on this, a more convenient business file splitting and summarizing scheme is needed.
Disclosure of Invention
The embodiment of the specification provides a method, a device and equipment for splitting and summarizing service files, which are used for solving the following problems: so as to provide a more convenient service experience implementation scheme.
Based on this, an embodiment of the present specification provides a method for splitting and summarizing a service file, including:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
and receiving a plurality of summarized files carrying the category identification returned by the file splitting system.
Meanwhile, an embodiment of the present specification provides another method for splitting and summarizing a service file, including:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
Correspondingly, an embodiment of the present specification further provides a device for splitting and summarizing a service file, including:
the system comprises a sending module, a file splitting system and a processing module, wherein the sending module is used for sending a service file to be processed to the file splitting system so that the file splitting system can split the service file to be processed and return a service record, and the service file consists of a plurality of service records;
the receiving module is used for receiving a plurality of service records returned by the file splitting system;
the confirming module confirms the category identification of each service record according to the service logic;
the sending module is further configured to send the service records carrying the category identifiers to the file splitting system, so that the file splitting system classifies the service records and generates a plurality of summary files carrying the category identifiers;
the receiving module is further configured to receive the plurality of summarized files carrying the category identifier returned by the file splitting system.
Correspondingly, an embodiment of the present specification further provides another device for splitting and summarizing a service file, including:
the receiving module is used for receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
the splitting module is used for splitting the service file to be processed into a plurality of service records;
the sending module returns the service record to the service system;
the receiving module is further used for receiving the service record which is returned by the service system and carries the category identification;
the generating module generates a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
the sending module is further configured to return the summary file carrying the category identifier to the service system.
Correspondingly, an embodiment of the present specification further provides a device for splitting and summarizing a service file, including:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
and receiving a plurality of summarized files carrying the category identification returned by the file splitting system.
Correspondingly, an embodiment of the present specification further provides another device for splitting and summarizing a service file, including:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
Correspondingly, embodiments of the present specification also provide a non-volatile computer storage medium storing computer-executable instructions configured to:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
and receiving a plurality of summarized files carrying the category identification returned by the file splitting system.
Correspondingly, embodiments of the present description also provide another non-volatile computer storage medium storing computer-executable instructions configured to:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
The embodiment of the specification adopts at least one technical scheme which can achieve the following beneficial effects:
and sending the service file to be processed to a file splitting system, splitting the file into a plurality of records, calling back to the service system, adding a classification identifier to the records by the service system according to service logic, and returning to the file splitting system, so that the service records can be classified according to the classification identifier and a summary file is generated. By the scheme, the file splitting system does not need to understand the service logic of the service system, and can perform summary processing according to the mode specified by the service system, so that the file splitting system is more convenient. In addition, by inputting format information, the method can adapt to different types of service files; the file splitting system can perform speed control, current limiting and other operations on the speed of the callback of the service record, is suitable for the performance of the service system and is more convenient. Finally, the file splitting system ensures that the whole process is not interrupted through the self recovery mechanism and stability; the service system can also screen the summary result according to actual needs and appoint the needed result; the service system and the file splitting system both support the processing mode of the cluster, and the processing capacity can be improved by adding machines.
Drawings
FIG. 1 is a schematic diagram of a process for file splitting using a file splitting system in the prior art;
FIG. 2 is a flow diagram illustrating aspects of a business system provided by embodiments of the present disclosure;
FIG. 3 is a schematic flow diagram illustrating aspects of a file splitting system provided by an embodiment of the present description;
FIG. 4 is a schematic diagram of an exemplary business process flow provided by embodiments of the present description;
fig. 5 is a schematic structural diagram of a service file splitting and summarizing device in the aspect of a service system provided in an embodiment of the present specification;
fig. 6 is a schematic structural diagram of a business file splitting and aggregating apparatus in the aspect of a file splitting system according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step are within the scope of the present application.
In the prior art, when splitting and summarizing files, the splitting and summarizing can be performed in the service system itself, and the method generally includes the following steps: 1, reading a service file in a memory in a stream mode, and storing each line of the file into a database. And 2, reading all records, grouping according to the service logic, and storing the grouped records into a database. And 3, independently writing the content of each group into a file. The above solution is clearly simple in logic, but in practice has many drawbacks: splitting and summarizing are carried out in a single machine of a service system, and if the machine is hung or files are too large and the memory is insufficient, the flow is interrupted; the splitting of the service file and the grouping processing of the service logic are two times of processing, and the processing efficiency is low; the service system generally does not set a recovery mechanism in an abnormal state for splitting, if the recovery mechanism is set, the resource consumption is large, and the recovery mechanism cannot be used in other systems, and the like. Therefore, in practice, a special file splitting system is adopted for splitting and summarizing, and the file splitting system is provided with a recovery mechanism, so that the normal operation of the process can be guaranteed. The process comprises the following steps: as shown in fig. 1, fig. 1 is a schematic flow diagram of file splitting performed by using a file splitting system in the prior art, and mainly includes the following steps:
s101, a service system sends a service file to a file splitting system;
s103, the file splitting system splits the service file to obtain a service record;
s105, returning the service record to the service system;
s107, the service system traverses the service records according to the service logic, processes, groups and stores;
and S109, traversing and grouping by the service system, and sequentially generating a summary file.
In the scheme, when the file is split, all the file contents are traversed to be split, and the file splitting system returns the service records one by one. When the service system obtains all the service records and performs service logic processing on a single service record and generates a summary file by itself, an exception may be generated due to the strong coupling service logic. The greater the amount of data (e.g., billions of payment records or transaction records, etc.), the less secure it is to do so at the business system. Because business systems do not have recovery mechanism functions such as breakpoint recovery, sub-task splitting, etc. (or, business systems are built to be focused on business services and not on file processing), errors are more likely to occur.
Based on this, the embodiments of the present specification provide a scheme for splitting and summarizing a service file, where the service file to be processed is sent to a file splitting system, split into multiple records, and then call back to the service system, and the service system adds a classification identifier to the service record according to service logic, and then returns to the file splitting system, so that the service record can be classified according to the classification identifier, and a summary file is generated.
The method for splitting and summarizing the service files provided by the embodiment of the present specification includes two aspects of a service system and a file splitting system, and as for the aspect of the service system, a flow schematic diagram executed by the service system is shown in fig. 2, and fig. 2 is the flow schematic diagram of the aspect of the service system provided by the embodiment of the present specification, and includes:
s201, sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file is composed of a plurality of service records.
The business file may be a business file such as a bid record, phone record, consumption record, payment record, transaction record, etc. It generally consists of a plurality of service records, each of which contains a plurality of fields (or column names) separated by a specific separator. In other words, the service file may have a variety of different formats.
S203, receiving a plurality of service records returned by the file splitting system. The received service record can be in a form of returning one by one or in a form of grouping a plurality of pieces.
S205, according to the service logic, confirming the category identification of each service record.
The service logic and the class identification can be determined according to actual needs. For example, for a certain bidding document, each record of the bidding document represents a bidding user, and it is necessary to determine whether each service record meets the bidding condition, the service logic at this time is to perform logic judgment on each field in the service record, and comprehensively evaluate the service record, if the condition is met, a category identifier "pass" is added, otherwise, a category identifier "fail" is added to the service record.
For another example, for the same bidding document, it is desired to summarize which bidding users exist in each province, and the business logic at this time is to determine a specific field in the business record, determine the province corresponding to the business record, and add corresponding classification identifiers such as "shanghai", "hunan", and "beijing" to each record for subsequent classification.
And S207, sending the service records carrying the class identifiers to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifiers.
S209, receiving a plurality of summary files carrying the category identifiers returned by the file splitting system.
As a specific implementation manner, for the step S201, sending the service file to be processed to the file splitting system, the method includes: and sending the service file to be processed and format information to a file splitting system, wherein the format information comprises an input format and a specified output format.
The input format is the format of the service file to be processed. Because the business files have various formats, the format of the input file is specified, so that the splitting system can identify the file, and the formats are specified before splitting the file every time, and the business processing is not influenced mutually, so that the whole processing mode has wider adaptability and is more flexible. For example: the file 1 format is: name, age, hobby; the format of the phone file 2 is: address, telephone, identification number. The file splitting system is enabled to correctly recognize by specifying in the input format. In addition, a Storage medium that specifies a Service file to be processed may be included in the format of the input file, for example, an Object Storage Service (OSS) address, a network disk link, a download address, and the like.
The specified output format functions similarly to the input format. But with further functions, such as name, age, hobbies, for the format entered for file 1. In this case, the output format can be specified as skiing hobby, name, and age. That is, the fields in the output format may be a subset of the fields in the input format to sort the files.
As a specific implementation manner, for the step S203, receiving the service record returned by the file splitting system includes: and receiving a subtask returned by the file splitting system, wherein the subtask consists of service records. Namely, the splitting system groups the split service records to obtain subtasks. The grouping rules may be preset according to actual needs, processing capabilities of the service systems, cluster states of the service systems, and the like. Thus, the file splitting system may assign the subtasks to multiple machines in the business system to be executed separately. In other words, by forming the service records into subtasks, the service system can support the processing of the whole service in a cluster manner, thereby improving the efficiency. Of course, there may be a service system to split the system for active fetching. In this processing manner, for the step S205, confirming the category identifier of the service record according to the service logic includes: and confirming the category identification of the service record contained in the subtasks item by item according to the service logic.
As a specific implementation manner, after obtaining a summary file carrying the classification identifier assembled by the splitting system, at this time, the service system may further preset a rule to summarize a required result. That is, the above scheme may further include: and screening the plurality of summary files carrying the category identification according to a preset condition to obtain the summary files meeting the preset condition, wherein the preset condition is related to the category identification.
Generally, the keywords in the preset condition generally have an inclusion or included relationship with the category identifier. For example, after splitting and aggregating the transaction file, four aggregated files carrying the region identifier are obtained, which respectively include: shanghai transaction summary documents, Hangzhou transaction summary documents, Beijing transaction summary documents, and Harbin transaction summary documents. At this time, if the business system focuses on the transaction situation of the southern city, the business system only needs to traverse and judge the region by presetting the KEY value of the southern city, and take out transaction summary files of Hangzhou and Shanghai.
In the method for splitting and summarizing the service file provided in the embodiment of the present specification, for the aspect of the file splitting system, the process schematic diagram executed by the method is shown in fig. 3, and fig. 3 is the process schematic diagram for the aspect of the file splitting system provided in the embodiment of the present specification, and includes:
s301, receiving a service file to be processed sent by a service system, wherein the service file is composed of a plurality of service records.
S303, splitting the service file to be processed into a plurality of service records.
When the specific splitting is performed, the splitting can be performed correspondingly according to the format of the service file to be processed. Generally, the format of the service file is a line record, and a line splitting is correspondingly adopted. If the format of the service file is column type record, column splitting can be correspondingly adopted.
S305, returning the service record to the service system.
And S307, receiving the service record carrying the category identifier returned by the service system.
S309, generating a plurality of summary files carrying the category identifiers according to the service records carrying the category identifiers. In the file splitting system, the stability is ensured due to the self recovery mechanism. That is, if the summary file is generated in failure, the processes such as retry and abandon can be adopted, and the problem of process blocking after the single machine goes down can not occur. In addition, when the file splitting system splits a service file or generates a summary file, a cluster mode can be adopted, and the capacity of processing the service is improved by increasing the number of machines.
S311, returning the summary file carrying the category identification to the service system.
As a specific implementation manner, for the receiving of the service file to be processed sent by the service system in S301, the method includes: receiving a service file to be processed and format information sent by a service system, wherein the format information comprises an input format and a specified output format, and identifying the service file to be processed according to the input format. Therefore, different types of service files can be identified according to the input format for splitting. Further, when generating a summary file, a summary file having a specified format may be generated. The specific input format and the specific output format have already been described in the foregoing, and are not described in detail here.
As a specific implementation manner, in S303, after splitting the service file to be processed into a plurality of service records, the method further includes: and grouping the plurality of service records to generate a subtask containing the service record. Thus for S305, returning the business record to the business system includes: and returning the subtasks to the service system. The time or number of the return subtasks of the file splitting system and the specific manner of the return subtasks to each machine in the service system cluster can be preset based on the processing capacity of the service system.
Specifically, the file splitting system processes the subtasks in a cluster mode, sends the subtasks to each machine in the cluster, and after each machine pulls the subtasks from the system, the pulled subtasks are divided into a plurality of units to be processed in each machine according to actual requirements such as speed limit and flow control, so that the units to be processed are sent to corresponding processing machines in the service system cluster, and processing efficiency is improved.
As a specific implementation manner, for S305, returning the service record to the service system includes: and returning the service records within the specified quantity range within the specified time interval. For example, 10000 to 11000 consumption records are returned within 1 minute. In other words, when returning to the service record, the operations of speed control, current limitation, etc. can be performed to adapt to the actual needs.
According to the scheme provided by the embodiment of the specification, the service file to be processed is sent to the file splitting system, split into a plurality of records and then called back to the service system, the service system adds the classification identifier to the record according to the service logic and then returns to the file splitting system, so that the service record can be classified according to the classification identifier and the summary file is generated. By the scheme, the file splitting system does not need to understand the service logic of the service system, and can perform summary processing according to the mode specified by the service system, so that the file splitting system is more convenient. In addition, by inputting format information, the method can adapt to different types of service files; the file splitting system can perform speed control, current limiting and other operations on the speed of the callback of the service record, is suitable for the performance of the service system and is more convenient. Finally, the file splitting system ensures that the whole process is not interrupted through the self recovery mechanism and stability; the service system can also screen the summary result according to actual needs and appoint the needed result; the service system and the file splitting system both support the processing mode of the cluster, and the processing capacity can be improved by adding machines.
In order to make the embodiments of the present disclosure more clearly understood, a specific application example is given below to describe the solution provided in the present disclosure, as shown in fig. 4, and fig. 4 is an exemplary business process flow diagram provided in the embodiments of the present disclosure.
A company carries out a supplier tenderer bid through a tenderer system, and a tenderer system (i.e., a business system) obtains bid documents related to bid records of suppliers nationwide, and now desires to obtain summary documents of suppliers meeting bid conditions. Thereby comprising the steps of:
s401, the tenderer system sends the bid file to a file splitting system;
s403, the file splitting system splits the bid files and stores the bid files in groups, wherein each group is a subtask containing a certain number of bid records;
s405, the file splitting system distributes subtasks to all machines in the cluster;
s407, each machine divides the distributed subtasks into a plurality of units to be processed;
s409, each machine sends the unit to be processed to each processing machine in the business recruitment system cluster;
s411, the tenderer system judges whether each bidding record meets the tendering condition according to the service logic, and adds a classification mark 'A' to the tendering record meeting the condition; adding a classification mark 'B' to the bidding records which do not meet the condition;
s413, the tenderer system sends the bidding records carrying the classification identifiers to a file splitting system;
s415, the file splitting system carries out bidding record classification according to the classification marks to generate a class A summary file and a class B summary file;
s417, the file splitting system sends the A-type summary file and the B-type summary file to the recruiter system;
s419, the recruiter system screens the A-type summary file.
In the process, the file splitting and bidding and the assembling and summarizing of the files are carried out in the file splitting system, the business processing logic of the business system does not need to be understood, the coupling with the business system is avoided, and the generation of the related business summarizing files is more convenient.
Based on the same idea, the present invention further provides a device for splitting and summarizing service files, as shown in fig. 5, fig. 5 is a schematic structural diagram of the device for splitting and summarizing service files in the aspect of a service system provided in the embodiment of the present specification, and the device includes:
a sending module 501, configured to send a service file to be processed to a file splitting system, so that the file splitting system splits the service file to be processed and returns a service record, where the service file is composed of multiple service records;
a receiving module 503, configured to receive multiple service records returned by the file splitting system;
a confirming module 505, configured to confirm the category identifier of each service record according to the service logic;
the sending module 501 is further configured to send the service records with the category identifiers to the file splitting system, so that the file splitting system classifies the service records and generates a plurality of summary files with the category identifiers;
the receiving module 503 is further configured to receive a plurality of summarized files carrying the category identifier, which are returned by the file splitting system.
Further, the sending module 501 sends the service file to be processed and format information to a file splitting system, where the format information includes an input format and a specified output format.
Further, the receiving module 503 receives a subtask returned by the file splitting system, where the subtask consists of a service record; the confirming module 505 confirms the category identifiers of the service records included in the subtasks one by one according to the service logic.
Further, the apparatus further includes a screening module 507, which screens the plurality of summary files carrying the category identifier according to a preset condition to obtain summary files meeting the preset condition, where the preset condition is related to the category identifier.
Based on the same idea, the present invention further provides another device for splitting and summarizing service files, as shown in fig. 6, fig. 6 is a schematic structural diagram of the device for splitting and summarizing service files in the aspect of a file splitting system provided in the embodiment of the present specification, and the device includes:
the receiving module 601 is configured to receive a service file to be processed, which is sent by a service system, wherein the service file is composed of a plurality of service records;
a splitting module 603, configured to split the service file to be processed into multiple service records;
a sending module 605, configured to return the service record to the service system;
the receiving module 601 is further configured to receive a service record carrying a category identifier, which is returned by the service system;
the generating module 607 generates a plurality of summary files carrying category identifiers according to the service records carrying category identifiers;
the sending module 605 is further configured to return the summary file carrying the category identifier to the service system.
Further, the receiving module 601 receives a to-be-processed service file and format information sent by a service system, where the format information includes an input format and a specified output format, so as to identify the to-be-processed service file according to the input format; the generating module 607 generates a plurality of summary files carrying category identifiers in a specified output format according to the service record carrying category identifier.
Further, the apparatus further includes a subtask generating module 609, which groups the plurality of service records to generate a subtask including a service record; the sending module 605 returns the subtasks to the service system.
Further, the sending module 605 returns the service records within the specified number range in the specified time interval.
Correspondingly, an embodiment of the present specification further provides a device for splitting and summarizing a service file, including:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
and receiving a plurality of summarized files carrying the category identification returned by the file splitting system.
Correspondingly, an embodiment of the present specification further provides another device for splitting and summarizing a service file, including:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
Based on the same inventive concept, embodiments of the present application further provide a corresponding non-volatile computer storage medium, in which computer-executable instructions are stored, where the computer-executable instructions are configured to:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
and receiving a plurality of summarized files carrying the category identification returned by the file splitting system.
Based on the same inventive concept, the embodiment of the present application further provides another corresponding non-volatile computer storage medium, in which computer-executable instructions are stored, where the computer-executable instructions are set as:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Especially, as for the device, apparatus and medium type embodiments, since they are basically similar to the method embodiments, the description is simple, and the related points may refer to part of the description of the method embodiments, which is not repeated here.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps or modules recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the units may be implemented in the same software and/or hardware or in one or more pieces of software and/or hardware when implementing the embodiments of the present description.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media (transient media) such as modulated data signal numbers and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
Embodiments of the present description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular transactions or implement particular abstract data types. Embodiments of the present description may also be practiced in distributed computing environments where transactions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Claims (16)
1. A method for splitting and summarizing service files comprises the following steps:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
receiving a plurality of summarized files carrying category identifications returned by the file splitting system;
receiving a service record returned by the file splitting system, wherein the service record comprises:
receiving a subtask returned by the file splitting system, wherein the subtask consists of a service record;
the determining the category identifier of each service record according to the service logic includes:
and confirming the category identification of the service record contained in the subtasks item by item according to the service logic.
2. The method of claim 1, sending a service file containing pending transactions to a file splitting system, comprising:
and sending the service file to be processed and format information to a file splitting system, wherein the format information comprises an input format and a specified output format.
3. The method of claim 1, after receiving a plurality of summarized files carrying class identifiers returned by the file splitting system, further comprising:
and screening the plurality of summary files carrying the category identification according to a preset condition to obtain the summary files meeting the preset condition, wherein the preset condition is related to the category identification.
4. A method for splitting and summarizing service files comprises the following steps:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
5. The method of claim 4, receiving the pending service file sent by the service system, comprising:
receiving a service file to be processed and format information sent by a service system, wherein the format information comprises an input format and a specified output format so as to identify the service file to be processed according to the input format;
generating a plurality of summary files carrying the category identifiers according to the service records carrying the category identifiers, including:
and generating a plurality of summary files carrying the category identifications in a specified output format according to the service records carrying the category identifications.
6. The method of claim 4, after splitting the service file to be processed into a plurality of service records, further comprising:
grouping the plurality of service records to generate a subtask containing the service records;
the returning the service record to the service system includes:
and returning the subtasks to the service system.
7. The method of claim 4, returning the business record to the business system, comprising:
and returning the service records within the specified quantity range within the specified time interval.
8. A business file splitting and summarizing device comprises:
the system comprises a sending module, a file splitting system and a processing module, wherein the sending module is used for sending a service file to be processed to the file splitting system so that the file splitting system can split the service file to be processed and return a service record, and the service file consists of a plurality of service records;
the receiving module is used for receiving a plurality of service records returned by the file splitting system;
the confirming module confirms the category identification of each service record according to the service logic;
the sending module is further configured to send the service records carrying the category identifiers to the file splitting system, so that the file splitting system classifies the service records and generates a plurality of summary files carrying the category identifiers;
the receiving module is further configured to receive the plurality of summarized files carrying the category identifiers, which are returned by the file splitting system;
the receiving module receives a subtask returned by the file splitting system, wherein the subtask consists of a service record;
and the confirming module confirms the category identification of the service record contained in the subtasks one by one according to the service logic.
9. The apparatus of claim 8, wherein the sending module sends the service file to be processed and format information to a file splitting system, and the format information includes an input format and a specified output format.
10. The apparatus according to claim 8, further comprising a screening module, configured to screen the plurality of summarized files carrying the category identifier according to a preset condition, so as to obtain summarized files meeting the preset condition, where the preset condition is related to the category identifier.
11. A business file splitting and summarizing device comprises:
the receiving module is used for receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
the splitting module is used for splitting the service file to be processed into a plurality of service records;
the sending module returns the service record to the service system;
the receiving module is further used for receiving the service record which is returned by the service system and carries the category identification;
the generating module generates a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
the sending module is further configured to return the summary file carrying the category identifier to the service system.
12. The apparatus of claim 11, wherein the receiving module receives a service file to be processed and format information sent by a service system, wherein the format information includes an input format and a specified output format, so as to identify the service file to be processed according to the input format; and the generating module generates a plurality of summarized files carrying the category identifications in a specified output format according to the service records carrying the category identifications.
13. The apparatus of claim 11, further comprising a subtask generation module that groups the plurality of business records to generate a subtask containing a business record; and the sending module returns the subtasks to the service system.
14. The apparatus of claim 11, the sending module returns a specified number of service records within a specified time interval.
15. A business file splitting and summarizing device comprises:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
sending a service file to be processed to a file splitting system so that the file splitting system can split the service file to be processed and return a service record, wherein the service file consists of a plurality of service records;
receiving a plurality of service records returned by the file splitting system;
confirming the category identification of each service record according to service logic;
sending the service records carrying the class identifications to the file splitting system so that the file splitting system can classify the service records and generate a plurality of summary files carrying the class identifications;
receiving a plurality of summarized files carrying category identifications returned by the file splitting system;
receiving a service record returned by the file splitting system, wherein the service record comprises:
receiving a subtask returned by the file splitting system, wherein the subtask consists of a service record;
the determining the category identifier of each service record according to the service logic includes:
and confirming the category identification of the service record contained in the subtasks item by item according to the service logic.
16. A business file splitting and summarizing device comprises:
the memory stores a business file splitting and summarizing program;
the processor calls the service file splitting and summarizing program in the memory and executes:
receiving a service file to be processed, which is sent by a service system, wherein the service file consists of a plurality of service records;
splitting the service file to be processed into a plurality of service records;
returning the service record to the service system;
receiving a service record carrying a category identifier returned by a service system;
generating a plurality of summary files carrying the category identifications according to the service records carrying the category identifications;
and returning the summary file carrying the category identification to the service system.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810077847.8A CN108446301B (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
| CN202111395669.1A CN113961523A (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810077847.8A CN108446301B (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111395669.1A Division CN113961523A (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108446301A CN108446301A (en) | 2018-08-24 |
| CN108446301B true CN108446301B (en) | 2021-10-29 |
Family
ID=63191025
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810077847.8A Active CN108446301B (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
| CN202111395669.1A Pending CN113961523A (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111395669.1A Pending CN113961523A (en) | 2018-01-26 | 2018-01-26 | Business file splitting and summarizing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN108446301B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110245115B (en) * | 2019-05-28 | 2023-06-20 | 创新先进技术有限公司 | File processing method, engine and system |
| CN112446770B (en) * | 2020-12-15 | 2022-09-23 | 天津普泽工程咨询有限责任公司 | Bidding method and system |
| CN117709909B (en) * | 2024-02-06 | 2024-04-09 | 绵阳师范学院 | Business data processing method based on large language model |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102231756A (en) * | 2011-06-28 | 2011-11-02 | 绍兴文理学院元培学院 | Internet literature scientific research data collection system |
| CN103200269A (en) * | 2013-04-12 | 2013-07-10 | 亿赞普(北京)科技有限公司 | Internet information statistical method and Internet information statistical system |
| CN103605715A (en) * | 2013-11-14 | 2014-02-26 | 北京国双科技有限公司 | Method and device used for data integration processing of multiple data sources |
| CN106294683A (en) * | 2016-08-05 | 2017-01-04 | 中国银行股份有限公司 | A kind of file declustering method and device |
| CN106528874A (en) * | 2016-12-08 | 2017-03-22 | 重庆邮电大学 | Spark memory computing big data platform-based CLR multi-label data classification method |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
| CN107391551A (en) * | 2017-06-06 | 2017-11-24 | 广东广业开元科技有限公司 | A kind of web business datums analysis method and system based on data mining |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7962452B2 (en) * | 2007-12-28 | 2011-06-14 | International Business Machines Corporation | Data deduplication by separating data from meta data |
-
2018
- 2018-01-26 CN CN201810077847.8A patent/CN108446301B/en active Active
- 2018-01-26 CN CN202111395669.1A patent/CN113961523A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102231756A (en) * | 2011-06-28 | 2011-11-02 | 绍兴文理学院元培学院 | Internet literature scientific research data collection system |
| CN103200269A (en) * | 2013-04-12 | 2013-07-10 | 亿赞普(北京)科技有限公司 | Internet information statistical method and Internet information statistical system |
| CN103605715A (en) * | 2013-11-14 | 2014-02-26 | 北京国双科技有限公司 | Method and device used for data integration processing of multiple data sources |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
| CN106294683A (en) * | 2016-08-05 | 2017-01-04 | 中国银行股份有限公司 | A kind of file declustering method and device |
| CN106528874A (en) * | 2016-12-08 | 2017-03-22 | 重庆邮电大学 | Spark memory computing big data platform-based CLR multi-label data classification method |
| CN107391551A (en) * | 2017-06-06 | 2017-11-24 | 广东广业开元科技有限公司 | A kind of web business datums analysis method and system based on data mining |
Non-Patent Citations (1)
| Title |
|---|
| 分布式虚拟化计算平台高可靠任务拆分系统的设计与实现;孔舟;《中国优秀硕士学位论文全文数据库 信息科技辑》;20111215;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108446301A (en) | 2018-08-24 |
| CN113961523A (en) | 2022-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107450979B (en) | Block chain consensus method and device | |
| CN107239324B (en) | Service flow processing method, device and system | |
| CN107391526B (en) | Data processing method and device based on block chain | |
| CN107391527B (en) | Data processing method and device based on block chain | |
| CN107577697B (en) | Data processing method, device and equipment | |
| CN106899666B (en) | Data processing method and device for service identification | |
| CN108599973B (en) | Log association method, device and equipment | |
| CN107066519B (en) | Task detection method and device | |
| CN108446301B (en) | Business file splitting and summarizing method, device and equipment | |
| CN111124476B (en) | Business rule configuration and creation method and device | |
| CN110008018A (en) | A kind of batch tasks processing method, device and equipment | |
| CN109104327B (en) | Service log generation method, device and equipment | |
| US11074246B2 (en) | Cluster-based random walk processing | |
| CN111245907B (en) | Event processing method and system based on rule matching | |
| CN107451204B (en) | Data query method, device and equipment | |
| CN114529417A (en) | Method for executing transaction, block chain, main node and slave node | |
| CN110297955B (en) | Information query method, device, equipment and medium | |
| CN112003769A (en) | Message processing method, device and equipment | |
| CN112015569A (en) | Message reminding processing method and device | |
| CN109376988B (en) | Service data processing method and device | |
| CN112860412A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN113256426A (en) | Data processing method, device, equipment and medium | |
| CN115878654A (en) | Data query method, device, equipment and storage medium | |
| CN110245136B (en) | Data retrieval method, device, equipment and storage equipment | |
| CN110633321B (en) | Data synchronization method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20201021 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20201021 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |