CN111475291A - Data processing method, system, server and medium - Google Patents
Data processing method, system, server and medium Download PDFInfo
- Publication number
- CN111475291A CN111475291A CN202010229525.8A CN202010229525A CN111475291A CN 111475291 A CN111475291 A CN 111475291A CN 202010229525 A CN202010229525 A CN 202010229525A CN 111475291 A CN111475291 A CN 111475291A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- storage directory
- server
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application is applicable to the technical field of data processing, and provides a data processing method, a system, a server and a medium, wherein the method comprises the following steps: when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction; determining a storage directory of the data to be processed; and distributing the data to be processed to each operation server according to the storage directory of the data to be processed so as to instruct each operation server to generate a component report according to the data to be processed and send the component report to a summary server for summary. By the method, a large amount of data can be stored in order, and the data can be processed efficiently.
Description
Technical Field
The present application belongs to the field of data processing technologies, and in particular, to a data processing method, system, server, and medium.
Background
With the development of informatization, the amount of data generated in the process of using computers or other devices to perform tasks is increasing.
In practical application, in order to understand the task execution situation and to clear the shortages of the current business, a large amount of data needs to be summarized and analyzed, but because the amount of data generated in the task execution process is large, if statistics of the large amount of data needs to be performed, the time consumption is very long, and the efficiency is very low.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing system, a server and a medium, which can solve the problem of low efficiency in processing a large amount of data.
In a first aspect, an embodiment of the present application provides a data processing method, including:
when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction;
determining a storage directory of the data to be processed;
and distributing the data to be processed to each operation server according to the storage directory of the data to be processed so as to instruct each operation server to generate a component report according to the data to be processed and send the component report to a summary server for summary.
In a second aspect, an embodiment of the present application provides a data processing system, including:
the temporary storage server is used for acquiring data to be processed according to the data summarizing instruction when the data summarizing instruction is received; determining a storage directory of the data to be processed; distributing the data to be processed to each operation server according to the storage catalog of the data to be processed; indicating each operation server to generate a component report according to the data to be processed and sending the component report to a summary server for summary;
the operation server is used for receiving the data to be processed distributed by the temporary storage server, generating a sub report according to the data to be processed and sending the sub report to the summary server;
the summarizing server is used for summarizing the sub-report tables reported by the operation servers and uploading a summarizing result to the output server;
and the output server is used for outputting the summary result.
In a third aspect, an embodiment of the present application provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the method according to the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to the first aspect.
In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on a server, causes the server to perform the method described in the first aspect.
Compared with the prior art, the embodiment of the application has the advantages that: in the embodiment of the application, after the temporary storage server acquires the data to be processed, the temporary storage server can determine the storage catalog of the data to be processed according to the acquisition time and serial number of the data to be processed, and uniformly distribute the data to be processed to each operation server according to the storage catalog; each operation server processes the data to be processed and generates a sub report, and then the sub report is sent to the summarizing server; the summarizing server summarizes the sub-report table and uploads a summarizing result to the output server; and the output server outputs the summary result when receiving the instruction. In this embodiment, the data to be processed is stored in different storage catalogues, which is equivalent to grouping the data to be processed for multiple times, so that a large amount of data can be managed and searched conveniently; and then the data to be processed is distributed to each operation server according to the grouping result, which is equivalent to splitting the processing of a large amount of data, and then the data is processed by a plurality of servers simultaneously, thereby improving the efficiency of data processing.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data processing method according to a second embodiment of the present application;
fig. 3 is a schematic flowchart of a data processing method according to a third embodiment of the present application;
FIG. 4 is a schematic diagram of a data processing system according to a fourth embodiment of the present application;
fig. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application, and as shown in fig. 1, the method includes:
s101, when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction;
the execution main body of the embodiment of the application is a temporary storage server in a data processing system, and the temporary storage server can be a computing device capable of acquiring, storing and transmitting data and comprises a cloud server.
In performing business using a computer, a lot of data is generated. For example, a company sends information to a destination terminal according to a client request, the generated data may include information content, information sending time, information sending quantity, whether the information is sent successfully or not, failure reasons, client information, destination terminal information and the like in the process of sending the information, a server may send the generated data to a Kafka platform after the information is sent, and the Kafka is a high-throughput distributed publish-subscribe messaging system which can process all action flow data in a website of a consumer scale.
When the temporary storage server receives the data summarizing instruction, the to-be-processed data can be acquired from the Kafka platform according to the data summarizing instruction. In addition, the temporary storage server can also acquire data from the Kafka platform according to the set timing.
S102, determining a storage directory of the data to be processed;
specifically, after the temporary storage server obtains data from the Kafka platform, the data needs to be stored, and in order to enable the stored data to be stored in the server in order, the data may be stored in different directories. The temporary storage server can group the data to be processed according to the information of the acquisition time, the data type, the state attribute and the like of the data to be processed, then create a plurality of root storage directories according to the grouping result, create a plurality of sub-storage directories under the root storage directories, and store the data to be processed under each corresponding sub-storage directory according to different groups.
S103, distributing the data to be processed to each operation server according to the storage directory of the data to be processed so as to instruct each operation server to generate a component report according to the data to be processed and send the component report to a summary server for summary.
Specifically, the data to be processed may be distributed to the respective arithmetic servers according to the storage directory of the data to be processed. When the storage directories of the data to be processed are determined, the data to be processed can be uniformly distributed under a certain level of directory, so that the amount of the data to be processed received by each operation server can be balanced when the data to be processed is distributed to each operation server according to the level of directory.
Each operation server collects the stored data, generates a sub report and then sends the sub report to a collection server; the summarizing server summarizes each sub report table and uploads a summarizing result to the output server; when the output server receives the data output instruction, the summarizing result corresponding to the data output instruction can be output.
In the embodiment, a large amount of data can be evenly sent to each operation server according to the storage directory, so that computing resources can be reasonably distributed; the data are processed by the plurality of operation servers, so that the data processing efficiency is improved.
Fig. 2 is a schematic flowchart of a data processing method provided in the second embodiment of the present application, and as shown in fig. 2, the method includes:
s201, when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction;
the execution subject is a temporary storage server, which may be a computing device capable of acquiring, storing and transmitting data, including a cloud server.
With the development of internet technology and information technology, many services can be completed by a computer, such as sending information, ticket booking, service reservation and the like, and these services can be performed on a server, and during the operation of the server, many data can be generated, including service execution results, failure reasons and the like, and the information contained in these data can help an enterprise to know the service status of a company, and is beneficial to maintaining the server and improving programs. Wherein the data may be logs generated during operation of the computing device.
The data to be processed may include data generated during the execution of the service, and the data may be stored on an intermediate platform, such as a Kafka platform. After receiving the data summarization instruction, the temporary storage server may obtain data from the Kafka platform.
In addition, the temporary storage server can also acquire data at regular time.
S202, determining a first storage directory of the data to be processed according to the acquisition time of the data to be processed;
specifically, the temporary storage server may obtain the time when the data to be processed is accurately obtained, identify the year, month, day, and hour in the data obtained data, create a root directory with information corresponding to the year as a name in the obtained data, sequentially create storage directories corresponding to the month information, the day information, and the hour information under the root directory, and use the storage directory corresponding to the hour information as the first storage directory.
Illustratively, if a to-be-processed data is obtained by the temporary storage server in 2019, 10, 28, and 14, the first storage directory of the to-be-processed data may be 2019\10\28\ 14'.
It should be noted that the minimum unit of the directory established according to the acquisition time may be adjusted according to the amount of the data, for example, when the amount of the data is small, the date information may also be used as the minimum storage directory in the first storage directory; when the volume of data is very large, the acquisition time can be accurately divided, a storage directory corresponding to the division information is established under the storage directory corresponding to the time information, and the storage directory corresponding to the division information is used as a first storage directory
S203, determining a second storage directory of the data to be processed according to the serial number of the data to be processed;
when the data to be processed is stored on the Kafka platform, the Kafka platform configures a serial number for the data, wherein the serial number is a continuous number, and one serial number points to one piece of data. For example, the serial number may be a 20-bit integer number, and the serial number of a certain data to be processed may be-423803484277086432.
Specifically, when the serial number is used to determine the second storage directory of the data to be processed, the user may set a modulus according to the number of the preset operation servers, divide the modulus by the serial number to obtain the remainder, and then use the value obtained by adding 1 to the absolute value of the remainder as the name of the second storage directory. Of course, other calculation methods may be used to determine the second storage directory using the serial number. The name of the second storage directory may also be selected from other legal forms corresponding to the value.
For example, a second storage directory of the data to be processed is determined according to the serial number-423803484277086432, if the modulo set according to the preset number of the operation servers is 20, -423803484277086432, the remainder obtained according to the modulo 20 is-1, -1 is 1 in absolute value, and 2 is obtained by adding 1 to 1, so the name of the second storage directory of the data to be processed can be 02.
S204, determining the storage directory of the data to be processed according to the first storage directory and the second storage directory.
Specifically, the names of the first storage directory and the second storage directory of the to-be-processed data are determined according to the steps S202 and S203, and a directory obtained by adding the second storage directory to the first storage directory may be used as the storage directory corresponding to the to-be-processed data. For example, if the first storage directory of the data to be processed is 2019\10\28\14\ and the second storage directory is 02, the storage directory of the data to be processed is 2019\10\28\14\ 02.
Certainly, if the temporary storage server already has a storage directory corresponding to the data to be processed, the data to be processed can be directly stored in the directory; if the temporary storage server does not have a storage directory corresponding to the data to be processed, the corresponding storage directory may be created first, and then the data to be processed is stored in the corresponding storage directory.
S205, determining an operation server corresponding to the second storage directory;
specifically, the data processing system may include a plurality of calculation servers, and the plurality of calculation servers may all perform data processing.
When the second storage directory is determined, the preset modulus may be equal to the number of the operation servers participating in data processing, and then the number of remainders obtained by the serial number of the data to be processed according to the preset modulus is equal to the number of the operation servers participating in data processing, so that the obtained second storage directory may correspond to the operation servers participating in data processing one to one. Therefore, the operation server corresponding to the data can be determined according to the second storage catalogue of the data to be processed.
And S206, distributing the data to be processed to the operation server corresponding to the second storage directory.
And distributing the data to be processed to the operation server corresponding to the second storage directory.
In the storage directories in the temporary storage server, the second storage directories correspond to the operation servers, so that the storage directories and the corresponding data in the directories can be completely transferred to the corresponding operation servers directly according to the names of the second storage directories.
For example, under the 2019\10\28\14\02 directory, a plurality of pieces of data to be processed may be stored, and the second storage directory of the pieces of data to be processed is 02, which corresponds to the No. 02 operation server, so that the storage directory of 2019\10\28\14\02 and the pieces of data to be processed stored in the directory can be all transferred to the No. 02 operation server.
The operation server can analyze and process the stored data and generate a component report; each operation server can upload the generated sub report tables to the summarizing server, the summarizing server uploads all the sub report tables to the output server after summarizing, and the output server displays summarized data according to the instructions.
In the embodiment, a large amount of data can be grouped according to time and then distributed to different operation servers in a balanced manner according to serial numbers, the number of the servers can be flexibly adjusted according to actual conditions, and resources are reasonably distributed; the plurality of operation servers respectively process the data and then perform summary processing, so that the data processing is more efficient.
Fig. 3 is a data processing method provided in a third embodiment of the present application, and as shown in fig. 3, the method includes:
s301, when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction;
s302, determining a first storage directory of the data to be processed according to the acquisition time of the data to be processed;
s303, determining a second storage directory of the data to be processed according to the serial number of the data to be processed;
s301 to S303 in this embodiment are similar to S201 to S203 described above, and reference may be made to each other, which is not described herein.
S304, determining a third storage directory of the data to be processed according to the data type and/or the data attribute of the data to be processed;
specifically, when the data is temporarily stored on the Kafka platform, the data can be mapped to a defined data type field, and the directory can be further divided for the data to be processed according to the data type. For example, the data type may include an access type or a resource type, the access type data may correspond to 01, the resource type data may correspond to 02, and if the data type of the to-be-processed data is the access type, the third storage directory of the to-be-processed data may be 01; if the data type of the data to be processed is resource type, the third storage directory of the data to be processed may be 02.
The data to be processed may also correspond to the attribute of the current state of the data, for example, in a short message sending service, if the data to be processed belongs to a short message to be sent, the third storage directory corresponding to the data to be processed may be corresponding to the data to be sent; if the data to be processed belongs to the short message successfully received by the user, the third storage directory corresponding to the data to be processed can be correspondingly successfully sent. If the data to be processed belongs to the short message to be sent, the third storage directory corresponding to the data to be processed can be corresponding to 01; if the data to be processed belongs to the short message successfully received by the user, the third storage directory corresponding to the data to be processed may be set as 02.
The third storage directory of the data to be processed may be individually determined by one of a data type or a data attribute of the data to be processed; or may be determined by both the data type and the data attributes of the data to be processed. For example, if the data type of the to-be-processed data is resource (02) and the data attribute is to be sent (01), the third storage directory of the to-be-processed data may be 02/01.
Of course, when determining the third storage directory of the data to be processed, other information of the data to be processed may also be considered, for example, a destination address of the data to be processed, customer information, and the like may be considered.
S305, determining a storage directory of the data to be processed according to the first storage directory, the second storage directory and the third storage directory;
specifically, the first storage directory, the second storage directory, and the third storage directory of the data to be processed may be sequentially combined as the storage directory of the data to be processed, and then the data to be processed is stored in the corresponding storage directory.
Illustratively, if the first storage directory of the data to be processed is 2019\10\28\14\02, the second storage directory is 02, and the third storage directory is 02, the storage directory of the data to be processed is 2019\10\28\14\02\ 02.
S306, determining an operation server corresponding to the second storage directory;
the operation servers can be multiple, the number of the operation servers can be determined according to actual conditions, and then the corresponding operation servers can be determined according to the second storage catalog of the data to be processed.
And S307, distributing the data to be processed to an operation server corresponding to the second storage directory.
Specifically, each data to be processed is transferred to the operation server corresponding to the second storage directory. For example, the storage directories with the second storage directory of 02, such as 2019\10\28\14\02\02, 2019\10\28\15\02\02, 2019\10\28\14\02\01, and the stored data to be processed, can be all transferred to the server number 02; and the storage directories with the second storage directory 03, such as 2019\10\28\14\03\02, 2019\10\28\15\03\02, 2019\10\28\14\03\01 and the like, and the stored to-be-processed data are all transferred to the server 03.
The data to be processed stored in each operation server is stored in each directory by taking time as a unit, and when the operation servers perform data processing, if the data needs to be summarized by taking time as a unit, the data can be divided according to the stored directories and then summarized without reclassification according to the acquisition time of the data.
In the storage directory of the operation server, the third storage directory may reflect the data type or the data attribute of the data, so that the data does not need to be classified according to the data type and the data attribute when the data is processed.
In addition, the data are orderly stored in each operation server according to the acquisition time, the serial number, the data type and/or the data attribute, so that the servers can search the data more conveniently in the data summarizing process.
Each operation server collects the data, generates a sub report and uploads the generated sub report to a collection server; and the summarizing server summarizes all the sub report tables and uploads the sub report tables to the output server, and the output server displays the summarized data according to the instruction.
In the embodiment of the application, when the storage directory is determined according to the data type and the data attribute of the data, the data is classified once according to the data type and the data attribute, so that the workload of the operation server in data processing is reduced, and the data processing efficiency is improved.
In order to better explain the data processing method in the present application, an example is described below.
With the development of mobile communication, short messages play an increasingly important role in daily life, and the server can send the short messages to a target terminal according to data provided by a client company. During the sending of the short message, the server generates a large amount of data, for example, a company may generate more than 10 hundred million pieces of data a day. When the data are counted and generated, the data can be uploaded through one or more gateways and are uniformly temporarily stored in the Kafka platform.
The temporary storage server in the data processing system can acquire various data generated in the short message sending process from the Kafka platform at regular time or after receiving the statistical instruction. Then sequentially establishing storage catalogues on a temporary storage server by taking hours as a minimum unit; if there are 20 calculation servers in the data processing system, then 20 storage directories with names 01, 02, … and 20 can be created under the directory corresponding to the time unit; establishing two storage directories with the names 01 and 02 under the 20 storage directories, wherein the directory No. 01 can store data with the data type of access, and the directory No. 02 can store data with the data type of resource; and then, under the directory established according to the data type, establishing a plurality of directories according to the data attribute, such as sent data, successfully sent data and the like.
After the temporary storage server obtains the data from the Kafka platform, the temporary storage server can determine a time catalog of the data according to the obtaining time of the data, determine a corresponding server catalog according to the serial number of the data, determine a type catalog of the data according to the data type of the data, determine an attribute catalog of the data according to the data attribute of the data, and combine all catalogs into a final storage catalog of the data. The data is then stored in the corresponding storage directory. Because serial numbers are consecutive, when data is distributed according to remainders obtained by serial number remainder, the amount of data in each server can be made to be approximately consistent.
According to the corresponding server directory in the storage directory, each directory and the stored data thereof can be transferred to the corresponding operation server. Each operation server processes the data and generates a sub report, then each operation server uploads the generated sub report to a summary server, and the summary server summarizes the sub report and outputs the summary result through an output server. For example, the summary result may include the reason summary of data transmission failure, the number of data transmissions, the success rate of data transmissions, and the like. And when the data output instruction is received, the output server displays corresponding data according to the data output instruction.
Fig. 4 is a schematic diagram of a data processing system according to a fourth embodiment of the present application, where as shown in fig. 4, the system includes:
the temporary storage server 41 is configured to, when receiving a data summarization instruction, obtain data to be processed according to the data summarization instruction; determining a storage directory of the data to be processed; distributing the data to be processed to each operation server according to the storage catalog of the data to be processed; indicating each operation server to generate a component report according to the data to be processed and sending the component report to a summary server for summary;
the operation server 42 is used for receiving the data to be processed distributed by the temporary storage server, generating a report according to the data to be processed and sending the report to the summary server;
a summarizing server 43, configured to summarize the sub-report tables reported by the operation servers, and upload a summarizing result to an output server;
and the output server 44 is used for outputting the summary result.
The temporary storage server 41 may specifically include:
the acquisition module is used for acquiring data to be processed according to the data summarization instruction when the data summarization instruction is received;
the determining module is used for determining a storage directory of the data to be processed;
and the distribution module is used for distributing the data to be processed to each operation server according to the storage directory of the data to be processed so as to instruct each operation server to generate a component report according to the data to be processed and send the component report to a summary server for summary.
In the above temporary storage server, the data to be processed has corresponding acquisition time and serial number, and the determining module may include:
the first storage directory determining submodule is used for determining a first storage directory of the data to be processed according to the acquisition time of the data to be processed;
the second storage directory determining submodule is used for determining a second storage directory of the data to be processed according to the serial number of the data to be processed;
and the storage directory determining submodule is used for determining the storage directory of the data to be processed according to the first storage directory and the second storage directory.
The first storage directory determining sub-module may include:
an identifying unit configured to identify a plurality of time units in the acquisition time, the plurality of time units including a year unit, a month unit, a day unit, and a time unit;
and the first storage catalog creating unit is used for creating a year storage catalog corresponding to the year unit, sequentially creating storage catalogs corresponding to the month unit, the day unit and the time unit under the year storage catalog, and taking the storage catalog corresponding to the time unit as the first storage catalog.
The second storage directory determining sub-module may include:
the residue taking unit is used for taking the number of the operation servers as a modulus and taking the residue of the serial number of the data to be processed;
and the remainder processing unit is used for determining a second storage directory of the data to be processed according to a numerical value obtained by adding 1 to the absolute value of the remainder, and the name of the second storage directory is the same as the numerical value.
The storage directory determining sub-module may specifically include:
and the second storage directory creating unit is used for creating a target directory with the same name as the second storage directory under the first storage directory as the storage directory of the to-be-processed data.
The determining module may further include:
a third storage directory determining submodule, configured to determine a third storage directory of the to-be-processed data according to the data type and/or the data attribute of the to-be-processed data;
and determining the storage directory of the data to be processed according to the first storage directory, the second storage directory and the third storage directory.
The distribution module may include:
the operation server determining submodule is used for determining an operation server corresponding to the second storage directory;
and the to-be-processed data distribution submodule is used for distributing the to-be-processed data to the operation server corresponding to the second storage directory.
Fig. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present application. As shown in fig. 5, the server 5 of this embodiment includes: at least one processor 50 (only one shown in fig. 5), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, the processor 50 implementing the steps in any of the various method embodiments described above when executing the computer program 52.
The server 5 may be a computing device such as a cloud server. The server may include, but is not limited to, a processor 50, a memory 51. Those skilled in the art will appreciate that fig. 5 is merely an example of the server 5, and does not constitute a limitation on the server 5, and may include more or less components than those shown, or combine certain components, or different components, such as input output devices, network access devices, etc.
The processor 50 may be a Central Processing Unit (CPU), and the processor 50 may be other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may in some embodiments be an internal storage unit of the server 5, such as a hard disk or a memory of the server 5, the memory 51 may in other embodiments also be an external storage device of the server 5, such as a plug-in hard disk provided on the server 5, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash memory card (FlashCard), etc. further, the memory 51 may also include both an internal storage unit of the server 5 and an external storage device, the memory 51 is used for storing an operating system, applications, a Boot loader (Boot L loader), data and other programs, such as program code of the computer program, etc. the memory 51 may also be used for temporarily storing data that has been or will be output.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a server, enables the server to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/server, a recording medium, computer memory, Read-only memory (ROM), random-access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A data processing method, comprising:
when a data summarizing instruction is received, acquiring data to be processed according to the data summarizing instruction;
determining a storage directory of the data to be processed;
and distributing the data to be processed to each operation server according to the storage directory of the data to be processed so as to instruct each operation server to generate a component report according to the data to be processed and send the component report to a summary server for summary.
2. The method of claim 1, wherein the data to be processed has a corresponding acquisition time and serial number, and wherein determining the storage directory for the data to be processed comprises:
determining a first storage directory of the data to be processed according to the acquisition time of the data to be processed;
determining a second storage directory of the data to be processed according to the serial number of the data to be processed;
and determining the storage directory of the data to be processed according to the first storage directory and the second storage directory.
3. The method of claim 2, wherein the determining the first storage directory of the to-be-processed data according to the acquisition time of the to-be-processed data comprises:
identifying a plurality of time units in the acquisition time, the plurality of time units including year units, month units, day units, and time units;
establishing a year storage directory corresponding to the year unit, and establishing storage directories corresponding to the month unit, the day unit and the time unit in sequence under the year storage directory, wherein the storage directory corresponding to the time unit is used as a first storage directory.
4. The method of claim 2, wherein determining the second storage directory of the data to be processed according to the serial number of the data to be processed comprises:
taking the number of the operation servers as a modulus, and taking a remainder for the serial number of the data to be processed;
and determining a second storage directory of the data to be processed according to a numerical value obtained by adding 1 to the absolute value of the remainder, wherein the name of the second storage directory is the same as the numerical value.
5. The method of claim 2, wherein determining the storage directory of the to-be-processed data from the first storage directory and the second storage directory comprises:
and under the first storage directory, creating a target directory with the same name as the second storage directory as the storage directory of the data to be processed.
6. The method of any of claims 2-5, wherein the data to be processed includes a corresponding data type and data attributes, and wherein determining the storage directory for the data to be processed further comprises:
determining a third storage directory of the data to be processed according to the data type and/or the data attribute of the data to be processed;
and determining the storage directory of the data to be processed according to the first storage directory, the second storage directory and the third storage directory.
7. The method of claim 6, wherein the distributing the to-be-processed data to each calculation server according to the storage directory of the to-be-processed data comprises:
determining an operation server corresponding to the second storage directory;
and distributing the data to be processed to an operation server corresponding to the second storage directory.
8. A data processing system, comprising:
the temporary storage server is used for acquiring data to be processed according to the data summarizing instruction when the data summarizing instruction is received; determining a storage directory of the data to be processed; distributing the data to be processed to each operation server according to the storage catalog of the data to be processed; indicating each operation server to generate a component report according to the data to be processed and sending the component report to a summary server for summary;
the operation server is used for receiving the data to be processed distributed by the temporary storage server, generating a sub report according to the data to be processed and sending the sub report to the summary server;
the summarizing server is used for summarizing the sub-report tables reported by the operation servers and uploading a summarizing result to the output server;
and the output server is used for outputting the summary result.
9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010229525.8A CN111475291A (en) | 2020-03-27 | 2020-03-27 | Data processing method, system, server and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010229525.8A CN111475291A (en) | 2020-03-27 | 2020-03-27 | Data processing method, system, server and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111475291A true CN111475291A (en) | 2020-07-31 |
Family
ID=71749156
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010229525.8A Pending CN111475291A (en) | 2020-03-27 | 2020-03-27 | Data processing method, system, server and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111475291A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115420A (en) * | 2020-09-21 | 2020-12-22 | 携程计算机技术（上海）有限公司 | Data statistical method, system, equipment and storage medium based on discrete grouping |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102693297A (en) * | 2012-05-16 | 2012-09-26 | 华为技术有限公司 | Data processing method, node and ETL (extract transform and load) system |
| CN106326456A (en) * | 2016-08-29 | 2017-01-11 | 广州御银自动柜员机技术有限公司 | Processing system and method of bill serial number picture file |
| CN108063808A (en) * | 2017-12-11 | 2018-05-22 | 海尔优家智能科技（北京）有限公司 | A kind of acquisition methods of business, device, storage medium and computer equipment |
| CN108256076A (en) * | 2018-01-18 | 2018-07-06 | 广州大学 | Distributed mass data processing method and processing device |
| CN109783449A (en) * | 2018-12-13 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Data query processing method, platform, system and readable storage medium storing program for executing |
-
2020
- 2020-03-27 CN CN202010229525.8A patent/CN111475291A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102693297A (en) * | 2012-05-16 | 2012-09-26 | 华为技术有限公司 | Data processing method, node and ETL (extract transform and load) system |
| CN106326456A (en) * | 2016-08-29 | 2017-01-11 | 广州御银自动柜员机技术有限公司 | Processing system and method of bill serial number picture file |
| CN108063808A (en) * | 2017-12-11 | 2018-05-22 | 海尔优家智能科技（北京）有限公司 | A kind of acquisition methods of business, device, storage medium and computer equipment |
| CN108256076A (en) * | 2018-01-18 | 2018-07-06 | 广州大学 | Distributed mass data processing method and processing device |
| CN109783449A (en) * | 2018-12-13 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Data query processing method, platform, system and readable storage medium storing program for executing |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115420A (en) * | 2020-09-21 | 2020-12-22 | 携程计算机技术（上海）有限公司 | Data statistical method, system, equipment and storage medium based on discrete grouping |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109299164B (en) | Data query method, computer readable storage medium and terminal equipment | |
| CN111078140B (en) | Nuclear power station file uploading management method and device, terminal equipment and medium | |
| CN111913738B (en) | Access request processing method, device, computing equipment and medium | |
| WO2022095699A1 (en) | Underlying data management method and system, and computer-readable storage medium | |
| CN110019367B (en) | Method and device for counting data characteristics | |
| CN112073395B (en) | File distribution method and device | |
| CN115168400A (en) | External data management system and method | |
| CN111258819A (en) | Data acquisition method, device and system for MySQL database backup file | |
| CN112818026A (en) | Data integration method and device | |
| CN113934733A (en) | Problem positioning method, device, system, storage medium and electronic equipment | |
| CN114125015A (en) | Data acquisition method and system | |
| CN113778947A (en) | Data import method, device and equipment of kafka stream processing platform | |
| CN111475291A (en) | Data processing method, system, server and medium | |
| CN111045983A (en) | Nuclear power station electronic file management method and device, terminal equipment and medium | |
| CN114461305B (en) | Data source determination method and device | |
| CN111723063A (en) | Method and device for processing offline log data | |
| CN111401819B (en) | Intersystem data pushing method and system | |
| CN114401239A (en) | Metadata transmission method and device, computer equipment and storage medium | |
| CN113407339A (en) | Resource request feedback method and device, readable storage medium and electronic equipment | |
| CN112597119A (en) | Method and device for generating processing log and storage medium | |
| CN105335470A (en) | Method and device for showing user login information | |
| CN113568803A (en) | Method, electronic device and computer program product for monitoring a storage system | |
| CN111291127A (en) | Data synchronization method, device, server and storage medium | |
| CN112988806A (en) | Data processing method and device | |
| CN109447386B (en) | Work assessment method and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |