CN112416910A - Data processing method, device, server and computer readable storage medium - Google Patents

Data processing method, device, server and computer readable storage medium Download PDF

Info

Publication number
CN112416910A
CN112416910A CN201910779219.9A CN201910779219A CN112416910A CN 112416910 A CN112416910 A CN 112416910A CN 201910779219 A CN201910779219 A CN 201910779219A CN 112416910 A CN112416910 A CN 112416910A
Authority
CN
China
Prior art keywords
modified
data processing
calculated
information
calculation result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910779219.9A
Other languages
Chinese (zh)
Inventor
严敏
刘玉霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SF Technology Co Ltd
Shenzhen SF Taisen Holding Group Co Ltd
SF Tech Co Ltd
Original Assignee
SF Technology Co Ltd
Shenzhen SF Taisen Holding Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SF Technology Co Ltd, Shenzhen SF Taisen Holding Group Co Ltd filed Critical SF Technology Co Ltd
Priority to CN201910779219.9A priority Critical patent/CN112416910A/en
Publication of CN112416910A publication Critical patent/CN112416910A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application discloses a data processing method, a data processing device, a server and a computer readable storage medium. The information to be modified and the information to be calculated are obtained through the relational database; then extracting the information to be modified and the information to be calculated to a big data processing system; determining a calculation result of the data to be calculated through the big data processing system, and modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result; and finally, pushing the modified calculation result to the relational database. According to the scheme, the calculation result can be modified according to the data to be modified through a big data processing system in the data processing device, and then the modified calculation result is pushed to the relational database, so that a user can obtain the modified calculation result through the device, and the performance of the device can be improved.

Description

Data processing method, device, server and computer readable storage medium
Technical Field
The present application relates to the field of big data technologies, and in particular, to a data processing method, an apparatus, a server, and a computer-readable storage medium.
Background
With the continuous progress and development of society, the application of computers is more and more extensive, the amount of processed data is larger and larger, the number of data in PB (byte) level per day is increased, but the traditional relational database has poor calculation performance on big data and low processing efficiency.
With the continuous development and maturity of big data technology, the big data processing system has high processing efficiency for big data volume, so that the high-efficiency computing performance of the big data processing system can be combined with the traditional relational database to obtain an efficient data processing device, the device can improve the processing efficiency of big data, but can not modify the data being processed according to the received modification information, a user can not obtain the data processed according to the modification information through the device, and the performance of the device is still to be improved.
Disclosure of Invention
Embodiments of the present application provide a data processing method, an apparatus, a server, and a computer-readable storage medium, which can improve performance of an apparatus.
In a first aspect, an embodiment of the present application provides a data processing method, including:
acquiring information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
extracting the information to be modified and the information to be calculated to a big data processing system;
determining a calculation result of the data to be calculated through the big data processing system;
modifying the calculation result through the big data processing system according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result;
and pushing the modified calculation result to the relational database.
In some embodiments, the modifying, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified to obtain a modified calculation result includes:
determining, by the big data processing system, an identifier corresponding to the identifier to be modified in the identifiers to be calculated as a target identifier;
determining, by the big data processing system, a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified;
and replacing the result to be modified with the data to be modified through the big data processing system to obtain the modified calculation result.
In some embodiments, the determining, by the big data processing system, a calculation result of the data to be calculated includes:
and determining the configuration information corresponding to the data to be calculated according to the preset configuration relationship in the big data processing system to obtain the calculation result.
In some embodiments, the extracting the information to be modified and the information to be calculated to a big data processing system includes:
detecting whether the system time of the big data processing system reaches a preset drawing time or not;
and if so, extracting the information to be modified and the information to be calculated in the relational database to a service library of the big data processing system through a drawing tool.
In some embodiments, after the modifying, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, and obtaining a modified calculation result, the method further includes:
and storing the modified calculation result into a project library of the big data processing system.
In some embodiments, said pushing said modified computation result to said relational database comprises:
detecting whether the system time reaches preset pushing time or not;
and if so, pushing the modified calculation result in the project library to the relational database.
In some embodiments, the method further comprises:
monitoring the big data processing system in real time;
and if the big data processing system is monitored to have a fault, sending fault alarm information to a monitoring terminal.
In a second aspect, an embodiment of the present application further provides a data processing apparatus, including:
the device comprises an acquisition unit, a calculation unit and a calculation unit, wherein the acquisition unit is used for acquiring information to be modified and information to be calculated through a relational database, the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
the extraction unit is used for extracting the information to be modified and the information to be calculated to a big data processing system;
the determining unit is used for determining a calculation result of the data to be calculated through the big data processing system;
the processing unit is used for modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified through the big data processing system to obtain a modified calculation result;
and the pushing unit is used for pushing the modified calculation result to the relational database.
In some embodiments, the processing unit is specifically configured to:
determining, by the big data processing system, an identifier corresponding to the identifier to be modified in the identifiers to be calculated as a target identifier;
determining, by the big data processing system, a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified;
and replacing the result to be modified with the data to be modified through the big data processing system to obtain the modified calculation result.
In some embodiments, the determining unit is specifically configured to:
and determining the configuration information corresponding to the data to be calculated according to the preset configuration relationship in the big data processing system to obtain the calculation result.
In some embodiments, the extraction unit is specifically configured to:
detecting whether the system time of the big data processing system reaches a preset drawing time or not;
and if so, extracting the information to be modified and the information to be calculated in the relational database to a service library of the big data processing system through a drawing tool.
In some embodiments, the apparatus further comprises:
and the storage unit is used for storing the modified calculation result into a project library of the big data processing system.
In some embodiments, the pushing unit is specifically configured to:
detecting whether the system time reaches preset pushing time or not;
and if so, pushing the modified calculation result in the project library to the relational database.
In some embodiments, the apparatus further comprises:
the monitoring unit is used for monitoring the big data processing system in real time;
and the sending unit is used for sending fault alarm information to the monitoring terminal if the big data processing system is monitored to have a fault.
In a third aspect, an embodiment of the present application further provides a server, including a memory and a processor, where the memory stores a computer program, and the processor executes, when calling the computer program in the memory, any step in any data processing method provided in the embodiment of the present application.
In a fourth aspect, the present application further provides a computer-readable storage medium, where a plurality of instructions are stored, and the instructions are adapted to be loaded by a processor to perform the steps in any one of the data processing methods provided in the present application.
In the embodiment of the application, a data processing device acquires information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises the data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated; then extracting the information to be modified and the information to be calculated to a big data processing system; determining a calculation result of the data to be calculated through the big data processing system, and modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result; and finally, pushing the modified calculation result to the relational database. According to the scheme, the calculation result can be modified according to the information to be modified through the big data processing system in the data processing device, and the modified calculation result is pushed to the relational database.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a data processing method provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram of a data processing method according to an embodiment of the present application;
FIG. 3 is another schematic flow chart diagram of a data processing method provided in an embodiment of the present application;
FIG. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 5 is a schematic diagram of another structure of a data processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description that follows, specific embodiments of the present invention are described with reference to steps and symbols executed by one or more computers, unless otherwise indicated. Accordingly, these steps and operations will be referred to, several times, as being performed by a computer, the computer performing operations involving a processing unit of the computer in electronic signals representing data in a structured form. This operation transforms the data or maintains it at locations in the computer's memory system, which may be reconfigured or otherwise altered in a manner well known to those skilled in the art. The data maintains a data structure that is a physical location of the memory that has particular characteristics defined by the data format. However, while the principles of the invention have been described in language specific to above, it is not intended to be limited to the specific form set forth herein, but on the contrary, it is to be understood that various steps and operations described hereinafter may be implemented in hardware.
The principles of the present invention are operational with numerous other general purpose or special purpose computing, communication environments or configurations. Examples of well known computing systems, environments, and configurations that may be suitable for use with the invention include, but are not limited to, hand-held telephones, personal computers, servers, multiprocessor systems, microcomputer-based systems, mainframe-based computers, and distributed computing environments that include any of the above systems or devices.
The terms "first", "second", and "third", etc. in the present invention are used for distinguishing different objects, not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions.
Referring to fig. 1, fig. 1 is a schematic view of an application scenario of a data processing method in an embodiment of the present application, in some embodiments, a user may modify queried data through a front-end service system, that is, the service system may receive information to be modified sent by the user, and then the service system may send the information to be modified to a back-end data processing device, and the data processing device receives and stores the data to be modified through a relational database.
Then the data processing device extracts the information to be modified and the information to be calculated to a service library in the big data processing system at regular time, a calculation module in the big data processing system extracts the information to be modified and the information to be calculated from the service library at regular time, determines configuration information corresponding to the data to be calculated according to a preset configuration relation, obtains a calculation result, and then modifies the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified, and obtains the modified calculation result.
After obtaining the modified calculation result, the calculation module stores the modified calculation result in a project library in the big data processing system, then pushes the modified calculation result in the project library to a relational database at regular time, and the business system obtains the modified calculation result through the relational database, so that a user can inquire the modified calculation result through the business system.
The data processing device in this embodiment can modify the calculation result according to the acquired data to be modified, and can improve the performance of the device and improve the interactivity between the user and the data processing device.
It should be noted that the big data processing system in the embodiment of the present application has efficient computing performance, and in combination with the conventional relational database, the processing efficiency of big data can be improved.
Referring to fig. 2, fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application. The execution main body of the data processing method may be the data processing apparatus provided in the embodiment of the present application, or a server integrated with the data processing apparatus, where the data processing apparatus may be implemented in a hardware or software manner. The data processing method may include:
201. and acquiring the information to be modified and the information to be calculated through the relational database.
In some embodiments, the to-be-modified information includes to-be-modified data and a to-be-modified identifier corresponding to the to-be-modified data, and the to-be-computed information includes to-be-computed data and a to-be-computed identifier corresponding to the to-be-modified data.
When the calculation result is modified, the to-be-modified identifier can find out the data to be modified in the calculation result through the association with the primary key of the to-be-modified identifier.
In some embodiments, a user may modify information of a previous calculation period through a front-end service system to generate information to be modified, and then send the information to be modified to a relational database in a data processing device through the service system, for example, a service person (user) finds that a code number of equipment (such as a frock) identified by an employee a in a previous calculation period is an L code, but the code number is actually larger for the employee a, so that the service person may modify the code number through the service system, for example, modify the code number into an M code, and then generate data to be modified corresponding to the employee a.
In some embodiments, the information to be modified further includes a modification time.
In some embodiments, the data to be calculated specifically includes the current position information of the employee, and since the position of the old employee may change, and there are situations that a new employee enters the position and the old employee leaves the position, the application needs to calculate the latest data to be calculated regularly.
In some embodiments, the relational database may be a relational database management system (mySQL) that may be used to store data, which may be used for presentation in the front end.
202. And extracting the information to be modified and the information to be calculated to a big data processing system.
In some embodiments, data in the relational database of the previous day may be extracted into the big data processing system in the early morning, and specifically, whether the system time of the big data processing system reaches a preset extraction time or not may be detected; and if so, extracting the information to be modified and the information to be calculated in the relational database into a service library of the big data processing system through a drawing tool, wherein the data stored in the service library is unprocessed data acquired from the service system, and the data is metadata to be processed by the big data processing system.
The system time of the big data processing system can be natural time, the preset drawing time can be 12:00 in the morning of each day, the information to be modified and the information to be calculated which need to be extracted are information stored in a relational database of the previous day, and the specific drawing time is not limited here. The drawing tool may be an ETL (extract Transform load) tool, which may be implemented in a big data processing system.
In some embodiments, the representation form of the service library may be a partition table, and the user may customize a partition manner, for example, the partition manner may be performed in an hour, day, or month manner.
In some embodiments, the big data processing System may be a Hive System (a data warehouse infrastructure over Hadoop, an off-line computing System), and data of the big data processing System is stored in a Distributed File System (Hdfs) in the Hive System.
203. And determining a calculation result of the data to be calculated through the big data processing system.
In some embodiments, specifically, the configuration information corresponding to the data to be calculated may be determined through a configuration relationship preset in the big data processing system, so as to obtain the calculation result.
In some embodiments, the preset configuration relationship is a corresponding relationship (configuration standard) between the post information of the employee and the equipment type, the data to be calculated includes the post information of the employee to be calculated, and the like, and in this embodiment, the equipment type of each employee to be calculated may be determined by a calculation module in the big data processing system according to the post information of each employee to be calculated and the corresponding relationship between the post information and the equipment type.
In some embodiments, the configuration relationship further includes a correspondence between the identifier of the employee and the historical equipment size of the employee (i.e., a historical size table), and the method further includes determining the equipment size corresponding to each identifier to be calculated according to a matching relationship between the identifier to be calculated and the historical size table.
At this time, the calculation result may include the equipment type and the equipment size corresponding to each identifier to be calculated (i.e., each employee to be calculated).
In some real-time examples, the calculation module may schedule the ETL tool to extract the information to be modified and the information to be calculated from the business repository at regular time (e.g., at 1 am of each day, where the specific time is not limited herein).
204. And modifying the calculation result through the big data processing system according to the identification to be modified, the identification to be calculated and the data to be modified to obtain the modified calculation result.
In some embodiments, modifying, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, and obtaining the modified calculation result specifically includes:
determining an identifier corresponding to the identifier to be modified in the identifier to be calculated as a target identifier through the big data processing system; determining a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified through the big data processing system; and replacing the result to be modified with the data to be modified through the big data processing system to obtain the modified calculation result.
After the calculation result of the data to be settled is determined, whether the identification to be calculated contains the identification to be modified is determined through the calculation module through the primary key association (that is, whether the calculation result contains the calculation sub-result to be modified, wherein one identification can correspond to one calculation sub-result), when the identification to be calculated exists, the identification corresponding to the identification to be modified in the identification to be calculated is determined as the target identification, then the calculation sub-result corresponding to the target identification is determined as the data to be modified, and finally the data to be modified is modified into the corresponding data to be modified, so that the modified calculation result is obtained.
For example, the identifier of employee a is 1234, in this application, the business personnel modifies the data of employee a in the previous day, for example, modifies the equipment size of employee a (original size is L, and modification is now made to M), at this time, the equipment size in the calculation sub-result identified as 1234 needs to be modified to M (no matter what the latest calculation result of employee a is, the equipment size is directly modified), and the other calculation sub-results that do not need to be modified in the calculation result do not need to be modified, and the calculation result is directly output.
In some embodiments, the data to be modified further includes modification time (time for a service person to modify the data through a service system), and at this time, the calculation result corresponding to the target identifier in the modified calculation result includes, in addition to the data to be modified, modification time corresponding to the data to be modified.
In some embodiments, when the calculation result is modified, in addition to directly replacing the to-be-modified result corresponding to the identifier in the calculation result with the to-be-modified data, another implementation manner may also be adopted, where the to-be-modified data is added to a position in the calculation result corresponding to the to-be-modified identifier, and the calculation sub-result corresponding to the to-be-modified identifier (i.e., the target identifier) includes the latest calculation result of the identifier and the modified result (i.e., the to-be-modified data).
In some embodiments, the big data processing system modifies the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, and after obtaining the modified calculation result, the method further includes: and saving the modified calculation result to a project library of the big data processing system.
In some embodiments, the representation form of the project library may be a partition table, and a user may customize a partition mode, for example, the partition mode may be performed in an hour, day, or month mode.
105. And pushing the modified calculation result to the relational database.
Specifically, pushing the modified calculation result to the relational database includes: detecting whether the system time reaches preset pushing time or not; and if so, pushing the modified calculation result in the project library to the relational database.
The push time may be seven points earlier in the day, or may be other push times, which is not limited specifically.
In some embodiments, a plurality of library tables exist in the relational database, the library tables can be divided according to time (for example, hours, days, months, and the like), and the modified calculation result can be pushed to the library table corresponding to the time of the day in the relational database.
In some embodiments, the present application further comprises: monitoring the big data processing system in real time; and if the big data processing system is monitored to have a fault, sending fault alarm information to the monitoring terminal.
For example, when it is monitored that the big data processing system does not calculate the acquired data, it may be determined that the big data processing system has a fault, and at this time, fault alarm information may be sent to the monitoring terminal, for example, the fault information may be sent to the monitoring terminal in a manner of mail or short message, where the monitoring terminal may be a mobile phone or a computer.
In addition, the embodiment may also monitor the relational database in the database processing apparatus, and if a fault is monitored, send fault information to the monitoring terminal.
After the modified calculation result is pushed to the relational database, the business system can acquire the modified calculation result from the relational database, and after the business system receives a query instruction of business personnel, the business system can present the modified calculation result to the business personnel.
In the embodiment of the application, a data processing device acquires information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises the data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated; then extracting the information to be modified and the information to be calculated to a big data processing system; determining a calculation result of the data to be calculated through the big data processing system, and modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result; and finally, pushing the modified calculation result to the relational database. According to the scheme, the calculation result can be modified according to the data to be modified through a big data processing system in the data processing device, and then the modified calculation result is pushed to the relational database, so that a user can obtain the modified calculation result through the device, and the performance of the device can be improved.
In addition, according to the scheme, the big data processing system is added in the traditional data processing device (the relational database is used for calculating the data), the big data processing system is used for calculating the big data, the calculation efficiency of the data can be improved, in addition, the whole calculation process is realized in an automatic mode, the scheduling task is monitored, the maintenance cost is low, and the system stability is high.
In addition, the embodiment can also be used for forecasting new staff receiving plans, adding and replacing forecasting of employees in work, forecasting of labor insurance material requirements of pre-recruited employees and the like.
The data processing method according to the above embodiment will be described in further detail below.
Referring to fig. 3, fig. 3 is another schematic flow chart of a data processing method according to an embodiment of the present disclosure. The data processing method can be applied to a server, wherein in this embodiment, the relational database is mySQL, and the big data processing system is Hive as an example, which is described in detail, as shown in fig. 3, the flow of the data processing method may be as follows:
301. and the server acquires the information to be modified and the information to be calculated from the service system through mySQL.
In this embodiment, the information to be modified includes data to be modified and a to-be-modified identifier corresponding to the data to be modified, where the information to be calculated includes data to be calculated and a to-be-calculated identifier corresponding to the data to be calculated, in some embodiments, the to-be-modified identifier may be an identifier that can prove the identity of an employee who needs to modify the information, such as a job number or an identity card of the employee who needs to calculate the information, such as an identifier that can prove the identity of the employee, the data to be modified may be data modified in the current calculation cycle, and the data to be calculated may be data that needs to be calculated in the current calculation cycle.
In some embodiments, the information to be modified further includes a modification time.
In some embodiments, the information to be modified is information obtained by modifying the calculation sub-result in the calculation result when the user queries the calculation result through the service system.
302. The server detects whether the system time of Hive reaches the preset drawing time.
In this embodiment, the system time of Hive may be a natural time, and the preset number of draws may be 12:00 in the morning of each day, where the specific number of draws is not limited herein.
303. If yes, the server extracts the information to be modified and the information to be calculated in the mySQL to the Hive service library through the ETL.
In this embodiment, if a preset number of times of drawing is reached, for example, 12:00 am, the information to be modified and the information to be calculated in mySQL are extracted into the Hive service library by ETL, where the extracted information to be modified and the information to be calculated are information stored in mySQL one day before, and ETL may be set in Hive.
The data stored in the service library is unprocessed data acquired from a service system, and the data is metadata to be processed by Hive.
In some embodiments, the representation form of the service library may be a partition table, and the user may customize a partition manner, for example, the partition manner may be performed in an hour, day, or month manner.
If the time of the drawing is not reached, the drawing operation is not required to be executed.
304. And the server determines the configuration information corresponding to the data to be calculated through the calculation module in the Hive and a preset configuration relation to obtain the calculation result.
In this implementation, first, a computation module in Hive pulls information to be computed and information to be modified from a service library, and then determines configuration information corresponding to each piece of data to be computed in the information to be computed according to a configuration relationship to obtain a computation result, where the computation result includes multiple computation sub-results, and each computation sub-result corresponds to one identifier to be computed.
In some embodiments, the preset configuration relationship is a corresponding relationship (configuration standard) between the position information of the employee and the equipment type, the data to be calculated includes the position information of the employee to be calculated, and the like, and in this embodiment, the equipment type of each employee to be calculated may be determined by the calculation module in the Hive according to the position information of each employee to be calculated and the corresponding relationship between the position information and the equipment type.
In some embodiments, the configuration relationship further includes a correspondence between the identifier of the employee and the historical equipment size of the employee (i.e., a historical size table), and the method further includes determining the equipment size corresponding to each identifier to be calculated according to a matching relationship between the identifier to be calculated and the historical size table.
At this time, the calculation result may include the equipment type and the equipment size corresponding to each identifier to be calculated (i.e., each employee to be calculated).
In some real-time examples, the calculation module may schedule the ETL tool to extract the information to be modified and the information to be calculated from the service library at regular time, for example, at 1 am every day, and the specific time is not limited herein.
The computing module in the embodiment of the application can realize computing logic through a Hive sql script.
305. And the server determines the identifier corresponding to the identifier to be modified in the identifier to be calculated as the target identifier through the calculation module.
In this embodiment, after the calculation result of the data to be calculated is determined, it is further determined whether an identifier that is the same as the identifier to be modified exists in the identifiers to be calculated respectively corresponding to each calculation sub-result in the calculation result through the primary key association, that is, it is determined whether a calculation sub-result that needs to be modified exists in the calculation sub-results, if so, the identifier that is the same as the identifier to be modified in the identifiers to be calculated is determined as the target identifier, and the calculation sub-result that corresponds to the target identifier is the result that needs to be modified.
And the calculation sub-result corresponding to the identifier which is different from the identifier to be modified in the identifier to be calculated is a result which does not need to be modified, and the result is directly output to the mySQL.
306. And the server determines a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified through the calculation module.
In this embodiment, after the target identifier is determined, the sub-calculation result corresponding to the target identifier in the calculation result is determined as a result to be modified, where the result to be modified is a result that needs to be modified, and is not a final result.
307. And the server replaces the result to be modified with the data to be modified through the computing module to obtain the modified computing result.
In this embodiment, after determining a to-be-modified result in the calculation result, the to-be-modified result is replaced with the to-be-modified data through the calculation module, where in some embodiments, the to-be-modified data further includes modification time of a user.
For example, the identifier of employee a is 1234, in this application, the business personnel modifies the data of employee a in the previous day, for example, modifies the equipment size of employee a (original size is L, and modification is now made to M), at this time, the equipment size in the calculation sub-result identified as 1234 needs to be modified to M (no matter what the latest calculation result of employee a is, the equipment size is directly modified), and the other calculation sub-results that do not need to be modified in the calculation result do not need to be modified, and the calculation result is directly output.
In other embodiments, the data to be modified may be added to the position of the result corresponding to the target identifier, that is, to the result to be modified, where the target identifier may correspond to two results, one is a result calculated according to the obtained data to be calculated, and the other is the corresponding data to be modified.
308. And the server stores the modified calculation result into the project library of the Hive.
After the calculation result is modified through the primary key association, the modified calculation result is stored in a project library of Hive, wherein data calculated by the calculation module is stored in the project library.
309. The server detects whether the system time reaches preset push time.
In this embodiment, the server further needs to detect whether the Hive system time reaches a preset pushing time in real time, where the pushing time may be seven points earlier in the day, or may be other pushing times, and this is not limited specifically.
310. If yes, the server pushes the modified calculation result in the project library to the mySQL.
In this embodiment, if the preset pushing time is reached, pushing a corresponding modified computation result (for example, a computation result determined according to information of a day before mySQL) in the project library to the mySQL, where in some embodiments, the mySQL has a plurality of library tables, and the library tables may be divided according to time.
If the preset pushing time is not reached, the data pushing is not needed.
311. And the server receives the query instruction sent by the service system.
In this embodiment, mySQL in the server may receive a query instruction sent by a user through the service system, where the query instruction is used to query a latest calculation result, for example, a modified calculation result determined by the query server according to information obtained on a previous day, and if the calculation result on the previous day is not modified, the modified calculation result is directly a result corresponding to information obtained on the previous day.
In some embodiments, the query instruction further includes a query time (e.g., a query date), and the query instruction is configured to obtain a calculation result corresponding to the query time in the server, where the mySQL of the server stores calculation results (including modified calculation results) corresponding to each historical calculation date.
Wherein, the query instruction is triggered and sent by the user through the service system.
312. And the server sends the modified calculation result to the service system according to the query instruction.
In this embodiment, after receiving the query instruction, the server may push a calculation result corresponding to the query instruction to the service system, for example, send a modified calculation result determined on the current day to the service system, and at this time, the user may obtain the modified calculation result through the service system.
In some embodiments, the present application further comprises: monitoring the Hive in real time; and if the Hive is monitored to have a fault, sending fault alarm information to the monitoring terminal.
For example, when it is monitored that Hive does not calculate the acquired data, it may be determined that Hive has a fault, and at this time, fault alarm information may be sent to the monitoring terminal, for example, fault information may be sent to the monitoring terminal in a manner of an email or a short message, where the monitoring terminal may be a mobile phone or a computer.
In addition, the embodiment may also monitor mySQL in the database processing apparatus, and if a fault is monitored, send fault information to the monitoring terminal.
In this embodiment, a user may modify a historical calculation result (for example, a calculation result of a previous calculation cycle) through a service system, and send modification information to a server, so that the server performs joint modification on a latest calculation result according to the modification information of the user to obtain a modified calculation result, and when the user re-queries the calculation result in a current calculation cycle, the calculation result returned by the server combines with the modification information of the user in the previous calculation cycle, so that the server in the present application may increase interactivity with the user.
In the embodiment of the application, a server acquires information to be modified and information to be calculated through mySQL, wherein the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated; then extracting the information to be modified and the information to be calculated to Hive; determining a calculation result of the data to be calculated through the Hive, and modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result; and finally pushing the modified calculation result to the mySQL. In the scheme, the calculation result can be modified according to the data to be modified through the Hive in the server, and then the modified calculation result is pushed to the mySQL, so that the user can obtain the modified calculation result through the server, and the performance of the server can be improved.
In order to better implement the data processing method provided by the embodiment of the present application, an embodiment of the present application further provides a device based on the data processing method. The terms are the same as those in the data processing method, and details of implementation can be referred to the description in the method embodiment.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure, where the data processing apparatus may include an obtaining unit 401, an extracting unit 402, a determining unit 403, a processing unit 404, a pushing unit 405, and the like. Wherein:
an obtaining unit 401, configured to obtain information to be modified and information to be calculated through a relational database, where the information to be modified includes data to be modified and an identifier to be modified corresponding to the data to be modified, and the information to be calculated includes data to be calculated and an identifier to be calculated corresponding to the data to be calculated;
an extracting unit 402, configured to extract the information to be modified and the information to be calculated to a big data processing system;
a determining unit 403, configured to determine a calculation result of the data to be calculated by the big data processing system;
a processing unit 404, configured to modify, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, so as to obtain a modified calculation result;
a pushing unit 405, configured to push the modified calculation result to the relational database.
In some embodiments, the processing unit 404 is specifically configured to:
determining, by the big data processing system, an identifier corresponding to the identifier to be modified in the identifiers to be calculated as a target identifier;
determining, by the big data processing system, a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified;
and replacing the result to be modified with the data to be modified through the big data processing system to obtain the modified calculation result.
In some embodiments, the determining unit 403 is specifically configured to:
and determining the configuration information corresponding to the data to be calculated according to the preset configuration relationship in the big data processing system to obtain the calculation result.
In some embodiments, the extracting unit 402 is specifically configured to:
detecting whether the system time of the big data processing system reaches a preset drawing time or not;
and if so, extracting the information to be modified and the information to be calculated in the relational database to a service library of the big data processing system through a drawing tool.
Referring to fig. 5, in some embodiments, the apparatus further includes:
a saving unit 406, configured to save the modified calculation result to a project library of the big data processing system.
In some embodiments, the pushing unit 405 is specifically configured to:
detecting whether the system time reaches preset pushing time or not;
and if so, pushing the modified calculation result in the project library to the relational database.
In some embodiments, the apparatus further comprises:
a monitoring unit 407, configured to perform real-time monitoring on the big data processing system;
a sending unit 408, configured to send failure alarm information to the monitoring terminal if it is monitored that the big data processing system fails.
In this embodiment of the application, the obtaining unit 401 obtains information to be modified and information to be calculated through a relational database, where the information to be modified includes data to be modified and an identifier to be modified corresponding to the data to be modified, and the information to be calculated includes data to be calculated and an identifier to be calculated corresponding to the data to be calculated; then, the extracting unit 402 extracts the information to be modified and the information to be calculated to the big data processing system; the determining unit 403 determines a calculation result of the data to be calculated through the big data processing system, and the processing unit 404 modifies the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, so as to obtain a modified calculation result; finally, the pushing unit 405 pushes the modified calculation result to the relational database. According to the scheme, the calculation result can be modified according to the data to be modified through a big data processing system in the data processing device, and then the modified calculation result is pushed to the relational database, so that a user can obtain the modified calculation result through the device, and the performance of the device can be improved.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Referring to fig. 6, the present embodiment provides a server 600, which may include one or more processors 601 of a processing core, one or more memories 602 of a computer-readable storage medium, a Radio Frequency (RF) circuit 603, a power supply 604, an input unit 605, and a display unit 606. Those skilled in the art will appreciate that the server architecture shown in FIG. 6 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the processor 601 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 602 and calling data stored in the memory 602, thereby performing overall monitoring of the server. Optionally, processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 601.
The memory 602 may be used to store software programs and modules, and the processor 601 executes various functional applications and data processing by operating the software programs and modules stored in the memory 602.
The RF circuitry 603 may be used for receiving and transmitting signals during the process of transmitting and receiving information.
The server also includes a power supply 604 (e.g., a battery) for powering the various components, which may preferably be logically coupled to the processor 601 via a power management system to manage charging, discharging, and power consumption management functions via the power management system.
The server may also include an input unit 605, and the input unit 605 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
The server may also include a display unit 606, and the display unit 606 may be used to display information input by the user or provided to the user, as well as various graphical user interfaces of the server, which may be made up of graphics, text, icons, video, and any combination thereof. Specifically, in this embodiment, the processor 601 in the server loads the executable file corresponding to the process of one or more application programs into the memory 602 according to the following instructions, and the processor 601 runs the application programs stored in the memory 602, thereby implementing various functions as follows:
acquiring information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
extracting the information to be modified and the information to be calculated to a big data processing system;
determining a calculation result of the data to be calculated through the big data processing system;
modifying the calculation result through the big data processing system according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result;
and pushing the modified calculation result to the relational database.
In the above embodiments, the descriptions of the embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed description of the data processing method, and are not described herein again.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, embodiments of the present application provide a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any data processing method provided by the embodiments of the present application. For example, the instructions may perform the steps of:
acquiring information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
extracting the information to be modified and the information to be calculated to a big data processing system;
determining a calculation result of the data to be calculated through the big data processing system;
modifying the calculation result through the big data processing system according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result;
and pushing the modified calculation result to the relational database.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the storage medium can execute the steps in any data processing method provided in the embodiments of the present application, beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.
The data processing method, the data processing apparatus, the server, and the computer-readable storage medium provided in the embodiments of the present application are described in detail above, and a specific example is applied in the present application to explain the principles and implementations of the present application, and the description of the above embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (10)

1. A data processing method, comprising:
acquiring information to be modified and information to be calculated through a relational database, wherein the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
extracting the information to be modified and the information to be calculated to a big data processing system;
determining a calculation result of the data to be calculated through the big data processing system;
modifying the calculation result through the big data processing system according to the identification to be modified, the identification to be calculated and the data to be modified to obtain a modified calculation result;
and pushing the modified calculation result to the relational database.
2. The method according to claim 1, wherein the modifying, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified to obtain a modified calculation result includes:
determining, by the big data processing system, an identifier corresponding to the identifier to be modified in the identifiers to be calculated as a target identifier;
determining, by the big data processing system, a calculation sub-result corresponding to the target identifier in the calculation result as a result to be modified;
and replacing the result to be modified with the data to be modified through the big data processing system to obtain the modified calculation result.
3. The method of claim 1, wherein the determining, by the big data processing system, the calculation of the data to be calculated comprises:
and determining the configuration information corresponding to the data to be calculated according to the preset configuration relationship in the big data processing system to obtain the calculation result.
4. The method of claim 1, wherein the extracting the information to be modified and the information to be computed to a big data processing system comprises:
detecting whether the system time of the big data processing system reaches a preset drawing time or not;
and if so, extracting the information to be modified and the information to be calculated in the relational database to a service library of the big data processing system through a drawing tool.
5. The method according to claim 4, wherein the modifying, by the big data processing system, the calculation result according to the identifier to be modified, the identifier to be calculated, and the data to be modified, and after obtaining the modified calculation result, the method further comprises:
and storing the modified calculation result into a project library of the big data processing system.
6. The method of claim 5, wherein pushing the modified computation result to the relational database comprises:
detecting whether the system time reaches preset pushing time or not;
and if so, pushing the modified calculation result in the project library to the relational database.
7. The method according to any one of claims 1 to 6, further comprising:
monitoring the big data processing system in real time;
and if the big data processing system is monitored to have a fault, sending fault alarm information to a monitoring terminal.
8. A data processing apparatus, comprising:
the device comprises an acquisition unit, a calculation unit and a calculation unit, wherein the acquisition unit is used for acquiring information to be modified and information to be calculated through a relational database, the information to be modified comprises data to be modified and a corresponding identifier to be modified, and the information to be calculated comprises the data to be calculated and a corresponding identifier to be calculated;
the extraction unit is used for extracting the information to be modified and the information to be calculated to a big data processing system;
the determining unit is used for determining a calculation result of the data to be calculated through the big data processing system;
the processing unit is used for modifying the calculation result according to the identification to be modified, the identification to be calculated and the data to be modified through the big data processing system to obtain a modified calculation result;
and the pushing unit is used for pushing the modified calculation result to the relational database.
9. A server, characterized by comprising a processor and a memory, in which a computer program is stored, the processor executing the data processing method according to any one of claims 1 to 7 when calling the computer program in the memory.
10. A computer-readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the data processing method of any one of claims 1 to 7.
CN201910779219.9A 2019-08-22 2019-08-22 Data processing method, device, server and computer readable storage medium Pending CN112416910A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910779219.9A CN112416910A (en) 2019-08-22 2019-08-22 Data processing method, device, server and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910779219.9A CN112416910A (en) 2019-08-22 2019-08-22 Data processing method, device, server and computer readable storage medium

Publications (1)

Publication Number Publication Date
CN112416910A true CN112416910A (en) 2021-02-26

Family

ID=74779292

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910779219.9A Pending CN112416910A (en) 2019-08-22 2019-08-22 Data processing method, device, server and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN112416910A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160140149A1 (en) * 2014-11-19 2016-05-19 Unisys Corporation Dynamic modification of database schema
US20180217999A1 (en) * 2017-01-31 2018-08-02 Xactly Corporation System and method for prior period adjustment processing
CN108846753A (en) * 2018-06-06 2018-11-20 北京京东尚科信息技术有限公司 Method and apparatus for handling data
CN109284276A (en) * 2018-07-13 2019-01-29 西安图迹信息科技有限公司 A kind of database accelerated method based on big data framework
CN109684093A (en) * 2018-12-24 2019-04-26 成都四方伟业软件股份有限公司 Data processing method and system
CN109933606A (en) * 2019-03-19 2019-06-25 上海达梦数据库有限公司 A kind of database update method, apparatus, equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160140149A1 (en) * 2014-11-19 2016-05-19 Unisys Corporation Dynamic modification of database schema
US20180217999A1 (en) * 2017-01-31 2018-08-02 Xactly Corporation System and method for prior period adjustment processing
CN108846753A (en) * 2018-06-06 2018-11-20 北京京东尚科信息技术有限公司 Method and apparatus for handling data
CN109284276A (en) * 2018-07-13 2019-01-29 西安图迹信息科技有限公司 A kind of database accelerated method based on big data framework
CN109684093A (en) * 2018-12-24 2019-04-26 成都四方伟业软件股份有限公司 Data processing method and system
CN109933606A (en) * 2019-03-19 2019-06-25 上海达梦数据库有限公司 A kind of database update method, apparatus, equipment and storage medium

Similar Documents

Publication Publication Date Title
CN109902105B (en) Data query system, method, device and storage medium for micro-service architecture
CN107480123B (en) Garbage bullet screen identification method and device and computer equipment
CN108681590A (en) Incremental data processing method and processing device, computer equipment, computer storage media
CN110750650A (en) Construction method and device of enterprise knowledge graph
CN109241068B (en) Method and device for comparing foreground and background data and terminal equipment
CN108628607B (en) Method, system and storage medium for assisting software development based on artificial intelligence
CN103235811A (en) Data storage method and device
CN113961643A (en) Search engine updating method and device, equipment, medium and product thereof
CN112559717A (en) Search matching method and device, electronic equipment and storage medium
CN111722973B (en) Event timeout monitoring method, system and storage medium
CN114328981A (en) Knowledge graph establishing and data obtaining method and device based on mode mapping
CN111680478B (en) Report generation method, device, equipment and storage medium based on configuration software
CN112199443A (en) Data synchronization method and device, computer equipment and storage medium
CN116955856A (en) Information display method, device, electronic equipment and storage medium
CN116521664A (en) Data monitoring method and device for data warehouse, computing equipment and storage medium
CN112416910A (en) Data processing method, device, server and computer readable storage medium
CN116450723A (en) Data extraction method, device, computer equipment and storage medium
CN116089446A (en) Optimization control method and device for structured query statement
CN115544010A (en) Mapping relation determining method and device, electronic equipment and storage medium
CN115495463A (en) Data processing method and device, intelligent equipment and storage medium
CN112015623B (en) Report data processing method, device, equipment and readable storage medium
CN114881313A (en) Behavior prediction method and device based on artificial intelligence and related equipment
CN113420099A (en) Buried point data access control method and device, computer equipment and storage medium
CN113296913A (en) Data processing method, device and equipment based on single cluster and storage medium
CN105468752A (en) Data product construction system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination