CN107948227A - The performance optimization method and device of distributed system platform - Google Patents
The performance optimization method and device of distributed system platform Download PDFInfo
- Publication number
- CN107948227A CN107948227A CN201610894140.7A CN201610894140A CN107948227A CN 107948227 A CN107948227 A CN 107948227A CN 201610894140 A CN201610894140 A CN 201610894140A CN 107948227 A CN107948227 A CN 107948227A
- Authority
- CN
- China
- Prior art keywords
- partitions
- distributed system
- system platform
- configuration file
- configuration information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
- H04L67/303—Terminal profiles
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of performance optimization method and device of distributed system platform, it is related to information technology field, the regulated efficiency of number of partitions in distributed system platform can be improved, the performance effect of optimization of distributed system can be lifted, the described method includes：The configuration file for carrying number of partitions configuration information is obtained, the configuration file is passed to when distributed system platform receives task；The configuration file is parsed, obtains the number of partitions configuration information；According to the number of partitions configuration information, the number of partitions in the distributed system platform is updated.The performance that the present invention is suitable for distributed system platform optimizes.
Description
Technical field
The present invention relates to information technology field, more particularly to a kind of performance optimization method and dress of distributed system platform
Put.
Background technology
In recent years, with the rapid development of information technology, distributed system is used widely, distributed system is to build
The software systems on network are found, every assisting for task is handled, then integrates out result.It is and parallel in distributed system
Operational performance is related to number of partitions, for example, spark is a big data distributed system calculating platform, not only realizes
The operators m ap functions and reduce functions and computation model of MapReduce, also provides the operator of more horn of plenty, and Partition divides
The quantity in area determines the ability of spark concurrent operations, and Partition number of partitions is more, and spark concurrent capabilities are relatively more
It is good.
At present, number of partitions is by way of coding, and uses corresponding API (Application Programming
Interface, application programming interface) interface is configured, exists equivalent to by number of partitions and the fusion of specific code
Together.
However, in the case where carrying out performance optimization to distributed system, when needing constantly adjustment number of partitions, pass through
The mode of above-mentioned setting number of partitions, is required for modification code and repacks, can cause the regulated efficiency of number of partitions every time
It is relatively low, and then have impact on the performance optimization of distributed system.
The content of the invention
In view of the above problems, it is proposed that the present invention overcomes the above problem in order to provide one kind or solves at least in part
State the performance optimization method and device of the distributed system platform of problem.
In order to achieve the above object, present invention generally provides following technical solution：
On the one hand, the present invention provides a kind of performance optimization method of distributed system platform, this method to include：
The configuration file for carrying number of partitions configuration information is obtained, the configuration file is connect in distributed system platform
It is passed to during receipts task；
The configuration file is parsed, obtains the number of partitions configuration information；
According to the number of partitions configuration information, the number of partitions in the distributed system platform is updated.
Further, before the acquisition carries the configuration file of number of partitions configuration information, the method further includes：
The number of partitions for needing to set in the distributed system platform is substituted by default global identifier；
According to the configuration-direct of the configuration file received, in the number of partitions configuration information of the configuration file with institute
The corresponding number of partitions of default global identifier is stated to be configured；
It is described to obtain the configuration file for carrying number of partitions configuration information, specifically include：
Obtain with the configuration file postponed.
Specifically, it is described according to the number of partitions configuration information, to the number of partitions in the distributed system platform
It is updated, specifically includes：
By number of partitions corresponding with the default global identifier in the distributed system platform, described point is replaced with
Number of partitions corresponding with the default global identifier in area's quantity configuration information.
Further, the method further includes：Interval detects whether the distributed system platform connects to schedule
Receive new configuration file；It is described to obtain the configuration file for carrying number of partitions configuration information, specifically include：If so, then obtain
Take the recently received configuration file of the distributed system platform.
Further, the method further includes：The distributed system platform is triggered to be drawn according to the number of partitions after renewal
Divide subregion, so that the distributed system platform, which passes through the subregion after division, performs the submission task.
On the other hand, the performance the present invention provides a kind of distributed system platform optimizes device, which includes：
Acquiring unit, can be used for obtaining the configuration file for carrying number of partitions configuration information, the configuration file is
It is passed to when distributed system platform receives task；
Resolution unit, can be used for parsing the configuration file, obtain the number of partitions configuration information；
Updating block, can be used for according to the number of partitions configuration information, to point in the distributed system platform
Area's quantity is updated.
Further, described device further includes：Substituting unit and dispensing unit；
The substituting unit, for the number of partitions for needing to set in the distributed system platform to be passed through the default overall situation
Identifier is substituted；
The dispensing unit, for the configuration-direct according to the configuration file received, to the subregion of the configuration file
Number of partitions corresponding with the default global identifier is configured in quantity configuration information；
The acquiring unit, specifically for obtaining with the configuration file postponed.
Specifically, the updating block, specifically for by the distributed system platform with the default overall identification
Corresponding number of partitions is accorded with, replaces with the number of partitions corresponding with the default global identifier in the number of partitions configuration information
Amount.
Further, described device further includes：Detection unit；
The detection unit, for be spaced to schedule the detection distributed system platform whether receive it is new
Configuration file；
The acquiring unit, if detecting that distributed system platform have received new match somebody with somebody specifically for the detection unit
File is put, then obtains the recently received configuration file of the distributed system platform.
Further, described device further includes：
Trigger element, subregion is divided for triggering the distributed system platform according to the number of partitions after renewal, so that
Obtain the distributed system platform and pass through the execution of the subregion after the division submission task.
By above-mentioned technical proposal, technical solution provided in an embodiment of the present invention at least has following advantages：
The performance optimization method and device of a kind of distributed system platform provided by the invention, obtain carry subregion first
The configuration file of quantity configuration information, configuration file are passed to when distributed system platform receives task；Then institute is parsed
Configuration file is stated, obtains number of partitions configuration information；Finally according to the number of partitions configuration information, to the distributed system
Number of partitions in platform is updated.Number of partitions phase is adjusted with being required for modification code every time at present and repacking
Than the configuration file configured is submitted to distributed system platform and is parsed by the present invention, to be taken according in configuration file
The number of partitions configuration information of band is adjusted number of partitions, it is possible to achieve when carrying out distributed system performance optimization every time, only
Need to change exterior arrangement file, you can modification number of partitions, it is not necessary to modify and code and repack every time, and then
The tuning time is greatlyd save, improves the regulated efficiency of number of partitions, so as to improve the performance optimization effect of distributed system
Fruit.
Brief description of the drawings
By reading the detailed description of hereafter preferred embodiment, it is various other the advantages of and benefit it is common for this area
Technical staff will be clear understanding.Attached drawing is only used for showing the purpose of preferred embodiment, and is not considered as to the present invention
Limitation.And in whole attached drawing, identical component is denoted by the same reference numerals.In the accompanying drawings：
Fig. 1 shows a kind of performance optimization method flow signal of distributed system platform provided in an embodiment of the present invention
Figure；
Fig. 2 shows the performance optimization method flow signal of another distributed system platform provided in an embodiment of the present invention
Figure；
Fig. 3 shows a kind of performance optimization apparatus structure signal of distributed system platform provided in an embodiment of the present invention
Figure；
Fig. 4 shows the performance optimization apparatus structure signal of another distributed system platform provided in an embodiment of the present invention
Figure.
Embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although the disclosure is shown in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
Limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
The performance optimization method of a kind of distributed system platform provided in an embodiment of the present invention, as shown in Figure 1, this method bag
Include：
101st, the configuration file for carrying number of partitions configuration information is obtained.
Wherein, configuration file is passed to when distributed system platform receives task.Particular content in configuration file
It can carry out writing configuration in advance by the number of partitions that technical staff adjusts as needed.Being included in number of partitions configuration information needs
The number of partitions to be configured.The number of partitions can be the partition data amount of each elasticity distribution formula data set.
Executive agent for the embodiment of the present invention can be that configuration is used to adjust number of partitions in a distributed system
Device.Specifically, a global identifier can be pre-set, for representing number of partitions, i.e., needs to set in distributed system
The place of number of partitions is put, can be represented with this global identifier, then the global identifier can pass through configuration file
It is configured, is passed to when the configuration file is by submitting task in distributed system, when the device detection of adjustment number of partitions
When receiving configuration file to distributed system, the configuration file can be obtained, to be parsed to the configuration file.
102nd, configuration file is parsed, obtains number of partitions configuration information.
For example, the exterior arrangement file that parsing is incoming, the configuration file is acquired according to the configuration item pre-defined
In the number of partitions configuration information that includes, and then the number of partitions that can be needed to configure.
103rd, according to number of partitions configuration information, the number of partitions in distributed system platform is updated.
Specifically, can be by the subregion in distributed system platform after the number of partitions needed to configure is acquired
Quantity replaces with this number of partitions acquired, realizes the number of partitions adjustment in distributed system, and then performing tool
During the task of body, subregion is divided according to the number of partitions after adjustment, and task is performed by subregion after division, so as to reach point
The purpose of cloth system function optimization, and follow-up when needing adjust number of partitions again, need to change configuration file.
A kind of performance optimization method of distributed system platform provided in an embodiment of the present invention, obtains carry subregion first
The configuration file of quantity configuration information, configuration file are passed to when distributed system platform receives task；Then institute is parsed
Configuration file is stated, obtains number of partitions configuration information；Finally according to number of partitions configuration information, in distributed system platform
Number of partitions is updated.With being required for modification code every time at present and repacking come compared with adjusting number of partitions, the present invention
The configuration file configured is submitted to distributed system platform and parsed by embodiment, so as to according to being carried in configuration file
Number of partitions configuration information is adjusted number of partitions, it is possible to achieve when carrying out distributed system performance optimization every time, it is only necessary to
Change exterior arrangement file, you can modification number of partitions, it is not necessary to modify and code and repack every time, and then significantly
The tuning time has been saved, has improved the regulated efficiency of number of partitions, so as to improve the performance effect of optimization of distributed system.
Specifically, an embodiment of the present invention provides the performance optimization method of another distributed system platform, with spark points
Exemplified by the adjustment of Partition number of partitions in cloth system platform, as shown in Fig. 2, this method includes：
201st, whether detection distributed system platform in interval receives new configuration file to schedule.
Wherein, predetermined time interval can be set as the case may be, can be 1 minute, 10 minutes etc., the present invention
Embodiment does not limit.
In embodiments of the present invention, before step 201, the above method further includes：It will be needed in distributed system platform
The number of partitions of setting is substituted by default global identifier；According to the configuration-direct of the configuration file received, to
Number of partitions corresponding with default global identifier in the number of partitions configuration information of file is put to be configured.Taken in configuration file
With number of partitions configuration information, number of partitions corresponding with default global identifier is included in number of partitions configuration information, in advance
If global identifier can be configured according to the actual requirements, be specifically as follows can be globally unique identifier.
For example, a global identifier is pre-set, for representing Partition number of partitions, the i.e. distributed systems of Spark
The place of setting Partition number of partitions is needed in system, can be represented with this global identifier, and this overall identification
Symbol can be configured by configuration file, and then user can need the Partition that adjusts well defined in configuration file
Number of partitions.The composition of configuration file can be xxxxx.properties, and wherein xxxxx represents filename, can arbitrarily repair
Change, but be consistent all the time with the profile name in the distributed system program write.
If the 202, distributed system platform have received new configuration file, the newest reception of distributed system platform is obtained
The configuration file arrived.
Wherein, the configuration file is passed to when distributed system platform receives task.
For example, new configuration file whether is received every 1 minute detection Spark distributed systems platform, if Spark points
Cloth system platform receives new configuration file, such as at this time exterior arrangement file by submitting Spark tasks to be submitted to Spark
In distributed system platform, the recently received configuration file of Spark distributed system platforms is obtained.
It should be noted that being detected automatically by being spaced to schedule, distributed system can be timely obtained
The new configuration file received on system platform, to carry out number of partitions adjustment in time, and then can improve distributed system
The effect of performance optimization.
203rd, configuration file is parsed, obtains number of partitions configuration information.
For example, parsing incoming configuration file, and the partition of the inside is got according to the configuration item pre-defined
Number of partitions, such as spark.partition.num=21, wherein spark.partition.num are used to represent spark
Partition, i.e., default global identifier, and 21 represent number, the partition number of partitions for being spark is 21.
204th, by number of partitions corresponding with default global identifier in distributed system platform, replace with number of partitions and match somebody with somebody
Number of partitions corresponding with default global identifier in confidence breath.
For example, when detecting that spark distributed system platforms receive new configuration file, obtain and parse and receive
Configuration file, partition number of partitions is obtained as 25, according to overall identification corresponding with the partition number of partitions
Symbol, determines to need the position a for replacing partition number of partitions parameters in spark distributed system platforms, by distributed system
Partition number of partitions 12 original position a replaces with 25 in platform, and then completes the adjustment of partition number of partitions,
Since partition number of partitions determines the ability of the concurrent operation of spark distributed system platforms, by partition points
Area's quantity, which is heightened, can accordingly improve spark distributed system platform concurrent capabilities；Then when spark distributed systems platform again
When receiving a configuration file, the configuration file received is obtained and parses, it is 35 to obtain partition number of partitions, will
Partition number of partitions 25 before a of position replaces with 35.
In embodiments of the present invention, by exterior configuration file, and number of partitions information configuration in exterior arrangement text
In part, when needing to carry out performance optimization to distributed system, it can directly change exterior arrangement file and optimize, without repairing
Change code and transmit, and then greatly save the tuning time, improve the regulated efficiency of number of partitions, so as to improve point
The performance effect of optimization of cloth system.
205th, trigger distributed system platform and subregion is divided according to the number of partitions after renewal.
Distributed system platform is triggered by above-mentioned steps 205 subregion is divided according to the number of partitions after renewal, can make
Obtain distributed system platform and pass through the execution submission task of the subregion after division.
For example, after partition number of partitions is adjusted, when spark distributed systems platform performs specific tasks, just
Specific partition subregions can be divided, so as to reach spark points according to the partition number of partitions after this adjustment
Cloth system platform performance optimization purpose, and subsequently also need to adjustment partition number of partitions when, only need to change configuration
File.
Further, in addition to adjusting number of partitions using the mode of configuration file, the above method can also include：When
When the predefined parameter information that distributed system platform receives has renewal, the number of partitions included in predefined parameter information is obtained
Configuration information；According to number of partitions configuration information, the number of partitions in the distributed system platform is updated.Wherein,
Predefined parameter information can make choice configuration according to the actual requirements.
For example, the configuration information of partition number of partitions can be pre-configured in the conf parameters of spark, profit
With the submit orders of spark, it is passed to when submitting spark tasks by the conf parameters in spark distributed platforms, so
Triggering is parsed afterwards, obtains partition number of partitions therein, and complete subregion according to the partition number of partitions
The adjustment of quantity, so that spark distributed systems platform is when performing specific tasks, according to the partition after this adjustment
Number of partitions, divides specific partition subregions, so that achieveed the purpose that spark distributed systems platform property optimizes,
And when subsequently also needing to adjustment partition number of partitions, only it need to change conf parameters.
The performance optimization method of another kind distributed system platform provided in an embodiment of the present invention, by schedule
Whether interval detection distributed system platform receives new configuration file, can timely obtain and be connect on distributed system platform
Received new configuration file, to carry out number of partitions adjustment in time, and then can improve distributed system performance optimization
Effect；Then configuration file is parsed, obtains number of partitions configuration information；Finally according to number of partitions configuration information, to distribution
Number of partitions in system platform is updated.Number of partitions is adjusted with being at present required for modification code every time and repacking
Compare, the configuration file configured is submitted to distributed system platform and is parsed by the embodiment of the present invention, so as to according to configuration
The number of partitions configuration information carried in file is adjusted number of partitions, it is possible to achieve it is excellent to carry out distributed system performance every time
During change, it is only necessary to change exterior arrangement file every time, you can modification number of partitions, it is not necessary to modify and code and beat again
Bag, and then the tuning time is greatlyd save, the regulated efficiency of number of partitions is improved, so as to improve the performance of distributed system
Effect of optimization.
Further, the specific implementation as method shown in Fig. 1, an embodiment of the present invention provides a kind of distributed system to put down
The performance optimization device of platform, as shown in figure 3, described device includes：Acquiring unit 31, resolution unit 32, updating block 33.
Acquiring unit 31, can be used for obtaining and carries the configuration file of number of partitions configuration information, configuration file be
Distributed system platform receives what is be passed to during task.
Resolution unit 32, can be used for parsing the configuration file that acquiring unit 31 obtains, obtains number of partitions configuration information.
Updating block 33, can be used for parsing obtained number of partitions configuration information according to resolution unit 32, to distribution
Number of partitions in system platform is updated.
It should be noted that involved by a kind of performance optimization device of distributed system platform provided in an embodiment of the present invention
Other corresponding corresponding descriptions for describing, may be referred to Fig. 1 of each functional unit, details are not described herein.
A kind of performance optimization device of distributed system platform provided in an embodiment of the present invention, including acquiring unit, parsing
Unit, updating block etc., obtain the configuration file for carrying number of partitions configuration information, the configuration by acquiring unit first
File is passed to when distributed system platform receives submission task；Then the configuration file is parsed by resolution unit,
Obtain the number of partitions configuration information；Final updating unit is according to the number of partitions configuration information, to the distributed system
Number of partitions in system platform is updated.Number of partitions phase is adjusted with being required for modification code every time at present and repacking
Than the configuration file configured is submitted to distributed system platform and is parsed by the embodiment of the present invention, so as to according to configuration text
The number of partitions configuration information carried in part is adjusted number of partitions, it is possible to achieve carries out distributed system performance optimization every time
When, it is only necessary to change exterior arrangement file, you can modification number of partitions, it is not necessary to modify and code and beat again every time
Bag, and then the tuning time is greatlyd save, the regulated efficiency of number of partitions is improved, so as to improve the performance of distributed system
Effect of optimization.
Further, the specific implementation as method shown in Fig. 2, an embodiment of the present invention provides another distributed system
The performance optimization device of platform, as shown in figure 4, described device includes：Acquiring unit 41, resolution unit 42, updating block 43.
Acquiring unit 41, can be used for obtaining and carries the configuration file of number of partitions configuration information, configuration file be
Distributed system platform receives what is be passed to during task.
Resolution unit 42, can be used for parsing the configuration file that acquiring unit 41 obtains, obtains number of partitions configuration information.
Updating block 43, can be used for parsing obtained number of partitions configuration information according to resolution unit 42, to distribution
Number of partitions in system platform is updated.
Further, above device further includes：Substituting unit 44, dispensing unit 45.
Substituting unit 44, can be used for passing through the number of partitions for needing to set in the distributed system platform default complete
Office's identifier is substituted.
Dispensing unit 45, can be used for the configuration-direct according to the configuration file received, to the number of partitions of configuration file
Number of partitions corresponding with default global identifier is configured in amount configuration information.
Acquiring unit 41, specifically can be used for obtaining the configuration file with postponing.
Updating block 43, specifically can be used for the number of partitions corresponding with default global identifier in distributed system platform
Amount, replaces with number of partitions corresponding with default global identifier in number of partitions configuration information.
Further, above device further includes：Detection unit 46.
Detection unit 46, can be used for being spaced whether the detection distributed system platform receives newly to schedule
Configuration file.
Acquiring unit 41, if specifically can be used for detection unit 46 detects that distributed system platform have received new match somebody with somebody
File is put, then obtains the recently received configuration file of distributed system platform.
Further, above device further includes：Trigger element 47,
Trigger element 47, can be used for triggering distributed system platform and divides subregion according to the number of partitions after renewal, with
So that distributed system platform performs submission task by the subregion after division.
It should be noted that involved by the performance optimization device of another kind distributed system platform provided in an embodiment of the present invention
And other corresponding corresponding descriptions for describing, may be referred to Fig. 2 of each functional unit, details are not described herein.
A kind of performance optimization device of distributed system platform provided in an embodiment of the present invention, including acquiring unit, parsing
Unit, updating block, detection unit etc., are spaced whether detection distributed system platform connects by detection unit to schedule
New configuration file is received, the new configuration file received on distributed system platform can be timely obtained, so as to timely
Number of partitions adjustment is carried out, and then the effect of distributed system performance optimization can be improved；Then institute is parsed by resolution unit
Configuration file is stated, obtains the number of partitions configuration information；Final updating unit is according to number of partitions configuration information, to distribution
Number of partitions in system platform is updated.Number of partitions is adjusted with being at present required for modification code every time and repacking
Compare, the configuration file configured is submitted to distributed system platform and is parsed by the embodiment of the present invention, so as to according to configuration
The number of partitions configuration information carried in file is adjusted number of partitions, it is possible to achieve it is excellent to carry out distributed system performance every time
During change, it is only necessary to change exterior arrangement file every time, you can modification number of partitions, it is not necessary to modify and code and beat again
Bag, and then the tuning time is greatlyd save, the regulated efficiency of number of partitions is improved, so as to improve the performance of distributed system
Effect of optimization.
The performance optimization device of the distributed system platform includes processor and memory, above-mentioned acquiring unit, parsing
Unit, updating block, substituting unit, dispensing unit, detection unit, trigger element etc. are stored in memory as program unit
In, above procedure unit stored in memory is performed by processor to realize corresponding function.
Kernel is included in processor, is gone in memory to transfer corresponding program unit by kernel.Kernel can set one
Or more, solve to be required for modification code every time in the prior art and repack to adjust subregion by adjusting kernel parameter
The problem of quantity, can cause the regulated efficiency of number of partitions relatively low, and then the performance that have impact on distributed system optimizes.
Memory may include computer-readable medium in volatile memory, random access memory (RAM) and/
Or the form such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM), memory includes at least one deposit
Store up chip.
Present invention also provides a kind of computer program product, when being performed on data processing equipment, is adapted for carrying out just
The program code of beginningization there are as below methods step：Obtain the configuration file for carrying number of partitions configuration information, the configuration text
Part is passed to when distributed system platform receives task；Then the configuration file is parsed, the number of partitions is obtained and matches somebody with somebody
Confidence ceases；According to the number of partitions configuration information, the number of partitions in the distributed system platform is updated.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the application can use the computer for wherein including computer usable program code in one or more
The computer program production that usable storage medium is implemented on (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The application is with reference to the flow according to the method for the embodiment of the present application, equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that it can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or square frame in journey and/or square frame and flowchart and/or the block diagram.These computer programs can be provided
The processors of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that the instruction performed by computer or the processor of other programmable data processing devices, which produces, to be used in fact
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which produces, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided and is used for realization in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a square frame or multiple square frames.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net
Network interface and memory.
Memory may include computer-readable medium in volatile memory, random access memory (RAM) and/
Or the form such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flashRAM).Memory is computer-readable medium
Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer-readable instruction, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only storage (ROM), electric erasable
Programmable read only memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read-only storage (CD-ROM),
Digital versatile disc (DVD) or other optical storages, magnetic cassette tape, the storage of tape magnetic rigid disk or other magnetic storage apparatus
Or any other non-transmission medium, the information that can be accessed by a computing device available for storage.Define, calculate according to herein
Machine computer-readable recording medium does not include temporary computer readable media (transitory media), such as data-signal and carrier wave of modulation.
It these are only embodiments herein, be not limited to the application.To those skilled in the art,
The application can have various modifications and variations.All any modifications made within spirit herein and principle, equivalent substitution,
Improve etc., it should be included within the scope of claims hereof.
Claims (10)
- A kind of 1. performance optimization method of distributed system platform, it is characterised in that including：The configuration file for carrying number of partitions configuration information is obtained, the configuration file is to receive to appoint in distributed system platform It is passed to during business；The configuration file is parsed, obtains the number of partitions configuration information；According to the number of partitions configuration information, the number of partitions in the distributed system platform is updated.
- 2. the performance optimization method of distributed system platform according to claim 1, it is characterised in that described obtain carries Before the configuration file for having number of partitions configuration information, the method further includes：The number of partitions for needing to set in the distributed system platform is substituted by default global identifier；According to the configuration-direct of the configuration file received, in the number of partitions configuration information of the configuration file with it is described pre- If the corresponding number of partitions of global identifier is configured；It is described to obtain the configuration file for carrying number of partitions configuration information, specifically include：Obtain with the configuration file postponed.
- 3. the performance optimization method of distributed system platform according to claim 2, it is characterised in that described in the basis Number of partitions configuration information, is updated the number of partitions in the distributed system platform, specifically includes：By number of partitions corresponding with the default global identifier in the distributed system platform, the number of partitions is replaced with Measure number of partitions corresponding with the default global identifier in configuration information.
- 4. the performance optimization method of distributed system platform according to claim 1, it is characterised in that described obtain carries Before the configuration file for having number of partitions configuration information, the method further includes：Interval detects whether the distributed system platform receives new configuration file to schedule；If so, then obtain the recently received configuration file of the distributed system platform.
- 5. the performance optimization method of distributed system platform according to any one of claims 1 to 4, it is characterised in that institute State according to the number of partitions configuration information, it is described after being updated to the number of partitions in the distributed system platform Method further includes：Trigger the distributed system platform and subregion is divided according to the number of partitions after renewal, so that the distributed system is put down Platform performs the submission task by the subregion after division.
- A kind of 6. performance optimization device of distributed system platform, it is characterised in that including：Acquiring unit, the configuration file of number of partitions configuration information is carried for obtaining, the configuration file is in distribution System platform receives what is be passed to during task；Resolution unit, the configuration file obtained for parsing the acquiring unit, obtains the number of partitions configuration information；Updating block, for the number of partitions configuration information parsed according to the resolution unit, to the distributed system Number of partitions in platform is updated.
- 7. the performance optimization device of distributed system platform according to claim 6, it is characterised in that described device is also wrapped Include：Substituting unit and dispensing unit；The substituting unit, the number of partitions for that will need to set in the distributed system platform pass through default overall identification Symbol is substituted；The dispensing unit, for the configuration-direct according to the configuration file received, to the number of partitions of the configuration file Number of partitions corresponding with the default global identifier is configured in configuration information；The acquiring unit, specifically for obtaining with the configuration file postponed.
- 8. the performance optimization device of distributed system platform according to claim 7, it is characterised in thatThe updating block, specifically for by subregion corresponding with the default global identifier in the distributed system platform Quantity, replaces with number of partitions corresponding with the default global identifier in the number of partitions configuration information.
- 9. the performance optimization device of distributed system platform according to claim 6, it is characterised in that described device is also wrapped Include：Detection unit；The detection unit, for being spaced whether the detection distributed system platform receives new configuration to schedule File；The acquiring unit, if detecting that distributed system platform have received new configuration text specifically for the detection unit Part, then obtain the recently received configuration file of the distributed system platform.
- 10. device is optimized according to the performance of claim 6 to 9 any one of them distributed system platform, it is characterised in that institute Device is stated to further include：Trigger element, divides subregion, so that institute for triggering the distributed system platform according to the number of partitions after renewal State distributed system platform and the submission task is performed by the subregion after division.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610894140.7A CN107948227B (en) | 2016-10-13 | 2016-10-13 | Performance optimization method and device of distributed system platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610894140.7A CN107948227B (en) | 2016-10-13 | 2016-10-13 | Performance optimization method and device of distributed system platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107948227A true CN107948227A (en) | 2018-04-20 |
| CN107948227B CN107948227B (en) | 2021-06-08 |
Family
ID=61928448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610894140.7A Active CN107948227B (en) | 2016-10-13 | 2016-10-13 | Performance optimization method and device of distributed system platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107948227B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116443388A (en) * | 2023-06-12 | 2023-07-18 | 合肥联宝信息技术有限公司 | Labeling system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040268342A1 (en) * | 2003-06-30 | 2004-12-30 | Dell Products L.P. | System for automated generation of config to order software stacks |
| CN104899561A (en) * | 2015-05-27 | 2015-09-09 | 华南理工大学 | Parallelized human body behavior identification method |
| CN105550296A (en) * | 2015-12-10 | 2016-05-04 | 深圳市华讯方舟软件技术有限公司 | Data importing method based on spark-SQL big data processing platform |
| CN105740424A (en) * | 2016-01-29 | 2016-07-06 | 湖南大学 | Spark platform based high efficiency text classification method |
-
2016
- 2016-10-13 CN CN201610894140.7A patent/CN107948227B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040268342A1 (en) * | 2003-06-30 | 2004-12-30 | Dell Products L.P. | System for automated generation of config to order software stacks |
| CN104899561A (en) * | 2015-05-27 | 2015-09-09 | 华南理工大学 | Parallelized human body behavior identification method |
| CN105550296A (en) * | 2015-12-10 | 2016-05-04 | 深圳市华讯方舟软件技术有限公司 | Data importing method based on spark-SQL big data processing platform |
| CN105740424A (en) * | 2016-01-29 | 2016-07-06 | 湖南大学 | Spark platform based high efficiency text classification method |
Non-Patent Citations (1)
| Title |
|---|
| 白银大法师: "spark中tasks数量的设置", 《HTTPS://BLOG.CSDN.NET/MASK1188/ARTICLE/DETAILS/52013828》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116443388A (en) * | 2023-06-12 | 2023-07-18 | 合肥联宝信息技术有限公司 | Labeling system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107948227B (en) | 2021-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107450941B (en) | Automatic packaging method, device, storage medium and computer equipment | |
| CN104133772B (en) | Automatic test data generation method | |
| US9280339B1 (en) | Class replacer during application installation | |
| US9471470B2 (en) | Automatically recommending test suite from historical data based on randomized evolutionary techniques | |
| US9921569B2 (en) | Field device commissioning system and method | |
| CN106302008A (en) | Data-updating method and device | |
| CN107609004B (en) | Application program embedding method and device, computer equipment and storage medium | |
| US8484617B2 (en) | Process-driven feedback of digital asset re-use | |
| EP2889767B1 (en) | Server provisioning based on job history analysis | |
| US20130125092A1 (en) | Generating deployable code from simulation models | |
| US10462261B2 (en) | System and method for configuring a data access system | |
| CN106201861A (en) | The detection method of a kind of code quality and device | |
| CN106611345A (en) | A method and apparatus for acquiring user behavior data | |
| US9098497B1 (en) | Methods and systems for building a search service application | |
| CN103714004A (en) | JVM online memory leak analysis method and system | |
| CN106897342A (en) | A kind of data verification method and equipment | |
| CN109634682A (en) | The configuration file update method and device of application program | |
| CN109388614A (en) | A kind of method, system and the equipment of catalogue file number quota | |
| CN110069676A (en) | Keyword recommendation method and device | |
| CN109445832A (en) | Language carries out the method and electronic equipment of hot update to application program based on programming | |
| US20140222871A1 (en) | Techniques for data assignment from an external distributed file system to a database management system | |
| CN106648839A (en) | Method and device for processing data | |
| US9569516B2 (en) | Method and device for executing an enterprise process | |
| CN111681071A (en) | Sub-cost data generation system and method, storage medium, and electronic device | |
| CN107085613A (en) | Enter the filter method and device of library file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Cuigong Hotel, 76 Zhichun Road, Shuangyushu District, Haidian District, Beijing Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |