CN108021450A - Job analysis method and apparatus based on YARN - Google Patents
Job analysis method and apparatus based on YARN Download PDFInfo
- Publication number
- CN108021450A CN108021450A CN201711262758.2A CN201711262758A CN108021450A CN 108021450 A CN108021450 A CN 108021450A CN 201711262758 A CN201711262758 A CN 201711262758A CN 108021450 A CN108021450 A CN 108021450A
- Authority
- CN
- China
- Prior art keywords
- carrying
- information
- yarn
- resource
- resource occupation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Abstract
This application provides a kind of job analysis method and apparatus based on YARN, this method includes：Access the declarative state transfer application programming interfaces of Hadoop clusters YARN, the job information for obtaining the total resources of Hadoop clusters and being carrying out；All operations being carrying out are traveled through, the resource occupation information of each operation being carrying out is obtained and the resources occupation rate of each operation is analyzed according to total resources.Whether the resource occupation for prejudging out every operation in advance occurs abnormal or there is a possibility that exception occur, prevents YARN from contention for resources and exception occur in service peak period.
Description
Technical field
This application involves Internet service technology, and in particular to a kind of job analysis method and apparatus based on YARN.
Background technology
Hadoop, which is one, can allow the light framework of user and the Distributed Computing Platform that uses.User can easily exist
The application program of processing mass data is developed and run on Hadoop.
Another resource coordination person (Yet Another Resource Negotiator, YARN) is a kind of Hadoop moneys
Source manager, it is a universal resource management system, and unified resource management and scheduling can be provided for upper layer application.It draws
Enter and bring big advantages in utilization rate, resource unified management and data sharing etc. for cluster.
The basic thought of YARN is by job trace device on the basis of first generation distributed computing system (MapReduce)
(JobTracker) two major functions (resource management and job scheduling/monitoring) separation.Main method is to create one entirely
Office resource manager (ResourceManager, RM) and several for application program application manager
(Application Master, AM).Here application program refers to (MapReduce) operation of traditional Distributed Calculation or work
The directed acyclic graph (Directed Acyclic Graph, DAG) of industry.At present usually using MapReduce or cluster meter of increasing income
Calculate environment Spark to handle task, YARN contention for resources and exception easily occurs in service peak period.
The content of the invention
In view of this, the embodiment of the present application provides a kind of job analysis method and apparatus based on YARN, to solve
Easily there is the technical problem of contention for resources in service peak period in YARN.
According to the one side of the embodiment of the present application, there is provided a kind of job analysis method based on YARN, the method
Including：Access the declarative state transfer application programming interfaces of Hadoop clusters YARN, obtain Hadoop clusters total resources and
The job information being carrying out；All operations being carrying out are traveled through, obtain the resource occupation letter of each operation being carrying out
Cease and the resources occupation rate of each operation is analyzed according to the total resources.
In one embodiment, the job analysis method based on YARN further includes：According to the total resources and
The job information of execution determines that each operation needle reaches different resource classification the standard of abnormal case.
In one embodiment, the job analysis method based on YARN further includes：The operation that each is carrying out
Resource occupation information be stored in MySQL database；The MySQL database is controlled to gather the resource occupation information of operation
Close, the resource occupation information of each operation is obtained according to default measurement period rank.
Preferably, the MySQL database is controlled to polymerize the resource occupation information of operation, according to default statistics week
The resource occupation information that phase rank obtains each operation includes：Control resource occupation information of the MySQL database to operation
It is polymerize, each operation resource occupation is obtained using day as measurement period rank and abnormal number occurs and occurs every time abnormal
Duration.
Preferably, the job analysis method based on YARN further includes：Described in being inquired about using structured query language
The resource occupation information of MySQL database storage, report is generated according to query result；By the report with mail he send to
Preset electronic addresses of items of mail.
Preferably, the job analysis method based on YARN further includes：Data display system at regular intervals is controlled from described
The data pulled every time are simultaneously generated corresponding report and are shown by MySQL database pulling data.
According to the other side of the embodiment of the present application, there is provided a kind of operation analysis apparatus based on YARN, the dress
Put including：Resource information acquisition module, the declarative state for accessing Hadoop clusters YARN shift application programming interfaces, obtain
Take the total resources of Hadoop clusters and the job information being carrying out；Operation source obtaining module, for traveling through all holding
Capable operation, the resource for obtaining the resource occupation information of each operation being carrying out and each operation being analyzed according to total resources account for
With rate.
In one embodiment, the operation analysis apparatus based on YARN further includes：Anomaly standard determining module, is used for
Determine that each operation needle reaches abnormal case to different resource classification according to the total resources and the job information being carrying out
Standard.
In one embodiment, the operation analysis apparatus based on YARN further includes：Information storage module, for will be every
The resource occupation information for the operation that item is carrying out is stored in MySQL database；Statisti-cal control module, it is described for controlling
MySQL database polymerize the resource occupation information of operation, and the money of each operation is obtained according to default measurement period rank
Source occupied information.
Preferably, Statisti-cal control module is further used for：Control resource occupation information of the MySQL database to operation
It is polymerize, each operation resource occupation is obtained using day as measurement period rank and abnormal number occurs and occurs every time abnormal
Duration.
Preferably, the operation analysis apparatus based on YARN further includes：Enquiry module, for utilizing structuralized query language
Speech inquires about the resource occupation information of the MySQL database storage, and report is generated according to query result；Report sending module, is used
In being sent the report to preset electronic addresses of items of mail with mail he.
Preferably, the operation analysis apparatus based on YARN further includes：Display systems control module, for controlling data
The data pulled every time periodically from the MySQL database pulling data, are generated corresponding report and are shown by display systems
Show.
According to the other side of the embodiment of the present application, there is provided a kind of electronic equipment, including memory and processor；Deposit
Reservoir is used to store one or more computer instruction, wherein, one or more computer is performed by the processor to realize
The above-mentioned job analysis method based on YARN.
According to the other side of the embodiment of the present application, there is provided a kind of computer-readable recording medium, is stored thereon with
The above-mentioned job analysis method based on YARN is realized in computer instruction, the instruction when being executed by processor.
The beneficial effect of the embodiment of the present application includes：The operation for obtaining the total resources of Hadoop clusters and being carrying out
(Job) information, travels through the resources occupation rate of currently performed each operation of job analysis, prejudges out the resource of every operation in advance
Whether occupancy occurs abnormal or there is a possibility that to occur abnormal, prevents YARN from contention for resources and different occur servicing peak period
Often.
Brief description of the drawings
By the description to the embodiment of the present application referring to the drawings, the above-mentioned and other purpose of the application, feature and
Advantage will be apparent from, in the accompanying drawings：
Fig. 1 is group operation analysis system configuration diagram of the embodiment of the present application based on YARN；
Fig. 2 is the flow chart of the job analysis method provided by the embodiments of the present application based on YARN；
Fig. 3 is the block diagram of the operation analysis apparatus provided by the embodiments of the present application based on YARN；
Fig. 4 is the structure diagram of electronic equipment provided by the embodiments of the present application.
Embodiment
The application is described below based on embodiment, but the application is not restricted to these embodiments.Under
Text is detailed to describe some specific detail sections in the detailed description of the application.Do not have for a person skilled in the art
The description of these detail sections can also understand the application completely.In order to avoid obscuring the essence of the application, known method, mistake
The not narration in detail of journey, flow, element and circuit.
In addition, it should be understood by one skilled in the art that provided herein attached drawing be provided to explanation purpose, and
What attached drawing was not necessarily drawn to scale.
Unless the context clearly requires otherwise, otherwise entire disclosure is similar with the " comprising " in claims, "comprising" etc.
Word should be construed to the implication included rather than exclusive or exhaustive implication；That is, it is containing for " including but not limited to "
Justice.
In the description of the present application, it is to be understood that term " first ", " second " etc. are only used for description purpose, without
It is understood that to indicate or implying relative importance.In addition, in the description of the present application, unless otherwise indicated, the implication of " multiple "
It is two or more.
The embodiment of the present application shifts (REpresentational State by the declarative state of YARN Page Services
Transfer, REST) application programming interface (Application Programming Interface, API) acquisition
The total resources of Hadoop clusters and operation (Job) information being carrying out, travel through currently performed each operation of job analysis
Whether resources occupation rate, the resource occupation for prejudging out every operation in advance occur abnormal or there is a possibility that exception occur,
Prevent YARN from contention for resources and exception occur in service peak period.
Fig. 1 is the configuration diagram of group operation analysis system of the embodiment of the present application based on YARN, including Hadoop collection
Group 10, client 11, MySQL database 12 and data display systems 13.Wherein, the inside of Hadoop clusters 10 includes YARN 101
With multiple active jobs 102, YARN 101 carries out operation 102 unified resource management and scheduling.YARN 101 is run
In Hadoop clusters 10, meet the constraints of RESTful and principle and externally provide REST API.
REST refers to one group of framework constraints and principle, meets the application programming of these constraintss and principle
It is exactly RESTful.REST uses Uniform Resource Identifier (Uniform Resource Identifier, URI), plus super text
This transport protocol (HyperText Transfer Protocol, HTTP) requesting method is published on interconnection so as to reach to one
Uniquely description and the operation of the resource of net.The Page Service that REST frameworks define effectively defines an interface specification, is a kind of
The Interface design of resourceoriented, all Interface designs, come what is designed, the application entity on network are all made both for resource
Treat for resource.
The Page Service REST API of Hadoop YARN are one group to be believed towards cluster, node, application program and applicating history
Breath provides the URI resources accessed, which is grouped into the type based on API return informations.According to API return informations
Type, these URI resources can be referred to different groups, some API return to collector (collector) type, some return single
Example pattern (singleton) type.The grammer of Page Service REST API is as follows：
http://{http address of service}/ws/{version}/{resourcepath}；
Wherein, { http address of service } represents the server address for needing to obtain information, supports to access money
Source manager (ResourceManager), node manager (NodeManager), Distributed Calculation application manager
(MapReduce application master) and history server (history server)；{ version } represents API's
Version, supports v1 at present；{ resourcepath } defines the path of singleton resources or collector resources.
If calling REST API, application program should perform the URI associated with resource HTTP operation (GET please
Ask), obtain corresponding resource.
Client 11 calls the Page Service REST API of YARN 101, obtains the total resources information of Hadoop clusters 10,
Including CPU occupancy, EMS memory occupation etc.；And the job information being carrying out is obtained, such as the operation being carrying out
“application_1388830974669_1540349”。
All operations being carrying out are traveled through by REST API again, obtain the resource occupation of each operation being carrying out
Information, with reference to the resources occupation rate of each operation being carrying out of the total resources information analysis of Hadoop clusters 10, so as in advance
Whether the resource occupation of the every operation of anticipation there is possibility abnormal or with the presence or absence of appearance exception.
Client 11 can periodically obtain the total resources of Hadoop clusters 10, and the dilatation of Hadoop clusters 10,
Capacity reducing is fully transparent to client 11.Client 11 is determined according to the total resources information got and the job information being carrying out
Each operation needle reaches different resource classification the standard of abnormal case, such as CPU takes threshold value, EMS memory occupation threshold value, operation
Perform time threshold etc..If 10 dilatation of Hadoop clusters, capacity reducing cause total resources information to change, different resource class
The standard for not reaching abnormal case also accordingly changes therewith.
Client 11 can store the resource occupation information for the operation being carrying out to MySQL database 12, with structure
The data mode of change preserves.Client 11 controls MySQL database 12 to be accounted for according to default measurement period rank to the resource of operation
It is polymerize with information, counts the resource occupation information of each operation of the rank, for example, with " my god " for measurement period rank
CPU occupancy, EMS memory occupation, there is the number of abnormal case, the duration of each abnormal case etc..
Client 11 can carry out inquiry operation using structured query language (SQL) to MySQL database 12, according to
The query result generation report that MySQL database 12 returns.Report is sent in a manner of Email default by client 11
E-mail address.For example, presetting the E-mail address of research staff in client 11, research staff inputs in client 11
After SQL query statement, the query result of return is generated report and is sent report by way of Email by client 11
To default E-mail address, checked for research staff, easy to determine follow-up prioritization scheme.
Client 11 can also control the resource occupation information access data display system 13 of each operation got
Data display system 13 is periodically or periodically from 11 pulling data of client.The cycle of 13 pulling data of data display system
The cycle synchronisation or same frequency of REST API can be accessed with client 11.The data that data display system 13 will pull every time
Report is generated in real time and by screen display, easy to research and develop and operation maintenance personnel real time inspection.
Based on described above, the embodiment of the present application provides a kind of job analysis method based on YARN, suitable for client
End 11, as shown in Fig. 2, this method comprises the following steps.
S20, accesses the declarative state transfer application programming interfaces of Hadoop clusters YARN, obtains the total of Hadoop clusters
Resource and the job information being carrying out；
S21, travels through all operations being carrying out, and obtains the resource occupation information and basis of each operation being carrying out
Total resources analyze the resources occupation rate of each operation.
In the present embodiment, the resource occupation of each operation being carrying out is obtained by the Page Service REST API of YARN
Information simultaneously analyzes its resources occupation rate, to check whether each operation resource occupation exception occurs or predict in advance whether to deposit
There is the possibility of abnormal case, preventing YARN from contention for resources and exception occur in service peak period.
In one embodiment, the job analysis method based on YARN further includes：
S22, determines that each operation needle reaches different to different resource classification according to total resources and the job information being carrying out
The standard of reason shape.
The present embodiment integrates the job run situation of Hadoop clusters, can dynamically adjust the resource occupation of every operation
Reach the standard of abnormal case, not only ensured the Effec-tive Function of operation, but also prevent contention for resources.
In one embodiment, the job analysis method based on YARN further includes：
S23, MySQL database is stored in by the resource occupation information of each operation being carrying out；
S24, control MySQL database polymerize the resource occupation information of operation, according to default measurement period rank
Obtain the resource occupation information of each operation.
Preferably, S24 is configured to control MySQL database to polymerize the resource occupation information of operation,
When each operation resource occupation is obtained using day as measurement period rank there is abnormal number and occur abnormal lasting every time
Between.
The present embodiment polymerize resource occupation information in MySQL database, obtains default measurement period rank
Occupation condition, easy to be analyzed from various time dimensions the occupation condition of Hadoop clusters.
Preferably, the job analysis method based on YARN is somebody's turn to do to further comprise：
S25, using the resource occupation information of structured query language inquiry MySQL database storage, according to query result
Generate report；
S26, report is sent to preset electronic addresses of items of mail with mail he.
The query result that MySQL database returns is sent to the E-mail address of research staff, easy to it with mail he
Check and determine follow-up prioritization scheme.
Preferably, the job analysis method based on YARN is somebody's turn to do can further include：
Control data display system at regular intervals from MySQL database pulling data and generate the data pulled every time corresponding
Report is simultaneously shown.
The resource occupation data got are shown in real time, easy to research and develop and operation maintenance personnel Real Time Observation Hadoop
The resource service condition of cluster.
The corresponding above-mentioned job analysis method based on YARN, the embodiment of the present application provide a kind of operation based on YARN point
Analysis apparatus, as shown in figure 3, the device includes：
Resource information acquisition module 30, the declarative state for accessing Hadoop clusters YARN shift application program and connect
Mouthful, obtain the total resources of the Hadoop clusters and the job information being carrying out；
Operation source obtaining module 31, for traveling through all operations being carrying out, obtains each operation being carrying out
Resource occupation information and the resources occupation rate of each operation is analyzed according to total resources.
In one embodiment, the operation analysis apparatus based on YARN further includes：
Anomaly standard determining module, for determining each operation needle to not according to total resources and the job information being carrying out
Reach the standard of abnormal case with resource class.
In one embodiment, the operation analysis apparatus based on YARN further includes：
Information storage module, the resource occupation information of the operation for each to be carrying out are stored in MySQL database；
Statisti-cal control module, for controlling MySQL database to polymerize the resource occupation information of operation, according to default
Measurement period rank obtains the resource occupation information of each operation.
Preferably, Statisti-cal control module is further used for：MySQL database is controlled to carry out the resource occupation information of operation
Polymerization, each operation resource occupation is obtained using day as measurement period rank and abnormal number occurs and occurs abnormal hold every time
The continuous time.
Preferably, the operation analysis apparatus based on YARN further comprises：
Enquiry module, for inquiring about the resource occupation information of MySQL database storage using structured query language, according to
Query result generates report；
Report sending module, for being sent report to preset electronic addresses of items of mail with mail he.
Preferably, the operation analysis apparatus based on YARN further comprises：
Display systems control module, will be each for controlling data display system at regular intervals from MySQL database pulling data
The data pulled generate corresponding report and are shown.
In addition, the above-mentioned operation analysis apparatus based on YARN can also pass through hardware processor (hardware
Processor) above-mentioned each program step is realized.Below with reference to Fig. 4, it illustrates suitable for for realizing the embodiment of the present application
Electronic equipment computer system 600 structure diagram.Electronic equipment shown in Fig. 4 is only an example, should not be right
The function and use scope of the embodiment of the present application bring any restrictions.
As shown in figure 4, computer system 600 includes central processing unit (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into program in random access storage device (RAM) 603 from storage part 608 and
Perform various appropriate actions and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
I/O interfaces 605 are connected to lower component：Importation 606 including keyboard, mouse etc.；Penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage part 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net performs communication process.Driver 610 is also according to needing to be connected to I/O interfaces 606.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., are installed on driver 610, in order to read from it as needed
Computer program be mounted into as needed storage part 608.
Especially, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product, it includes being carried on computer-readable medium
On computer program, the computer program include be used for execution flow chart shown in method program code.In such reality
Apply in example, which can be downloaded and installed by communications portion 609 from network, and/or from detachable media
611 are mounted.When the computer program is performed by central processing unit (CPU) 601, perform what is limited in the present processes
Above-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media or
Computer-readable recording medium either the two any combination.Computer-readable recording medium for example can be --- but
Be not limited to --- electricity, magnetic, optical, electromagnetic, system, device or the device of infrared ray or semiconductor, or it is any more than combination.
The more specifically example of computer-readable recording medium can include but is not limited to：Electrical connection with one or more conducting wires,
Portable computer diskette, hard disk, random access storage device (RAM), read-only storage (ROM), erasable type may be programmed read-only deposit
Reservoir (EPROM or flash memory), optical fiber, portable compact disc read-only storage (CD-ROM), light storage device, magnetic memory
Part or above-mentioned any appropriate combination.In this application, computer-readable recording medium can any be included or store
The tangible medium of program, the program can be commanded the either device use or in connection of execution system, device.And
In the application, computer-readable signal media can include believing in a base band or as the data that a carrier wave part is propagated
Number, wherein carrying computer-readable program code.The data-signal of this propagation can take various forms, including but not
It is limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer
Any computer-readable medium beyond readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit use
In by instruction execution system, device either device use or program in connection.Included on computer-readable medium
Program code any appropriate medium can be used to transmit, include but not limited to：Wirelessly, electric wire, optical cable, RF etc., Huo Zheshang
Any appropriate combination stated.
Flow chart and block diagram in attached drawing, it is illustrated that according to the system of the various embodiments of the application, method and computer journey
Architectural framework in the cards, function and the operation of sequence product.At this point, each square frame in flow chart or block diagram can generation
The part of one module of table, program segment or code, a part for the module, program segment or code include one or more
The executable instruction of logic function as defined in being used for realization.It should also be noted that some as replace realization in, institute in square frame
The function of mark can also be with different from the order marked in attached drawing generation.For example, two square frames succeedingly represented are actual
On can perform substantially in parallel, they can also be performed in the opposite order sometimes, this is depending on involved function.Also
It is noted that the combination of each square frame and block diagram in block diagram and/or flow chart and/or the square frame in flow chart, Ke Yiyong
The dedicated hardware based systems of functions or operations as defined in execution is realized, or can be referred to specialized hardware and computer
The combination of order is realized.
Being described in module involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be set within a processor, for example, can be described as：A kind of processor bag
Include resource information acquisition module and operation source obtaining module.
As on the other hand, present invention also provides a kind of nonvolatile computer storage media, the non-volatile calculating
Machine storage medium can be nonvolatile computer storage media included in device described in above-described embodiment；Can also be
Individualism, without the nonvolatile computer storage media in supplying electronic equipment.Above-mentioned non-volatile computer storage is situated between
Matter is stored with one or more program, when one or more program stored is performed by an equipment so that described
Equipment：Access the declarative state transfer application programming interfaces of Hadoop clusters YARN, obtain Hadoop clusters total resources and
The job information being carrying out；All operations being carrying out are traveled through, obtain the resource occupation letter of each operation being carrying out
Cease and the resources occupation rate of each operation is analyzed according to total resources.
The foregoing is merely the preferred embodiment of the application, the application is not limited to, for those skilled in the art
For, the application can have various modifications and changes.All any modifications made within spirit herein and principle, be equal
Replace, improve etc., it should be included within the protection domain of the application.
Claims (10)
- A kind of 1. job analysis method based on YARN, it is characterised in that the described method includes：The declarative state transfer application programming interfaces of Hadoop clusters YARN are accessed, obtain the total resources of the Hadoop clusters With the job information being carrying out；All operations being carrying out are traveled through, obtain the resource occupation information of each operation being carrying out and according to total money Analyze the resources occupation rate of each operation in source.
- 2. according to the method described in claim 1, it is characterized in that, the method further includes：Determine that each operation needle reaches abnormal feelings to different resource classification according to the total resources and the job information being carrying out The standard of shape.
- 3. according to the method described in claim 1, it is characterized in that, the method further includes：The resource occupation information of each operation being carrying out is stored in MySQL database；Control the MySQL database to polymerize the resource occupation information of operation, obtained according to default measurement period rank The resource occupation information of each operation.
- 4. according to the method described in claim 3, it is characterized in that, control resource occupation of the MySQL database to operation Information is polymerize, and the resource occupation information of each operation is obtained according to default measurement period rank and is included：Control the MySQL database to polymerize the resource occupation information of operation, obtained using day as measurement period rank There is abnormal number and occurs the abnormal duration every time in each operation resource occupation.
- 5. according to the method described in claim 3, it is characterized in that, the method further includes：The resource occupation information of the MySQL database storage is inquired about using structured query language, is generated according to query result Report；The report is sent to preset electronic addresses of items of mail with mail he.
- 6. according to the method described in claim 3, it is characterized in that, the method further includes：Control data display system at regular intervals from the MySQL database pulling data and generate the data pulled every time corresponding Report is simultaneously shown.
- 7. a kind of operation analysis apparatus based on YARN, it is characterised in that described device includes：Resource information acquisition module, the declarative state for accessing Hadoop clusters YARN shift application programming interfaces, obtain The total resources of the Hadoop clusters and the job information being carrying out；Operation source obtaining module, for traveling through all operations being carrying out, obtains the resource of each operation being carrying out Occupied information and the resources occupation rate that each operation is analyzed according to the total resources.
- 8. device according to claim 7, it is characterised in that described device further includes：Anomaly standard determining module, for determining each operation needle to not according to the total resources and the job information being carrying out Reach the standard of abnormal case with resource class.
- 9. a kind of electronic equipment, it is characterised in that including memory and processor；The memory is used to store one or more Computer instruction, wherein, one or more computer is performed by the processor to realize that claim 1 to 6 is any one The step of item the method.
- 10. a kind of computer-readable recording medium, is stored thereon with computer instruction, it is characterised in that the instruction is by processor The step of claim 1 to 6 any one the method is realized during execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711262758.2A CN108021450A (en) | 2017-12-04 | 2017-12-04 | Job analysis method and apparatus based on YARN |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711262758.2A CN108021450A (en) | 2017-12-04 | 2017-12-04 | Job analysis method and apparatus based on YARN |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108021450A true CN108021450A (en) | 2018-05-11 |
Family
ID=62078335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711262758.2A Pending CN108021450A (en) | 2017-12-04 | 2017-12-04 | Job analysis method and apparatus based on YARN |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108021450A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109769020A (en) * | 2019-01-09 | 2019-05-17 | 网宿科技股份有限公司 | CDN cluster accelerates serve resources management method and device, electronic equipment and storage medium |
| CN110597621A (en) * | 2019-08-09 | 2019-12-20 | 苏宁金融科技(南京)有限公司 | Method and system for scheduling cluster resources |
| CN110795301A (en) * | 2018-08-01 | 2020-02-14 | 马上消费金融股份有限公司 | Job monitoring method, device, terminal and computer storage medium |
| CN111290917A (en) * | 2020-02-26 | 2020-06-16 | 深圳市云智融科技有限公司 | YARN-based resource monitoring method and device and terminal equipment |
| CN111414381A (en) * | 2020-03-04 | 2020-07-14 | 腾讯科技（深圳）有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111813587A (en) * | 2020-05-28 | 2020-10-23 | 国网山东省电力公司 | Software interface evaluation and fault early warning method and system |
| CN111813545A (en) * | 2020-06-29 | 2020-10-23 | 北京字节跳动网络技术有限公司 | Resource allocation method, device, medium and equipment |
| CN112749055A (en) * | 2020-12-29 | 2021-05-04 | 拉卡拉支付股份有限公司 | Resource consumption metering method and device, electronic equipment and storage medium |
| WO2024000859A1 (en) * | 2022-06-28 | 2024-01-04 | 深圳前海微众银行股份有限公司 | Job scheduling method, job scheduling apparatus, job scheduling system, and storage medium |
| CN111813587B (en) * | 2020-05-28 | 2024-04-26 | 国网山东省电力公司 | Software interface evaluation and fault early warning method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246592A (en) * | 2013-05-13 | 2013-08-14 | 北京搜狐新媒体信息技术有限公司 | Monitoring acquisition system and method |
| CN104915407A (en) * | 2015-06-03 | 2015-09-16 | 华中科技大学 | Resource scheduling method under Hadoop-based multi-job environment |
| CN105868070A (en) * | 2015-12-25 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Method and apparatus for determining resources consumed by tasks |
| WO2017079631A1 (en) * | 2015-11-05 | 2017-05-11 | Preventice Technologies, Inc. | Securing resources with a representational state transfer application program interface |
-
2017
- 2017-12-04 CN CN201711262758.2A patent/CN108021450A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246592A (en) * | 2013-05-13 | 2013-08-14 | 北京搜狐新媒体信息技术有限公司 | Monitoring acquisition system and method |
| CN104915407A (en) * | 2015-06-03 | 2015-09-16 | 华中科技大学 | Resource scheduling method under Hadoop-based multi-job environment |
| WO2017079631A1 (en) * | 2015-11-05 | 2017-05-11 | Preventice Technologies, Inc. | Securing resources with a representational state transfer application program interface |
| CN105868070A (en) * | 2015-12-25 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Method and apparatus for determining resources consumed by tasks |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795301A (en) * | 2018-08-01 | 2020-02-14 | 马上消费金融股份有限公司 | Job monitoring method, device, terminal and computer storage medium |
| CN109769020A (en) * | 2019-01-09 | 2019-05-17 | 网宿科技股份有限公司 | CDN cluster accelerates serve resources management method and device, electronic equipment and storage medium |
| CN109769020B (en) * | 2019-01-09 | 2021-09-24 | 网宿科技股份有限公司 | CDN cluster acceleration service resource management method and device, electronic equipment and storage medium |
| CN110597621A (en) * | 2019-08-09 | 2019-12-20 | 苏宁金融科技(南京)有限公司 | Method and system for scheduling cluster resources |
| CN111290917A (en) * | 2020-02-26 | 2020-06-16 | 深圳市云智融科技有限公司 | YARN-based resource monitoring method and device and terminal equipment |
| CN111414381A (en) * | 2020-03-04 | 2020-07-14 | 腾讯科技（深圳）有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111813587A (en) * | 2020-05-28 | 2020-10-23 | 国网山东省电力公司 | Software interface evaluation and fault early warning method and system |
| CN111813587B (en) * | 2020-05-28 | 2024-04-26 | 国网山东省电力公司 | Software interface evaluation and fault early warning method and system |
| CN111813545A (en) * | 2020-06-29 | 2020-10-23 | 北京字节跳动网络技术有限公司 | Resource allocation method, device, medium and equipment |
| CN112749055A (en) * | 2020-12-29 | 2021-05-04 | 拉卡拉支付股份有限公司 | Resource consumption metering method and device, electronic equipment and storage medium |
| WO2024000859A1 (en) * | 2022-06-28 | 2024-01-04 | 深圳前海微众银行股份有限公司 | Job scheduling method, job scheduling apparatus, job scheduling system, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108021450A (en) | Job analysis method and apparatus based on YARN | |
| US10713090B2 (en) | Context aware prioritization in a distributed environment using tiered queue allocation | |
| CN110096344A (en) | Task management method, system, server cluster and computer-readable medium | |
| TW201820165A (en) | Server and cloud computing resource optimization method thereof for cloud big data computing architecture | |
| CN110310034A (en) | A kind of service orchestration applied to SaaS, business flow processing method and apparatus | |
| US20180232433A1 (en) | System and Method for Dynamically Refactoring Business Data Objects | |
| CN110019080A (en) | Data access method and device | |
| CN110795147A (en) | Interface protocol file management method and device | |
| CN106464710A (en) | Profile-based cache management | |
| US9141936B2 (en) | Systems and methods for simulating a resource constrained process | |
| CN109726004A (en) | A kind of data processing method and device | |
| CN111126948A (en) | Processing method and device for approval process | |
| CN109343975A (en) | Method and apparatus for asynchronous communication between the page | |
| CN107347093A (en) | Collocation method and device for distributed server system | |
| CN109582528A (en) | State monitoring method, device, electronic equipment and computer readable storage medium | |
| CN112256695B (en) | Visualized graph calculation method and system, storage medium and electronic device | |
| CN111008767B (en) | Internet financial technology architecture evaluation method, device, electronic equipment and medium | |
| US20180210804A1 (en) | System and method for dynamic scaling of concurrent processing threads | |
| US11175970B2 (en) | Messaging in a multi-cloud computing environment | |
| US20200210164A1 (en) | Preventing database package updates to fail customer requests and cause data corruptions | |
| CN106851189A (en) | Video information processing method, system and device | |
| CN109582580A (en) | For debugging system, the method and apparatus of the page | |
| CN110347654A (en) | A kind of method and apparatus of online cluster features | |
| CN112000323B (en) | Data processing method and device | |
| CN109978486A (en) | Method, apparatus, system and medium for Row control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180511 |