CN105550305A - Map/reduce-based real-time response method and system - Google Patents
Map/reduce-based real-time response method and system Download PDFInfo
- Publication number
- CN105550305A CN105550305A CN201510923573.6A CN201510923573A CN105550305A CN 105550305 A CN105550305 A CN 105550305A CN 201510923573 A CN201510923573 A CN 201510923573A CN 105550305 A CN105550305 A CN 105550305A
- Authority
- CN
- China
- Prior art keywords
- module
- service request
- request information
- mapper
- main control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
Abstract
The invention discloses a map/reduce-based real-time response method and system. The method comprises the following steps: a master control module initializes and starts map/reduce operation, and then the master control module receives a service request message sent from a service request module; the master control module distributes the service request message to an idle mapper module; and the idle mapper module receives the service request message, judges whether the service request message is a service termination message or not, processes a service request if the service request message is not the service termination message, takes a processing result as a service response message, returns the service response message to the master control module and waits for a next service request message. The problem of low efficiency when a conventional map/reduce program is used for work which needs to be quickly responded and involves processing of a great number of small-data tasks is solved, and the data processing efficiency is improved.
Description
Technical field
The present invention relates to data processing technique, particularly relate to a kind of real-time response method and system based on map/reduce.
Background technology
Before parallel computational model map/reduce occurs, to programme field at java, utilize multiple stage computer parallel processing system data, allow and to carry out cooperating very difficult between these computing machines, cause to be associated between mass data.
Map/reduce model provides the disposal route to mass data, its principle is the computing machine unified management multiple stage in a LAN (Local Area Network) only with common computing power, make them can parallel processing mass data, and among the process of process data, carry out the calculating of having shared out the work and helped one another some service logics, realize being associated between mass data or carrying out statistical study with this.
Map/reduce program is mainly divided into 3 modules: main control module, mapper module and reduce module.Wherein, main control module is responsible for initialization, and mapper module and reduce module in charge carry out service logic calculating and process.
Once map/reduce computing starts, the service request information being submitted to hadoop platform is just sent to main control module by service request module, service request information is distributed to mapper module and processes by main control module, main control module is notified after mapper resume module, and exiting this map/reduce computing, main control module again can start next map/reduce computing or carry out other process.
Map/reduce model can be transferred interior all computing machines of trooping and carry out parallel computation and collaborative process data, and the powerful of it is apparent.But in general application scenarios, map/reduce also has its deficiency, such as, the time of each startup optimization of map/reduce all needs 30 to 50 seconds, after starting if each, map/reduce only performs little time, and the efficiency of map/reduce is just very low.Therefore, existing map/reduce program for needing quick response, process the work of a large amount of small data task time, efficiency is lower.
Summary of the invention
In view of this, the object of the invention is to propose a kind of real-time response method and system based on map/reduce, to solve existing map/reduce program for needing quick response, processing the lower problem of the work limitation rate of a large amount of small data task, improve data-handling efficiency.
For achieving the above object, the present invention adopts following technical scheme:
On the one hand, embodiments provide a kind of real-time response method based on map/reduce, comprising:
Main control module initialization after starting map/reduce computing, described main control module receives the service request information that service request module sends;
Described service request information is distributed to idle mapper module by described main control module;
Described idle mapper module receives described service request information, judge whether described service request information is the message stopping service, if not, then processes services request, and result is returned to after main control module as service response message, the next service request information of wait-for-response.
Further, also comprise:
If judge, described service request information is the message stopping service, then the operation of described idle mapper end-of-module wait-for-response service request information.
Further, also comprise before starting map/reduce computing in main control module initialization:
Described main control module is that each mapper module is numbered;
Correspondingly, described service request information is distributed to idle mapper module by described main control module, comprising:
Described main control module obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module;
Described service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module by described main control module.
Further, described main control module is that each mapper module is numbered and comprises:
Described main control module obtains a multirow being pre-created and often row has a digital text;
Described main control module sends a line text of described text to each mapper module, to make each mapper module obtain the numeral of different texts, and using the numbering of described numeral as mapper module.
Further, when described idle mapper module responds services request, also comprise:
Stand-by period arranges module and the stand-by period of hadoop system acceptance mapper module progress report is set to 0, to prevent described idle mapper module by hadoop system force termination.
On the other hand, embodiments provide a kind of real-time response system based on map/reduce, comprise service request module, main control module and mapper module;
Described service request module comprises service request information transmitting element, for sending service request information to described main control module; And service response message receiving element, for receiving the service response message that described main control module forwards;
Described main control module comprises map/reduce start unit, starts map/reduce computing for initialization; Service request information receiving element, for receiving the service request information that described service request module sends; Service request information allocation units, for distributing to idle mapper module by described service request information; And service response message retransmission unit, for receiving the service response message that idle mapper module returns, and described service response message is transmitted to described service request module;
Described mapper module comprises service request information second receiving element, for receiving the service request information that main control module distributes; Service request information judging unit, for judging whether described service request information is the message stopping service; Service response message returns unit, if for judging described service request information not as stopping the message of service, then process services request, and result is returned to main control module as service response message; And service request information waits for unit, for after described service response message is returned to main control module, the next service request information of wait-for-response.
Further, described mapper module also comprises:
Response stops unit, if for judging that described service request information is the message stopping service, then stop the operation of described idle mapper module wait-for-response service request information.
Further, described main control module also comprises:
Mapper module numbered cell, in main control module initialization and before starting map/reduce computing, for each mapper module is numbered;
Correspondingly, described service request information allocation units specifically for:
Obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module;
Described service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module.
Further, described mapper module numbered cell specifically for:
Obtain a multirow being pre-created and often row has a digital text;
A line text of described text is sent, to make each mapper module obtain the numeral of different texts, and using the numbering of described numeral as mapper module to each mapper module.
Further, described system also comprises:
Stand-by period arranges module, for before described idle mapper resume module services request, the stand-by period of hadoop system acceptance mapper module progress report is set to 0, to prevent described idle mapper module by hadoop system force termination.
The invention has the beneficial effects as follows: a kind of real-time response method and system based on map/reduce provided by the invention, when using the work of a large amount of small data task of map/reduce routine processes, after map/reduce computing one-shot, mapper module just can wait-for-response service request information always, when mapper module receives service request information, when judging service request information not as stopping the message of serving, services request is processed, and result is returned to main control module as service response message, afterwards, mapper module continues wait-for-response service request information, once receive service request information, just make an immediate response this services request, and again need not open map/reduce computing, thus the response time is decreased, improve the efficiency of data processing.
Accompanying drawing explanation
Exemplary embodiment of the present invention will be described in detail by referring to accompanying drawing below, the person of ordinary skill in the art is more clear that above-mentioned and other feature and advantage of the present invention, in accompanying drawing:
Fig. 1 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention one provides;
Fig. 2 is the mapper module work schematic flow sheet that the embodiment of the present invention one provides;
Fig. 3 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention two provides;
Fig. 4 is the schematic flow sheet be numbered mapper module that the embodiment of the present invention two provides;
Fig. 5 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention three provides;
Fig. 6 is the structured flowchart of the real-time response system based on map/reduce that the embodiment of the present invention four provides;
Fig. 7 is the structured flowchart of the service request module that the embodiment of the present invention four provides;
Fig. 8 is the structured flowchart of the main control module that the embodiment of the present invention four provides;
Fig. 9 is the structured flowchart of the mapper module that the embodiment of the present invention four provides;
Figure 10 is the structured flowchart of the real-time response system preferred version based on map/reduce that the embodiment of the present invention four provides;
Figure 11 is the fundamental diagram of the real-time response system based on map/reduce that the embodiment of the present invention five provides.
Embodiment
Technical scheme of the present invention is further illustrated by embodiment below in conjunction with accompanying drawing.Be understandable that, specific embodiment described herein is only for explaining the present invention, but not limitation of the invention.It also should be noted that, for convenience of description, illustrate only part related to the present invention in accompanying drawing but not entire infrastructure.
Embodiment one
Fig. 1 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention one provides.The method is not only applicable to the situation simultaneously using multiple computers parallel computation, when being also applicable to the work performing a large amount of small data task, realize the situation of real-time response service request information, the method can be performed by the real-time response system based on map/reduce, and this system can be realized by the mode of software and/or hardware.As shown in Figure 1, the method comprises:
Step 110, main control module initialization after starting map/reduce computing, main control module receives the service request information that service request module sends.
Wherein, service request information comprises the request message such as the logical calculated of business, the statistical study of data.In the present embodiment, service request information is submitted to hadoop platform by user, when performing the process operation to service request information, just can initialization start map/reduce computing based on the main control module in the real-time response system of map/reduce, service request information is sent to main control module by service request module, meanwhile, this service request information of mapper module wait-receiving mode, to respond this service request information.
Service request information is distributed to idle mapper module by step 120, main control module.
Service request information is distributed to idle mapper module by main control module, to improve treatment effeciency.Exemplary, after main control module receives service request information, check idle mapper module, service request information is distributed to suitable idle mapper module, wherein, suitable idle mapper module can be the mapper module that priority is the highest, or processes the fastest mapper module of this service request information.
Step 130, idle mapper module receive service request information, judge whether service request information is the message stopping service.
In this operation, stop the message of service for stopping mapper resume module service request information, the message of this termination service is the character that pre-sets of user or character string etc., can issue according to the disposition of business or data, such as, after data statistic analysis, user can submit to " OK " to real-time response system.
If step 140 judges service request information not as stopping the message of service, then idle mapper resume module services request, and returns to after main control module using result as service response message, the next service request information of wait-for-response.
In the present embodiment, if judge service request information not as stopping the message of service, idle mapper module is after processing a service request information, this idle mapper module can not be out of service, but whether detection in real time has other service request information to send over, the next service request information of wait-for-response.
The real-time response method based on map/reduce that the embodiment of the present invention one provides, when using the work of a large amount of small data task of map/reduce routine processes, after map/reduce computing one-shot, mapper module just can wait-for-response service request information always, when mapper module receives service request information, when judging service request information not as stopping the message of serving, services request is processed, and result is returned to main control module as service response message, afterwards, mapper module continues wait-for-response service request information, once receive service request information, just make an immediate response this services request, and again need not open map/reduce computing, thus the response time is decreased, improve the efficiency of data processing.
Further, in such scheme, if judge, service request information is the message stopping service, then the operation of idle mapper end-of-module wait-for-response service request information.
Thus, idle mapper module stops receiving service request information, completes the work processing all service request information.
Accordingly, see Fig. 2, in the present embodiment, mapper module performs following operation:
Step 210, wait-for-response service request information.
In the present embodiment, map/reduce computing is once start, and mapper module is moment wait-for-response service request information just, to receive service request information, process service request information and return service response message to main control module.
Step 220, reception service request information.
Step 230, judge that whether service request information is the message stopping service; If so, then step 240 is performed, otherwise, perform step 250.
The operation of step 240, termination wait-for-response service request information.
Exemplary, after data statistic analysis, user can submit to the message " OK " of the service of termination to real-time response system, and mapper module is just out of service after receiving the message of this termination service.
Step 250, process services request, and result is returned to main control module as service response message; Return step 210.
In the present embodiment, when judging service request information not as stopping the message of serving, mapper module can circulate always receive service request information, process service request information and return service response message to the operation of main control module.
Embodiment two
Fig. 3 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention two provides.The present embodiment is optimized based on embodiment one, before starting map/reduce computing, adds step: main control module is that each mapper module is numbered in main control module initialization; Correspondingly, the service request information that service request module is sent by main control module distributes to idle mapper module, comprising: main control module receives the service request information that service request module sends; Main control module obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module; Service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module by main control module.
Thus, main control module is that each mapper module is numbered, so that main control module selects suitable idle mapper module to process service request information.
Further, main control module is that each mapper module is numbered and comprises:
Main control module creates a multirow and often row has a digital text;
A line text that main control module sends text file to each mapper module, to make each mapper module obtain the numeral of different texts, and using the numbering of numeral as mapper module.
Accordingly, the method for the present embodiment comprises the steps:
Step 310, main control module are that each mapper module is numbered.
In the present embodiment, main control module is that each mapper module is numbered, and to distinguish different mapper modules, numbering can be the different character such as numeral, letter, and the numbering of each mapper module has uniqueness.
Exemplary, see Fig. 4, this step can comprise step 311 and step 312 further:
Step 311, main control module obtain a multirow being pre-created and often row has a digital text.
Wherein, the numeral of often going is all not identical, and multirow numeral order can arrange from top to bottom from 1, and also can arrange at random, above-mentioned text can be stored in advance in the server of hadoop platform.
A line text that step 312, main control module send text file to each mapper module, to make each mapper module obtain the numeral of different texts, and using the numbering of numeral as mapper module.
Exemplary, main control module is split text, a line text is sent to a mapper module, mapper module obtains the numeral of text, set up the corresponding relation of numeral and mapper module, realize the numbering to mapper module, the numbering of each mapper module has uniqueness.
Step 320, main control module initialization after starting map/reduce computing, main control module receives the service request information that service request module sends.
Step 331, main control module obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module.
In the present embodiment, main control module can check the state of each mapper module, and mark the state of each mapper module for " free time " or " busy ", wherein, " free time " represents that mapper module is in wait-for-response service request information, and " busy " represents that mapper module is in process service request information.Main control module by the state of each mapper module of inquiry, obtain the numbering of available free mapper module, select the numbering of an idle mapper module according to the priority of idle mapper module or processing speed.
Service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module by step 332, main control module.
Exemplary, after the numbering choosing idle mapper module, main control module is direct idle mapper module service request information being sent to the numbering correspondence of selected idle mapper module just.
Step 340, idle mapper module receive service request information, judge whether service request information is the message stopping service.
If step 350 judges service request information not as stopping the message of service, then idle mapper resume module services request, and returns to after main control module using result as service response message, the next service request information of wait-for-response.
The real-time response method based on map/reduce that the embodiment of the present invention two provides, in main control module initialization and before starting map/reduce computing, main control module is that each mapper module is numbered, and facilitates main control module to select suitable idle mapper module to process service request information.
Embodiment three
Fig. 5 is the schematic flow sheet of the real-time response method based on map/reduce that the embodiment of the present invention three provides.The present embodiment is optimized based on above-described embodiment, before idle mapper resume module services request, adds step: the stand-by period arranges module and the stand-by period of hadoop system acceptance mapper module progress report is set to 0.
Thus, long playing mapper module can be prevented by hadoop system force termination, make mapper module before receiving the message of end of service, can real-time response service request information always.
Accordingly, the method for the present embodiment comprises the steps:
Step 410, main control module initialization after starting map/reduce computing, main control module receives the service request information that service request module sends.
Service request information is distributed to idle mapper module by step 420, main control module.
Step 430, idle mapper module receive service request information, judge whether service request information is the message stopping service.
Step 440, stand-by period arrange module and the stand-by period of hadoop system acceptance mapper module progress report are set to 0.
Exemplary, at set intervals, mapper module can send progress report to hadoop system, to show normal operation, if hadoop system does not receive the progress report of mapper module for a long time, can think that task is overtime, and by the mapper module force termination of wait-for-response service request information, make this mapper module cannot continue to process service request information.Therefore, the stand-by period of the present embodiment arranges module and the stand-by period of hadoop system acceptance mapper module progress report can be set to 0, namely mapred.task.timeout=0 is set, to make mapper module always in running order before the message receiving the service of termination, improve the stability of real-time response system cloud gray model.
If step 450 judges service request information not as stopping the message of service, then idle mapper resume module services request, and returns to after main control module using result as service response message, the next service request information of wait-for-response.
The real-time response method based on map/reduce that the embodiment of the present invention three provides, module is set by the stand-by period stand-by period of hadoop system acceptance mapper module progress report is set to 0, make long playing mapper module by hadoop system force termination, the stability of real-time response system cloud gray model can not be improve.
Embodiment four
Fig. 6 is the structured flowchart of the real-time response system based on map/reduce that the embodiment of the present invention four provides.As shown in Figure 6, this system comprises: service request module 10, main control module 20 and mapper module 30;
Wherein, see Fig. 7, service request module 10 comprises service request information transmitting element 101, for sending service request information to main control module 20; And service response message receiving element 102, for receiving the service response message that main control module 20 forwards;
See Fig. 8, main control module 20 comprises map/reduce start unit 201, starts map/reduce computing for initialization; Service request information first receiving element 202, for receiving the service request information that service request module 10 sends; Service request information allocation units 203, for distributing to idle mapper module by service request information; And service response message retransmission unit 204, for receiving the service response message that idle mapper module returns, and service response message is transmitted to service request module 10;
Service request information second receiving element 301 is comprised, for receiving the service request information that main control module 20 distributes see Fig. 9, mapper module 30; Service request information judging unit 302, for judging whether service request information is the message stopping service; Service response message returns unit 303, if for judging service request information not as stopping the message of service, then process services request, and result is returned to main control module 20 as service response message; And service request information waits for unit 304, for after service response message is returned to main control module 20, the next service request information of wait-for-response.
The real-time response system based on map/reduce that the embodiment of the present invention four provides may be used for the real-time response method based on map/reduce that the execution embodiment of the present invention provides, and possesses corresponding function and beneficial effect.
Further, see Fig. 9, in such scheme, mapper module 30 also comprises:
Response stops unit 305, if for judging that service request information is the message stopping service, then stop the operation of idle mapper module wait-for-response service request information.
Further, see Fig. 8, in such scheme, main control module 20 also comprises:
Mapper module numbered cell 205, in main control module 20 initialization and before starting map/reduce computing, for each mapper module 30 is numbered;
Correspondingly, service request information allocation units specifically for:
Obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module;
Service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module.
Preferably, mapper module numbered cell 205 specifically for:
Obtain a multirow being pre-created and often row has a digital text;
To a line text that each mapper module 20 sends text file, to make each mapper module 20 obtain the numeral of different texts, and using the numbering of numeral as mapper module 20.
Further, see Figure 10, in such scheme, the real-time response system based on map/reduce also comprises:
Stand-by period arranges module 40, for before idle mapper resume module services request, the stand-by period of hadoop system acceptance mapper module progress report is set to 0, to prevent idle mapper module by hadoop system force termination.
Embodiment five
Figure 11 is the fundamental diagram of the real-time response system based on map/reduce that the embodiment of the present invention five provides.
As shown in figure 11, main control module is numbered for each mapper module in advance, and when starting operation, main control module initialization also starts map/reduce computing, now, and mapper module wait-for-response services request; Service request module sends service request information to main control module, after main control module receives service request information, select the numbering of idle mapper module, and distribute above-mentioned service request module to the idle mapper module of the numbering correspondence of selected idle mapper module, now, main control module arranges the state of selected idle mapper module is busy; This idle mapper module receives above-mentioned service request information, when judging this service request information not as stopping the message of serving, this service request information is processed, and result is returned to main control module as service response message, afterwards, this idle mapper module does not quit work, but the next service request information of wait-for-response always, until the message receiving the service of termination just stops process service request information; Main control module receives service response message, and service response message is transmitted to service request module, service response message is received to make service request module, result is checked for user, now, the state that main control module arranges above-mentioned selected idle mapper module is for idle, so that main control module continues to other service request information of this idle mapper module assignment.
Thus, as long as should an initialization be performed based on the real-time response system of map/reduce and start the operation of map/reduce computing, multiple mapper module just can be made to process multiple service request information simultaneously, and each mapper module can ceaselessly process different service request information, after saving first time process service request information, each process service request information all will reinitialize and start the time of map/reduce computing, reaches the effect of real-time response service request information.
Note, above are only preferred embodiment of the present invention and institute's application technology principle.Skilled person in the art will appreciate that and the invention is not restricted to specific embodiment described here, various obvious change can be carried out for a person skilled in the art, readjust and substitute and can not protection scope of the present invention be departed from.Therefore, although be described in further detail invention has been by above embodiment, the present invention is not limited only to above embodiment, when not departing from the present invention's design, can also comprise other Equivalent embodiments more, and scope of the present invention is determined by appended right.
Claims (10)
1., based on a real-time response method of map/reduce, it is characterized in that, comprising:
Main control module initialization after starting map/reduce computing, described main control module receives the service request information that service request module sends;
Described service request information is distributed to idle mapper module by described main control module;
Described idle mapper module receives described service request information, judge whether described service request information is the message stopping service, if not, then processes services request, and result is returned to after main control module as service response message, the next service request information of wait-for-response.
2. method according to claim 1, is characterized in that, also comprises:
If judge, described service request information is the message stopping service, then the operation of described idle mapper end-of-module wait-for-response service request information.
3. method according to claim 1, is characterized in that, also to comprise before starting map/reduce computing in main control module initialization:
Described main control module is that each mapper module is numbered;
Correspondingly, described service request information is distributed to idle mapper module by described main control module, comprising:
Described main control module obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module;
Described service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module by described main control module.
4. method according to claim 3, is characterized in that, described main control module is that each mapper module is numbered and comprises:
Described main control module obtains a multirow being pre-created and often row has a digital text;
Described main control module sends a line text of described text to each mapper module, to make each mapper module obtain the numeral of different texts, and using the numbering of described numeral as mapper module.
5. method according to claim 1, is characterized in that, before described idle mapper resume module services request, also comprises:
Stand-by period arranges module and the stand-by period of hadoop system acceptance mapper module progress report is set to 0, to prevent described idle mapper module by hadoop system force termination.
6. based on a real-time response system of map/reduce, it is characterized in that, comprise service request module, main control module and mapper module;
Described service request module comprises service request information transmitting element, for sending service request information to described main control module; And service response message receiving element, for receiving the service response message that described main control module forwards;
Described main control module comprises map/reduce start unit, starts map/reduce computing for initialization; Service request information first receiving element, for receiving the service request information that described service request module sends; Service request information allocation units, for distributing to idle mapper module by described service request information; And service response message retransmission unit, for receiving the service response message that idle mapper module returns, and described service response message is transmitted to described service request module;
Described mapper module comprises service request information second receiving element, for receiving the service request information that main control module distributes; Service request information judging unit, for judging whether described service request information is the message stopping service; Service response message returns unit, if for judging described service request information not as stopping the message of service, then process services request, and result is returned to main control module as service response message; And service request information waits for unit, for after described service response message is returned to main control module, the next service request information of wait-for-response.
7. system according to claim 6, is characterized in that, described mapper module also comprises:
Response stops unit, if for judging that described service request information is the message stopping service, then stop the operation of described idle mapper module wait-for-response service request information.
8. system according to claim 6, is characterized in that, described main control module also comprises:
Mapper module numbered cell, in main control module initialization and before starting map/reduce computing, for each mapper module is numbered;
Correspondingly, described service request information allocation units specifically for:
Obtain the numbering of available free mapper module, and select the numbering of one of them idle mapper module;
Described service request information is sent to the idle mapper module of the numbering correspondence of selected idle mapper module.
9. system according to claim 8, is characterized in that, described mapper module numbered cell specifically for:
Obtain a multirow being pre-created and often row has a digital text;
A line text of described text is sent, to make each mapper module obtain the numeral of different texts, and using the numbering of described numeral as mapper module to each mapper module.
10. system according to claim 6, is characterized in that, described system also comprises:
Stand-by period arranges module, for before described idle mapper resume module services request, the stand-by period of hadoop system acceptance mapper module progress report is set to 0, to prevent described idle mapper module by hadoop system force termination.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510923573.6A CN105550305B (en) | 2015-12-14 | 2015-12-14 | A kind of real-time response method and system based on map/reduce |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510923573.6A CN105550305B (en) | 2015-12-14 | 2015-12-14 | A kind of real-time response method and system based on map/reduce |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105550305A true CN105550305A (en) | 2016-05-04 |
| CN105550305B CN105550305B (en) | 2019-11-22 |
Family
ID=55829494
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510923573.6A Active CN105550305B (en) | 2015-12-14 | 2015-12-14 | A kind of real-time response method and system based on map/reduce |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105550305B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193643A (en) * | 2017-03-28 | 2017-09-22 | 中国科学院信息工程研究所 | A kind of highly reliable distributed traffic real-time statistical method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102209087A (en) * | 2010-03-31 | 2011-10-05 | 国际商业机器公司 | Method and system for MapReduce data transmission in data center having SAN |
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
| CN102866918A (en) * | 2012-07-26 | 2013-01-09 | 中国科学院信息工程研究所 | Resource management system for distributed programming framework |
| CN102999377A (en) * | 2012-11-30 | 2013-03-27 | 北京东方通科技股份有限公司 | Service concurrent access control method and device |
| CN103970520A (en) * | 2013-01-31 | 2014-08-06 | 国际商业机器公司 | Resource management method and device in MapReduce framework and framework system with device |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
-
2015
- 2015-12-14 CN CN201510923573.6A patent/CN105550305B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102209087A (en) * | 2010-03-31 | 2011-10-05 | 国际商业机器公司 | Method and system for MapReduce data transmission in data center having SAN |
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
| CN102866918A (en) * | 2012-07-26 | 2013-01-09 | 中国科学院信息工程研究所 | Resource management system for distributed programming framework |
| CN102999377A (en) * | 2012-11-30 | 2013-03-27 | 北京东方通科技股份有限公司 | Service concurrent access control method and device |
| CN103970520A (en) * | 2013-01-31 | 2014-08-06 | 国际商业机器公司 | Resource management method and device in MapReduce framework and framework system with device |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193643A (en) * | 2017-03-28 | 2017-09-22 | 中国科学院信息工程研究所 | A kind of highly reliable distributed traffic real-time statistical method and system |
| CN107193643B (en) * | 2017-03-28 | 2019-11-05 | 中国科学院信息工程研究所 | A kind of highly reliable distributed traffic real-time statistical method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105550305B (en) | 2019-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7463544B2 (en) | Blockchain message processing method, apparatus, computer device, and computer program | |
| CN107087019B (en) | Task scheduling method and device based on end cloud cooperative computing architecture | |
| Kliazovich et al. | CA-DAG: Modeling communication-aware applications for scheduling in cloud computing | |
| US20160378570A1 (en) | Techniques for Offloading Computational Tasks between Nodes | |
| CN107025139A (en) | A kind of high-performance calculation Scheduling Framework based on cloud computing | |
| CN104834722A (en) | CDN (Content Delivery Network)-based content management system | |
| CN111770157B (en) | Business processing method and device, electronic equipment and storage medium | |
| CN101986272A (en) | Task scheduling method under cloud computing environment | |
| CN101951411A (en) | Cloud scheduling system and method and multistage cloud scheduling system | |
| CN104158909B (en) | A kind of disributed media processing method and system thereof | |
| CN101179553A (en) | Efficient order-preserving delivery method and device for concurrent messages | |
| CN114024972B (en) | Long connection communication method, system, device, equipment and storage medium | |
| Kliazovich et al. | CA-DAG: Communication-aware directed acyclic graphs for modeling cloud computing applications | |
| CN102904961A (en) | Method and system for scheduling cloud computing resources | |
| WO2017185615A1 (en) | Method for determining service status of service processing device and scheduling device | |
| CN102724132A (en) | Method and device for improving transmission control protocol (TCP) connection multiplexing processing efficiency | |
| CN102025783A (en) | Cluster system, message processing method thereof and protocol forward gateway | |
| US11256547B2 (en) | Efficient allocation of cloud computing resources to job requests | |
| CN105049485A (en) | Real-time video processing oriented load-aware cloud calculation system | |
| CN103763206B (en) | Network scheduling method and gateway | |
| CN110300188A (en) | Data transmission system, method and apparatus | |
| US20130219386A1 (en) | Dynamic allocation of compute resources | |
| CN105786447A (en) | Method and apparatus for processing data by server and server | |
| CN102891851A (en) | Access control method, equipment and system of virtual desktop | |
| CN104468710A (en) | Mixed big data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |