US20170060538A1 - Fusion recommendation for performance management in streams - Google Patents
Fusion recommendation for performance management in streams Download PDFInfo
- Publication number
- US20170060538A1 US20170060538A1 US14/868,679 US201514868679A US2017060538A1 US 20170060538 A1 US20170060538 A1 US 20170060538A1 US 201514868679 A US201514868679 A US 201514868679A US 2017060538 A1 US2017060538 A1 US 2017060538A1
- Authority
- US
- United States
- Prior art keywords
- fusion
- streams
- data
- actors
- recommendation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
Definitions
- the present disclosure relates to stream computing, and more particularly to the use of an integrated development environment in streams computing.
- Database systems are typically configured to separate the process of storing data from accessing, manipulating, or using data stored in a database. More specifically, database systems use a model in which data is first stored and indexed in a memory before subsequent querying and analysis. In general, database systems may not be well suited for performing real-time processing and analyzing streaming data. In particular, database systems may be unable to store, index, and analyze large amounts of streaming data efficiently or in real time.
- An integrated development environment may be a software application that provides facilities to computer programmers for software development.
- An IDE may be designed to facilitate programmer productivity.
- An IDE may consist of a source code editor, build automation tools, and a debugger.
- An IDE may also have intelligent code completion capabilities.
- Embodiments of the present disclosure may be directed toward a method for recommending streams application actors for fusion, where the streams application actors are dispersed on an operator graph within a streaming environment.
- a system may generate a streams application actor profile for each of two or more streams application actors.
- the two or more streams application actors may be determined to be fusion candidates, based on the streams application actors satisfying a fusion candidate rule.
- Each of the fusion candidate rules may be associated with one or more fusion recommendations.
- a fusion recommendation associated with the rule may be identified and an identity of each of the two or more streams application actors along with the fusion recommendation may be displayed.
- Embodiments of the present disclosure may also be directed toward a system or computer program product for recommending streams application actors for fusion.
- Advantages of the invention over other art includes the ability to hint to developers an improvement or improvements for the streams system. Rather than requiring a developer have full understanding of both the streams coding environment as well as the current deficiencies or strengths of a particular streams operator graph, the system can identify fusion candidates and make recommendations to combine or separate the streams application actors according to parameters. In this way, a developer can make modifications and improvements to a system using current and informed metrics.
- the fusion recommendation may be to fuse the fusion candidates together in the same processing element. In other embodiments, the fusion recommendation may be to separate the two fusion candidates from the same processing element.
- some of the data in the streams application actor profile on which the fusion candidates can be identified can include historical central processing unit (CPU) usage, processing time, tuple rates, machine configuration, and current CPU usage.
- CPU central processing unit
- the system can analyze historical and real time data like CPU usage and processing time when determining which streams application actors can be identified as fusion candidates. This data or analysis of this data can be provided to a developer so the developer can make informed decisions.
- FIG. 1 illustrates a computing infrastructure configured to execute a stream computing application, according to various embodiments.
- FIG. 2 illustrates a more detailed view of a compute node of FIG. 1 , according to various embodiments.
- FIG. 3 illustrates a more detailed view of the management system of FIG. 1 , according to various embodiments.
- FIG. 4 illustrates a more detailed view of the development system of FIG. 1 , according to various embodiments.
- FIG. 5 illustrates an operator graph for a stream computing application, according to various embodiments.
- FIG. 6 illustrates a flow diagram for making fusion recommendations for streams application actors, according to various embodiments.
- FIG. 7 illustrates a block diagram of a user interface, according to various embodiments.
- FIG. 8 illustrates a block diagram of a user interface, according to various embodiments.
- FIG. 9 illustrates a tabular display, according to various embodiments.
- FIG. 10 illustrates a method for recommending fusion candidates in a streams computing environment, according to various embodiments.
- FIG. 11 depicts a cloud computing node according to an embodiment of the present invention.
- FIG. 12 depicts a cloud computing environment according to an embodiment of the present invention.
- FIG. 13 depicts abstraction model layers according to an embodiment of the present invention.
- aspects of the present disclosure relate to stream computing, and in particular, to the use of an integrated development environment in streams computing. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.
- Stream-based computing and stream-based database computing are emerging as a developing technology for database systems.
- Products are available which allow users to create applications that process and query streaming data before it reaches a database file.
- users can specify processing logic to apply to inbound data records while they are “in flight,” with the results available in a very short amount of time, often in fractions of a second. Constructing an application using this type of processing has opened up a new programming paradigm that will allow for development of a broad variety of innovative applications, systems, and processes, as well as present new challenges for application programmers and database developers.
- stream operators are connected to one another such that data flows from one stream operator to the next (e.g., over a TCP/IP socket).
- a stream operator may perform operations, such as analysis logic, which may change the tuple by adding or subtracting attributes, or updating the values of existing attributes within the tuple.
- analysis logic may change the tuple by adding or subtracting attributes, or updating the values of existing attributes within the tuple.
- a new tuple is then sent to the next stream operator.
- Scalability is achieved by distributing an application across nodes by creating executables (i.e., processing elements), as well as replicating processing elements on multiple nodes and load balancing among them.
- Stream operators in a stream computing application can be fused together to form a processing element that is executable.
- processing elements can be inserted or removed dynamically from an operator graph representing the flow of data through the stream computing application.
- a particular stream operator may not reside within the same operating system process as other stream operators.
- stream operators in the same operator graph may be hosted on different nodes, e.g., on different compute nodes or on different cores of a compute node.
- a tuple is a sequence of one or more attributes associated with an entity. Attributes may be any of a variety of different types, e.g., integer, float, Boolean, string, etc. The attributes may be ordered. In addition to attributes associated with an entity, a tuple may include metadata, i.e., data about the tuple. A tuple may be extended by adding one or more additional attributes or metadata to it.
- stream or “data stream” refers to a sequence of tuples. Generally, a stream may be considered a pseudo-infinite sequence of tuples.
- Tuples are received and output by stream operators and processing elements.
- An input tuple corresponding with a particular entity that is received by a stream operator or processing element is generally not considered to be the same tuple that is output by the stream operator or processing element, even if the output tuple corresponds with the same entity or data as the input tuple.
- An output tuple need not be changed in some way from the input tuple.
- an output tuple may be changed in some way by a stream operator or processing element.
- An attribute or metadata may be added, deleted, or modified.
- a tuple will often have two or more attributes.
- a stream operator or processing element may receive the tuple having multiple attributes and output a tuple corresponding with the input tuple. The stream operator or processing element may only change one of the attributes so that all of the attributes of the output tuple except one are the same as the attributes of the input tuple.
- a particular tuple output by a stream operator or processing element may not be considered to be the same tuple as a corresponding input tuple even if the input tuple is not changed by the processing element.
- an output tuple that has the same data attributes or is associated with the same entity as a corresponding input tuple will be referred to herein as the same tuple unless the context or an express statement indicates otherwise.
- Stream computing applications handle massive volumes of data that need to be processed efficiently and in real time. For example, a stream computing application may continuously ingest and analyze hundreds of thousands of messages per second and up to petabytes of data per day. Accordingly, each stream operator in a stream computing application may be required to process a received tuple within fractions of a second. Unless the stream operators are located in the same processing element, it is necessary to use an inter-process communication path each time a tuple is sent from one stream operator to another. Inter-process communication paths can be a critical resource in a stream computing application. According to various embodiments, the available bandwidth on one or more inter-process communication paths may be conserved. Efficient use of inter-process communication bandwidth can speed up processing.
- An operator graph can be an execution path for a plurality of stream operators to process a stream of tuples.
- the operator graph can refer to an execution path for processing elements and the dependent stream operators of the processing elements to process the stream of tuples.
- the operator graph can have a plurality of stream operators that produce a particular end result, e.g., calculate an average.
- computing applications may be distributed over a series of nodes by distributing processing elements that are connected together.
- a streams computing environment may deviate from a traditional computing environment.
- a computer programmer may not understand performance implications of the streams environment, including underutilized CPU usage and distribution of processing tasks over a variety of streams application actors (e.g., operators, processing elements, or others).
- Processing tasks could also be distributed over a variety of streams operators including functors, joins, or other operators designed for a particular task or tasks.
- fusion a process where streams application actors can be joined together within a single actor, may be recommended to a programmer in order to better distribute executables.
- separation of streams application actors may be recommended to a programmer.
- Fusion recommendations may recommend two or more streams application actors as fusion candidates.
- Fusion candidates may be streams application actors (e.g., operators) that are recommended to be fused together or be separated from each other or from a particular configuration (e.g., a same processing element).
- a fusion recommendation may indicate or “hint” at a combination or separation of fusion candidates, as is appropriate based on parameters and described herein.
- the streams application actors may be identified as fusion candidates based on profiles associated with the streams application actors.
- the profiles can include historical runtime data, real-time processing data, tuple flow data, machine configuration, or other data relevant processing time, processing demand, or capacity for each streams application actor.
- the data included in the profile for each streams application actor could also include data about the types of processes conducted by each streams application actor.
- the profile data could include other data deemed relevant by a system administrator or user to fusion recommendations.
- the system can detect a set of two or more streams application actors that is a potential fusion candidate.
- the two or more candidates may be identified based on one or more fusion candidate rules. These rules can be set by an administrator, determined by an individual programmer, or in another way. For example, a rule could be that if two operators are performing a similar function, and the input and output time of the first operator is a majority of the processing time for the operation, the first and second operators may be identified as the fusion candidate set. Once the set is identified as satisfying a fusion candidate rule, the identity of each of the streams application actors (e.g., operators one and two), along with the recommendation to fuse the two, may be displayed on a user interface.
- the identity of each of the streams application actors e.g., operators one and two
- the recommendation to fuse the two may be displayed on a user interface.
- This user interface may be a part of an integrated development environment. It could also be a part of another type of performance management display. Thus, a developer could receive indications or “hints” as to how the distribution of process across an operator graph may be improved. The developer can also be provided with real-time and historical data, and analysis of this data, to determine the appropriateness of the fusion recommendation to the particular operator graph.
- FIG. 1 illustrates one exemplary computing infrastructure 100 that may be configured to execute a stream computing application, according to some embodiments.
- the computing infrastructure 100 includes a management system 105 and two or more compute nodes 110 A— 110 D—i.e., hosts—which are communicatively coupled to each other using one or more communications networks 120 .
- the communications network 120 may include one or more servers, networks, or databases, and may use a particular communication protocol to transfer data between the compute nodes 110 A- 110 D.
- a development system 102 may be communicatively coupled with the management system 105 and the compute nodes 110 either directly or via the communications network 120 .
- the communications network 120 may include a variety of types of physical communication channels or “links.”
- the links may be wired, wireless, optical, or any other suitable media.
- the communications network 120 may include a variety of network hardware and software for performing routing, switching, and other functions, such as routers, switches, or bridges.
- the communications network 120 may be dedicated for use by a stream computing application or shared with other applications and users.
- the communications network 120 may be any size.
- the communications network 120 may include a single local area network or a wide area network spanning a large geographical area, such as the Internet.
- the links may provide different levels of bandwidth or capacity to transfer data at a particular rate.
- the bandwidth that a particular link provides may vary depending on a variety of factors, including the type of communication media and whether particular network hardware or software is functioning correctly or at full capacity.
- the bandwidth that a particular link provides to a stream computing application may vary if the link is shared with other applications and users.
- the available bandwidth may vary depending on the load placed on the link by the other applications and users.
- the bandwidth that a particular link provides may also vary depending on a temporal factor, such as time of day, day of week, day of month, or season.
- FIG. 2 is a more detailed view of a compute node 110 , which may be the same as one of the compute nodes 110 A- 110 D of FIG. 1 , according to various embodiments.
- the compute node 110 may include, without limitation, one or more processors (CPUs) 205 , a network interface 215 , an interconnect 220 , a memory 225 , and a storage 230 .
- the compute node 110 may also include an I/O device interface 210 used to connect I/O devices 212 , e.g., keyboard, display, and mouse devices, to the compute node 110 .
- I/O device interface 210 used to connect I/O devices 212 , e.g., keyboard, display, and mouse devices, to the compute node 110 .
- Each CPU 205 retrieves and executes programming instructions stored in the memory 225 or storage 230 . Similarly, the CPU 205 stores and retrieves application data residing in the memory 225 .
- the interconnect 220 is used to transmit programming instructions and application data between each CPU 205 , I/O device interface 210 , storage 230 , network interface 215 , and memory 225 .
- the interconnect 220 may be one or more busses.
- the CPUs 205 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments.
- a processor 205 may be a digital signal processor (DSP).
- DSP digital signal processor
- One or more processing elements 235 may be stored in the memory 225 .
- a processing element 235 may include one or more stream operators 240 (described below). In one embodiment, a processing element 235 is assigned to be executed by only one CPU 205 , although in other embodiments the stream operators 240 of a processing element 235 may include one or more threads that are executed on two or more CPUs 205 .
- the memory 225 is generally included to be representative of a random access memory, e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or Flash.
- the storage 230 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), or removable memory cards, optical storage, flash memory devices, network attached storage (NAS), or connections to storage area network (SAN) devices, or other devices that may store non-volatile data.
- the network interface 215 is configured to transmit data via the communications network 120 .
- a stream computing application may include one or more stream operators 240 that may be compiled into a “processing element” container 235 . Two or more processing elements 235 may run on the same memory 225 , each processing element having one or more stream operators 240 . Each stream operator 240 may include a portion of code that processes tuples flowing into a processing element and outputs tuples to other stream operators 240 in the same processing element, in other processing elements, or in both the same and other processing elements in a stream computing application. Processing elements 235 may pass tuples to other processing elements that are on the same compute node 110 or on other compute nodes that are accessible via communications network 120 . For example, a processing element 235 on compute node 110 A may output tuples to a processing element 235 on compute node 110 B.
- the storage 230 may include a buffer 260 . Although shown as being in storage, the buffer 260 may be located in the memory 225 of the compute node 110 or in a combination of both memories. Moreover, storage 230 may include storage space that is external to the compute node 110 , such as in a cloud.
- the compute node 110 may include one or more operating systems 262 .
- An operating system 262 may be stored partially in memory 225 and partially in storage 230 .
- an operating system may be stored entirely in memory 225 or entirely in storage 230 .
- the operating system provides an interface between various hardware resources, including the CPU 205 , and processing elements and other components of the stream computing application.
- an operating system provides common services for application programs, such as providing a time function.
- FIG. 3 is a more detailed view of the management system 105 of FIG. 1 according to some embodiments.
- the management system 105 may include, without limitation, one or more processors (CPUs) 305 , a network interface 315 , an interconnect 320 , a memory 325 , and a storage 330 .
- the management system 105 may also include an I/O device interface 310 connecting I/O devices 312 , e.g., keyboard, display, and mouse devices, to the management system 105 .
- I/O device interface 310 connecting I/O devices 312 , e.g., keyboard, display, and mouse devices, to the management system 105 .
- Each CPU 305 retrieves and executes programming instructions stored in the memory 325 or storage 330 . Similarly, each CPU 305 stores and retrieves application data residing in the memory 325 or storage 330 .
- the interconnect 320 is used to move data, such as programming instructions and application data, between the CPU 305 , I/O device interface 310 , storage unit 330 , network interface 315 , and memory 325 .
- the interconnect 320 may be one or more busses.
- the CPUs 305 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In one embodiment, a processor 305 may be a DSP.
- Memory 325 is generally included to be representative of a random access memory, e.g., SRAM, DRAM, or Flash.
- the storage 330 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), removable memory cards, optical storage, Flash memory devices, network attached storage (NAS), connections to storage area-network (SAN) devices, or the cloud.
- the network interface 315 is configured to transmit data via the communications network 120 .
- the memory 325 may store a stream manager 134 . Additionally, the storage 330 may store an operator graph 132 . The operator graph 132 may define how tuples are routed to processing elements 235 ( FIG. 2 ) for processing.
- the management system 105 may include one or more operating systems 332 .
- An operating system 332 may be stored partially in memory 325 and partially in storage 330 .
- an operating system may be stored entirely in memory 325 or entirely in storage 330 .
- the operating system provides an interface between various hardware resources, including the CPU 305 , and processing elements and other components of the stream computing application.
- an operating system provides common services for application programs, such as providing a time function.
- FIG. 4 is a more detailed view of the development system 102 of FIG. 1 according to some embodiments.
- the development system 102 may include, without limitation, one or more processors (CPUs) 405 , a network interface 415 , an interconnect 420 , a memory 425 , and storage 430 .
- the development system 102 may also include an I/O device interface 410 connecting I/O devices 412 , e.g., keyboard, display, and mouse devices, to the development system 102 .
- I/O device interface 410 connecting I/O devices 412 , e.g., keyboard, display, and mouse devices, to the development system 102 .
- Each CPU 405 retrieves and executes programming instructions stored in the memory 425 or storage 430 . Similarly, each CPU 405 stores and retrieves application data residing in the memory 425 or storage 430 .
- the interconnect 420 is used to move data, such as programming instructions and application data, between the CPU 405 , I/O device interface 410 , storage unit 430 , network interface 415 , and memory 425 .
- the interconnect 420 may be one or more busses.
- the CPUs 405 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In one embodiment, a processor 405 may be a DSP.
- Memory 425 is generally included to be representative of a random access memory, e.g., SRAM, DRAM, or Flash.
- the storage 430 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), removable memory cards, optical storage, flash memory devices, network attached storage (NAS), connections to storage area-network (SAN) devices, or to the cloud.
- the network interface 415 is configured to transmit data via the communications network 120 .
- the development system 102 may include one or more operating systems 432 .
- An operating system 432 may be stored partially in memory 425 and partially in storage 430 .
- an operating system may be stored entirely in memory 425 or entirely in storage 430 .
- the operating system provides an interface between various hardware resources, including the CPU 405 , and processing elements and other components of the stream computing application.
- an operating system provides common services for application programs, such as providing a time function.
- the memory 425 may store a compiler 136 .
- the compiler 136 compiles modules, which include source code or statements, into the object code, which includes machine instructions that execute on a processor.
- the compiler 136 may translate the modules into an intermediate form before translating the intermediate form into object code.
- the compiler 136 may output a set of deployable artifacts that may include a set of processing elements and an application description language file (ADL file), which is a configuration file that describes the stream computing application.
- ADL file application description language file
- the compiler 136 may be a just-in-time compiler that executes as part of an interpreter. In other embodiments, the compiler 136 may be an optimizing compiler.
- the compiler 136 may perform peephole optimizations, local optimizations, loop optimizations, inter-procedural or whole-program optimizations, machine code optimizations, or any other optimizations that reduce the amount of time required to execute the object code, to reduce the amount of memory required to execute the object code, or both.
- the output of the compiler 136 may be represented by an operator graph, e.g., the operator graph 132 of FIG. 1 .
- the compiler 136 may also provide the application administrator with the ability to optimize performance through profile-driven fusion optimization. Fusing operators may improve performance by reducing the number of calls to a transport. While fusing stream operators may provide faster communication between operators than is available using inter-process communication techniques, any decision to fuse operators requires balancing the benefits of distributing processing across multiple compute nodes with the benefit of faster inter-operator communications.
- the compiler 136 may automate the fusion process to determine how to best fuse the operators to be hosted by one or more processing elements, while respecting user-specified constraints. This may be a two-step process, including compiling the application in a profiling mode and running the application, then re-compiling and using the optimizer during this subsequent compilation. The end result may, however, be a compiler-supplied deployable application with an optimized application configuration.
- FIG. 5 illustrates an exemplary operator graph 500 for a stream computing application beginning from one or more sources 502 through to one or more sinks 504 , 506 , according to some embodiments.
- This flow from source to sink may also be generally referred to herein as an execution path.
- a flow from one processing element to another may be referred to as an execution path in various contexts.
- FIG. 5 is abstracted to show connected processing elements PE 1 -PE 10
- the operator graph 500 may include data flows between stream operators 240 ( FIG. 2 ) within the same or different processing elements.
- processing elements such as processing element 235 ( FIG.
- an operator graph may be much more complex and may include many individual operator graphs that may be statically or dynamically linked together.
- the example operator graph shown in FIG. 5 includes ten processing elements (labeled as PE 1 -PE 10 ) dispersed over and running on the compute nodes 110 A- 110 D.
- a processing element may include one or more stream operators fused together to form an independently running process with its own process ID (PID) and memory space.
- PID process ID
- inter-process communication may occur using a “transport,” e.g., a network socket, a TCP/IP socket, or shared memory.
- Inter-process communication paths used for inter-process communications can be a critical resource in a stream computing application. However, when stream operators are fused together, the fused stream operators can use more rapid communication techniques for passing tuples among stream operators in each processing element.
- the operator graph 500 begins at a source 502 and ends at a sink 504 , 506 .
- Compute node 110 A includes the processing elements PE 1 , PE 2 , and PE 3 .
- Source 502 flows into the processing element PE 1 , which in turn outputs tuples that are received by PE 2 and PE 3 .
- PE 1 may split data attributes received in a tuple and pass some data attributes in a new tuple to PE 2 , while passing other data attributes in another new tuple to PE 3 .
- PE 1 may pass some received tuples to PE 2 while passing other tuples to PE 3 .
- Tuples that flow to PE 2 are processed by the stream operators contained in PE 2 , and the resulting tuples are then output to PE 4 on compute node 110 B.
- the tuples output by PE 4 flow to PE 6 before being transmitted to a sink 504 .
- tuples flowing from PE 3 to PE 5 also reach PE 6 and are sent to a sink 504 .
- PE 6 could be configured to perform a join operation, combining tuples received from PE 4 and PE 5 , before sending the data to the sink 504 .
- This example operator graph also shows tuples flowing from PE 3 to PE 7 on compute node 110 C, which itself shows tuples flowing to PE 8 and looping back to PE 7 .
- Tuples output from PE 8 flow to PE 9 on compute node 110 D, which in turn outputs tuples to be processed by operators in PE 10 before being sent to a sink 506 .
- the tuple received by a particular processing element 235 is generally not considered to be the same tuple that is output downstream.
- the output tuple is changed in some way.
- An attribute or metadata may be added, deleted, or changed.
- a particular tuple output by a processing element may not be considered to be the same tuple as a corresponding input tuple even if the input tuple is not changed by the processing element.
- Processing elements 235 may be configured to receive or output tuples in various formats, e.g., the processing elements or stream operators could exchange data marked up as XML documents.
- each stream operator 240 within a processing element 235 may be configured to carry out any form of data processing functions on received tuples, including, for example, writing to database tables or performing other database operations such as data joins, splits, reads, etc., as well as performing other data analytic functions or operations.
- the stream manager 134 of FIG. 1 may be configured to monitor a stream computing application running on compute nodes, e.g., compute nodes 110 A- 110 D, as well as to change the deployment of an operator graph, e.g., operator graph 132 .
- the stream manager 134 may move processing elements from one compute node 110 to another, for example, to manage the processing loads of the compute nodes 110 A- 110 D in the computing infrastructure 100 .
- stream manager 134 may control the stream computing application by inserting, removing, fusing, un-fusing, or otherwise modifying the processing elements and stream operators (or what tuples flow to the processing elements) running on the compute nodes 110 A- 110 D.
- a processing element may be a collection of fused stream operators, it is equally correct to describe the operator graph as one or more execution paths between specific stream operators, which may include execution paths to different stream operators within the same processing element.
- FIG. 5 illustrates execution paths between processing elements for the sake of clarity.
- FIG. 6 illustrates a flow diagram for recommending streams application actors as fusion candidates, according to various embodiments.
- the method 600 can start, per 602 , when profiles are generated for each streams application actor, per 604 .
- a streams application actor could be, for example, a streams operator, a processing element, or another processing entity in a streams operator graph.
- Each profile can include data relevant to the particular streams application actor.
- a profile could include historical central processing unit (CPU) usage, current or real-time CPU usage, processing speeds, tuple rates, and other data relevant to the data processing occurring over a particular streams application actor.
- a profile can be created for each streams application actor on a particular operating graph within a streams processing environment.
- a set of fusion candidates may be identified, per 606 .
- three streams operators could have profiles with historical usage data that indicates that, when running on the same hardware configuration as in the past, the operators would be able to more quickly process data (e.g., with fewer delays) if they were fused together in a single processing element. This could be based on a parameter or set of parameters established by an application manager or a system administrator, or in another way.
- a determination that two or more streams application actors (e.g., stream operators) are fusion candidates may also indicate that the streams application actors can be separated from a same processing element (e.g., “un-fused”).
- a fusion candidate recommendation can recommend that the streams application actors be merged or separated.
- the identity of the streams application actors and the fusion candidate recommendation may then be displayed, per 608 .
- the fusion candidate rule satisfied by the streams application actors can also be displayed. For example, identifying data about each of the three streams operators can be displayed.
- the recommendation to fuse the three operators into a single processing element can also be displayed.
- the rule or parameter that was met in order to qualify them as fusion candidates could also be displayed.
- Other data which could be helpful to a developer or other user in determining whether or not the fusion candidate recommendation should be accepted can also be displayed.
- the display can be a part of a user interface that is part of a job execution recommendation of a performance management system.
- an integrated development environment could display the results to a developer, in order to hint to the developer an appropriate configuration or reconfiguration for a particular portion of an operator graph.
- the method can then end, per 610 .
- the system can also continue to monitor for fusion candidates, continue to update the streams application actor profiles, and generate new profiles as streams application actors are added to a graph.
- FIG. 7 illustrates a block diagram of a user interface 700 , according to various embodiments.
- the user interface 700 may be a graphical user interface and may be displayed on one or more I/O devices (e.g., I/O devices 212 in FIG. 2 ), and the user interface 700 may be generated by one or more processors (e.g., processors 205 in FIG. 2 ) executing instructions contained in a stream manager or stream computing application (e.g., stream manager 134 in FIG. 1 or stream computing application 232 in FIG. 2 ).
- processors e.g., processors 205 in FIG. 2
- a stream manager or stream computing application e.g., stream manager 134 in FIG. 1 or stream computing application 232 in FIG. 2 .
- a user presented with the user interface 700 can monitor a stream computing application.
- the user interface 700 can also provide a visual representation of the processes of the various streams computing elements including streams application actors.
- the user interface 700 can also present a user with one or more fusion candidate recommendations and relevant metrics associated with the fusion candidate recommendation.
- the user interface 700 can include, individually or in combination, a graphical display (e.g., 702 and 704 ), a tabulated display (e.g., 718 b ), or any other representation suitable for a user.
- graphical display 702 can provide a graphical overview of a current operator graph 706 that is being developed, monitored, or edited.
- operator graph 706 can comprise a number of compute nodes 708 A, 708 B, 708 C, and 708 D.
- Each compute node can host a number of processing elements (PE 1 - 10 ) that process tuples of data as they flow from a source 710 to a sink 712 or 714 .
- PE 1 - 10 processing elements
- each processing element may be a selectable option, which allows a user to select the processing element in order to see the one or more stream operators or other streams application actors that comprise the processing element.
- the operator graph 706 depicted herein may correspond to the graph illustrated in FIG. 5 .
- Data points or performance metrics for a compute node 708 or each particular processing element may also be included on the operator graph 706 .
- a user presented with the user interface 700 can view particular operations in the operator graph 706 , while still maintaining a perspective of the overall data processing occurring over the entire operator graph 706 .
- the user interface 700 can also include a graphical display 704 that includes a fusion candidate recommendation 718 .
- the graphical display 704 which herein may be referred to as a “fusion recommendation window” can include both the fusion candidate recommendation 718 and a data window 716 , which can include additional data related to the fusion candidate recommendation.
- the fusion recommendation window 716 may include a graphical representation of the two or more streams operators for which fusion or separation is being recommended (e.g., OP 4 and OP 5 ). It may also include text or a graphical depiction which indicates that the operators are to be fused or separated. It may also include tabular data for each of these operators, here 718 a - b . An example of this tabular data 718 b is depicted for illustrative purposes. This tabular data 718 b can include data specific to each particular streams application actor that has been identified as a fusion candidate (e.g. OP 5 ).
- Example non-limiting performance metrics can include: tuple flow rates, processing power demand, location of operator processing, hardware configurations, or other data. Further discussion of the performance metrics that may be included in a tabular display 718 b may be found at FIG. 9 .
- the graphical and tabular displays depicted in FIG. 7 are intended to provide examples only and are not intended to restrict the appearance or values present on the user interface 700 . Rather, the various displays, arrangement of the displays, types of data shown, and other features may be configured to accommodate user needs or preferences.
- FIG. 8 illustrates a graphical display, according to embodiments.
- the graphical display 800 can be a part or the whole of a user interface.
- the user interface may be a part of an IDE for a streams computing application.
- the graphical display 800 may be phase-integrated into a more complex user interface (e.g., graphical display 704 in FIG. 7 ).
- graphical display 800 may be referred to as a fusion recommendation window 800 .
- the fusion recommendation window 800 can include one or more graphical depictions of streams operators 804 a , 804 b , 804 c , or other streams application actors that may be candidates for fusion.
- operators 804 a and 804 b have been identified as candidates for fusion, as described herein.
- streams operator 804 c has not been selected as a candidate for fusion.
- one or more surrounding but not recommended streams operators e.g., 804 c
- These may be displayed based on a setting or based on a threshold value, or in another way. For example, a user may set a threshold that if reached, indicates that the recommended operators are extremely strong candidates for fusion.
- surrounding operators that are not candidates for fusion may not be displayed. However, if the threshold is not reached based on various metrics, then the failure to meet the threshold could indicate that the recommended operators are only moderately strong candidates for fusion. In this case, where the threshold is not reached, one or more surrounding operators (and the performance metrics 806 c of those one or more surrounding operators) may be displayed in the fusion recommendation window 800 . This way, a user (e.g., a developer) can, with relative ease, access and use surrounding performance data in confirming or denying fusion recommendations.
- Each streams operator 804 may be displayed with performance metrics or other relevant data associated with the stream operator. This data may be displayed in one or more tables 806 a - c . Performance metrics can be displayed to assist a user in understanding the reasoning or confidence of a fusion recommendation. FIG. 9 includes more details on this data.
- a graphical representation may indicate to a user that the one or more streams operators 804 a and 804 b are candidates for fusion.
- This recommendation 802 may appear as a selectable option, a graphic, or in other manner.
- the recommendation 802 may include a selectable option 808 .
- This option could link to performance metrics, analysis, or other data that can help the user understand why the recommendation was made.
- the data in selectable option 808 could also indicate performance benefits from fusing or separating the operators, or other data.
- Elements described herein are provided for illustrative purposes only, and a graphical display or fusion recommendation window 800 may be configured to any appearance or design deemed useful.
- the fusion recommendation window 800 may be configured to display fewer performance metrics, and the fusion may occur automatically (i.e., without additional user input) upon, for example, exceeding a particular threshold.
- the system may display the aforementioned data and more, in order to provide the developer with as much useful data as possible, so that the developer can be well-informed to make a decision relatively independent of the system's recommendation.
- FIG. 9 illustrates a tabular display 900 according to embodiments.
- the tabular display 900 may be a table like the example one shown here.
- the table is not limited to the categories given here, and may include performance metrics, analytics, fusion details, or other categories.
- the table shown here indicates performance metrics for a particular streams operator, OP 1 .
- example performance metrics include: tuple flow rates, processing power demand, location of operator processing, hardware configurations of processing, or other data. These metrics may be displayed in a table for each particular operator of interest.
- the performance metrics may be listed as an individual column, including: Performance metric A, Performance metric B, Performance metric C, and Performance metric D. Additional or fewer columns may be included, in order to suit the preferences of a particular user. Another column, here including “1”, “2”, “3”, . . . “current” indicates the particular processing cycle for which the performance metrics were calculated. Additional rows including projected or speculative metrics for future processing cycles could also be included.
- the column may be configured to display between one or all processing cycles, may be selectable in order to see additional data (e.g., hardware configurations) associated with the particular cycle, and may be updated in real-time to reflect the most current processing metrics.
- performance metrics may be listed in another type of visual display, including a graph, chart, or other type of visual display.
- the tabular data updated in real-time and displayed in table 900 can be used by a developer in order to monitor the effects of a fusion or separation, after the fact and in real-time. For example, if a fusion recommendation window (e.g., 800 in FIG. 8 ) contained predictive data that indicated CPU usage would be decreased by x % upon a fusion of three particular operators, the developer could accept the fusion recommendation and monitor the processing consumption for the next y number of cycles to determine whether or not the predication was accurate. In this way, a developer can monitor in real-time the impact and efficacy of the fusion or separation of particular types of operators, and adjust thresholds or other recommendation configurations accordingly.
- a fusion recommendation window e.g., 800 in FIG. 8
- the developer could accept the fusion recommendation and monitor the processing consumption for the next y number of cycles to determine whether or not the predication was accurate.
- a developer can monitor in real-time the impact and efficacy of the fusion or separation of particular types of operators, and adjust thresholds
- FIG. 10 illustrates a method 1000 for recommending fusion candidates in a streams computing environment, according to various embodiments.
- the method 1000 begins when a first streams application actor profile is generated, per 1002 , and a second streams application actor profile is generated, per 1004 .
- These profiles can be generated simultaneously or in any order.
- profiles may be developed and maintained for each streams application actor (e.g., stream operators, processing elements, or other actors) in an operator graph. If two or more profiles satisfy a fusion candidate rule, per 1006 , the streams application actors associated with the profiles may be identified as fusion candidates, per 1008 .
- the fusion candidates' identities could then be displayed as recommended candidates for fusion or separation, along with the recommendation to fuse or separate the actors, per 1010 .
- Other data could be displayed with this recommendation including data regarding the recommendation like detected or evaluated inefficiencies that could be alleviated by the fusion or other data.
- the present invention may be a system, a method, and/or a computer program product.
- the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
- the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
- the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
- a non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
- RAM random access memory
- ROM read-only memory
- EPROM or Flash memory erasable programmable read-only memory
- SRAM static random access memory
- CD-ROM compact disc read-only memory
- DVD digital versatile disk
- memory stick a floppy disk
- a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon
- a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
- Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
- the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
- a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
- Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
- the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
- These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
- the functions noted in the block may occur out of the order noted in the figures.
- two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
- Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service.
- This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.
- On-demand self-service a cloud consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.
- Resource pooling the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
- Rapid elasticity capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.
- Measured service cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service.
- level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts).
- SaaS Software as a Service: the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure.
- the applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail).
- a web browser e.g., web-based e-mail
- the consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.
- PaaS Platform as a Service
- the consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.
- IaaS Infrastructure as a Service
- the consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).
- Private cloud the cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on-premises or off-premises.
- Public cloud the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.
- Hybrid cloud the cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).
- a cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability.
- An infrastructure comprising a network of interconnected nodes.
- Cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, cloud computing node 10 is capable of being implemented and/or performing any of the functionality set forth hereinabove.
- cloud computing node 10 there is a computer system/server 12 , which is operational with numerous other general purpose or special purpose computing system environments or configurations.
- Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with computer system/server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.
- Computer system/server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system.
- program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.
- Computer system/server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network.
- program modules may be located in both local and remote computer system storage media including memory storage devices.
- computer system/server 12 in cloud computing node 10 is shown in the form of a general-purpose computing device.
- the components of computer system/server 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 that couples various system components including system memory 28 to processor 16 .
- Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures.
- bus architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.
- Computer system/server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12 , and it includes both volatile and non-volatile media, removable and non-removable media.
- System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 .
- Computer system/server 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media.
- storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”).
- a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”).
- an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided.
- memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
- Program/utility 40 having a set (at least one) of program modules 42 , may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment.
- Program modules 42 generally carry out the functions and/or methodologies of embodiments of the invention as described herein.
- Computer system/server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24 , etc.; one or more devices that enable a user to interact with computer system/server 12 ; and/or any devices (e.g., network card, modem, etc.) that enable computer system/server 12 to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces 22 . Still yet, computer system/server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter 20 .
- LAN local area network
- WAN wide area network
- public network e.g., the Internet
- network adapter 20 communicates with the other components of computer system/server 12 via bus 18 .
- bus 18 It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server 12 . Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.
- cloud computing environment 50 comprises one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 54 A, desktop computer 54 B, laptop computer 54 C, and/or automobile computer system 54 N may communicate.
- Nodes 10 may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof.
- This allows cloud computing environment 50 to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device.
- computing devices 54 A-N shown in FIG. 12 are intended to be illustrative only and that computing nodes 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).
- FIG. 13 a set of functional abstraction layers provided by cloud computing environment 50 ( FIG. 12 ) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 13 are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:
- Hardware and software layer 60 includes hardware and software components.
- hardware components include: mainframes 61 ; RISC (Reduced Instruction Set Computer) architecture based servers 62 ; servers 63 ; blade servers 64 ; storage devices 65 ; and networks and networking components 66 .
- software components include network application server software 67 and database software 68 .
- Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .
- management layer 80 may provide the functions described below.
- Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment.
- Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may comprise application software licenses.
- Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources.
- User portal 83 provides access to the cloud computing environment for consumers and system administrators.
- Service level management 84 provides cloud computing resource allocation and management such that required service levels are met.
- Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.
- SLA Service Level Agreement
- Workloads layer 90 provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation 91 ; software development and lifecycle management 92 ; virtual classroom education delivery 93 ; data analytics processing 94 ; transaction processing 95 ; and providing fusion hints 96 .
Abstract
Description
- The present disclosure relates to stream computing, and more particularly to the use of an integrated development environment in streams computing.
- Database systems are typically configured to separate the process of storing data from accessing, manipulating, or using data stored in a database. More specifically, database systems use a model in which data is first stored and indexed in a memory before subsequent querying and analysis. In general, database systems may not be well suited for performing real-time processing and analyzing streaming data. In particular, database systems may be unable to store, index, and analyze large amounts of streaming data efficiently or in real time.
- An integrated development environment (IDE) may be a software application that provides facilities to computer programmers for software development. An IDE may be designed to facilitate programmer productivity. An IDE may consist of a source code editor, build automation tools, and a debugger. An IDE may also have intelligent code completion capabilities.
- While database systems that are configured to analyze large amounts of streaming data efficiently in real time may provide a number of benefits, their introduction to the development world is relatively new. Thus, programmers and developers may be unfamiliar with a data streaming environment, which can make it difficult for developers to best design, troubleshoot, and optimize a system. Without an advanced understanding of arrangement and compartmentalization of streams operations in a streams operator graph, many of the benefits of streams applications may be lost.
- Embodiments of the present disclosure may be directed toward a method for recommending streams application actors for fusion, where the streams application actors are dispersed on an operator graph within a streaming environment. A system may generate a streams application actor profile for each of two or more streams application actors. The two or more streams application actors may be determined to be fusion candidates, based on the streams application actors satisfying a fusion candidate rule. Each of the fusion candidate rules may be associated with one or more fusion recommendations. A fusion recommendation associated with the rule may be identified and an identity of each of the two or more streams application actors along with the fusion recommendation may be displayed. Embodiments of the present disclosure may also be directed toward a system or computer program product for recommending streams application actors for fusion.
- Advantages of the invention over other art includes the ability to hint to developers an improvement or improvements for the streams system. Rather than requiring a developer have full understanding of both the streams coding environment as well as the current deficiencies or strengths of a particular streams operator graph, the system can identify fusion candidates and make recommendations to combine or separate the streams application actors according to parameters. In this way, a developer can make modifications and improvements to a system using current and informed metrics.
- In some embodiments, the fusion recommendation may be to fuse the fusion candidates together in the same processing element. In other embodiments, the fusion recommendation may be to separate the two fusion candidates from the same processing element. For example only, some of the data in the streams application actor profile on which the fusion candidates can be identified can include historical central processing unit (CPU) usage, processing time, tuple rates, machine configuration, and current CPU usage. In this way, in some embodiments of the disclosed, the system can analyze historical and real time data like CPU usage and processing time when determining which streams application actors can be identified as fusion candidates. This data or analysis of this data can be provided to a developer so the developer can make informed decisions.
- The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure.
- The drawings included in the present application are incorporated into, and form part of, the specification. They illustrate embodiments of the present disclosure and, along with the description, serve to explain the principles of the disclosure. The drawings are only illustrative of certain embodiments and do not limit the disclosure.
-
FIG. 1 illustrates a computing infrastructure configured to execute a stream computing application, according to various embodiments. -
FIG. 2 illustrates a more detailed view of a compute node ofFIG. 1 , according to various embodiments. -
FIG. 3 illustrates a more detailed view of the management system ofFIG. 1 , according to various embodiments. -
FIG. 4 illustrates a more detailed view of the development system ofFIG. 1 , according to various embodiments. -
FIG. 5 illustrates an operator graph for a stream computing application, according to various embodiments. -
FIG. 6 illustrates a flow diagram for making fusion recommendations for streams application actors, according to various embodiments. -
FIG. 7 illustrates a block diagram of a user interface, according to various embodiments. -
FIG. 8 illustrates a block diagram of a user interface, according to various embodiments. -
FIG. 9 illustrates a tabular display, according to various embodiments. -
FIG. 10 illustrates a method for recommending fusion candidates in a streams computing environment, according to various embodiments. -
FIG. 11 depicts a cloud computing node according to an embodiment of the present invention. -
FIG. 12 depicts a cloud computing environment according to an embodiment of the present invention. -
FIG. 13 depicts abstraction model layers according to an embodiment of the present invention. - While the invention is amenable to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.
- Aspects of the present disclosure relate to stream computing, and in particular, to the use of an integrated development environment in streams computing. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.
- Stream-based computing and stream-based database computing are emerging as a developing technology for database systems. Products are available which allow users to create applications that process and query streaming data before it reaches a database file. With this emerging technology, users can specify processing logic to apply to inbound data records while they are “in flight,” with the results available in a very short amount of time, often in fractions of a second. Constructing an application using this type of processing has opened up a new programming paradigm that will allow for development of a broad variety of innovative applications, systems, and processes, as well as present new challenges for application programmers and database developers.
- In a stream computing application, stream operators are connected to one another such that data flows from one stream operator to the next (e.g., over a TCP/IP socket). When a stream operator receives data, it may perform operations, such as analysis logic, which may change the tuple by adding or subtracting attributes, or updating the values of existing attributes within the tuple. When the analysis logic is complete, a new tuple is then sent to the next stream operator. Scalability is achieved by distributing an application across nodes by creating executables (i.e., processing elements), as well as replicating processing elements on multiple nodes and load balancing among them. Stream operators in a stream computing application can be fused together to form a processing element that is executable. Doing so allows processing elements to share a common process space, resulting in much faster communication between stream operators than is available using inter-process communication techniques (e.g., using a TCP/IP socket). Further, processing elements can be inserted or removed dynamically from an operator graph representing the flow of data through the stream computing application. A particular stream operator may not reside within the same operating system process as other stream operators. In addition, stream operators in the same operator graph may be hosted on different nodes, e.g., on different compute nodes or on different cores of a compute node.
- Data flows from one stream operator to another in the form of a “tuple.” A tuple is a sequence of one or more attributes associated with an entity. Attributes may be any of a variety of different types, e.g., integer, float, Boolean, string, etc. The attributes may be ordered. In addition to attributes associated with an entity, a tuple may include metadata, i.e., data about the tuple. A tuple may be extended by adding one or more additional attributes or metadata to it. As used herein, “stream” or “data stream” refers to a sequence of tuples. Generally, a stream may be considered a pseudo-infinite sequence of tuples.
- Tuples are received and output by stream operators and processing elements. An input tuple corresponding with a particular entity that is received by a stream operator or processing element, however, is generally not considered to be the same tuple that is output by the stream operator or processing element, even if the output tuple corresponds with the same entity or data as the input tuple. An output tuple need not be changed in some way from the input tuple.
- Nonetheless, an output tuple may be changed in some way by a stream operator or processing element. An attribute or metadata may be added, deleted, or modified. For example, a tuple will often have two or more attributes. A stream operator or processing element may receive the tuple having multiple attributes and output a tuple corresponding with the input tuple. The stream operator or processing element may only change one of the attributes so that all of the attributes of the output tuple except one are the same as the attributes of the input tuple.
- Generally, a particular tuple output by a stream operator or processing element may not be considered to be the same tuple as a corresponding input tuple even if the input tuple is not changed by the processing element. However, to simplify the present description and the claims, an output tuple that has the same data attributes or is associated with the same entity as a corresponding input tuple will be referred to herein as the same tuple unless the context or an express statement indicates otherwise.
- Stream computing applications handle massive volumes of data that need to be processed efficiently and in real time. For example, a stream computing application may continuously ingest and analyze hundreds of thousands of messages per second and up to petabytes of data per day. Accordingly, each stream operator in a stream computing application may be required to process a received tuple within fractions of a second. Unless the stream operators are located in the same processing element, it is necessary to use an inter-process communication path each time a tuple is sent from one stream operator to another. Inter-process communication paths can be a critical resource in a stream computing application. According to various embodiments, the available bandwidth on one or more inter-process communication paths may be conserved. Efficient use of inter-process communication bandwidth can speed up processing.
- An operator graph can be an execution path for a plurality of stream operators to process a stream of tuples. In addition to stream operators, the operator graph can refer to an execution path for processing elements and the dependent stream operators of the processing elements to process the stream of tuples. Generally, the operator graph can have a plurality of stream operators that produce a particular end result, e.g., calculate an average.
- In stream based computing, as described herein, computing applications may be distributed over a series of nodes by distributing processing elements that are connected together. Thus, a streams computing environment may deviate from a traditional computing environment. As such, a computer programmer may not understand performance implications of the streams environment, including underutilized CPU usage and distribution of processing tasks over a variety of streams application actors (e.g., operators, processing elements, or others). Processing tasks could also be distributed over a variety of streams operators including functors, joins, or other operators designed for a particular task or tasks.
- In particular, fusion, a process where streams application actors can be joined together within a single actor, may be recommended to a programmer in order to better distribute executables. Alternatively, separation of streams application actors may be recommended to a programmer. These recommendations, referred to herein as fusion recommendations, may be made in a performance management system or on a user interface as part of an integrated development environment.
- These fusion recommendations may recommend two or more streams application actors as fusion candidates. Fusion candidates may be streams application actors (e.g., operators) that are recommended to be fused together or be separated from each other or from a particular configuration (e.g., a same processing element). In this way, a fusion recommendation may indicate or “hint” at a combination or separation of fusion candidates, as is appropriate based on parameters and described herein.
- The streams application actors may be identified as fusion candidates based on profiles associated with the streams application actors. The profiles can include historical runtime data, real-time processing data, tuple flow data, machine configuration, or other data relevant processing time, processing demand, or capacity for each streams application actor. The data included in the profile for each streams application actor could also include data about the types of processes conducted by each streams application actor. The profile data could include other data deemed relevant by a system administrator or user to fusion recommendations.
- Based on the streams application actor profiles, the system can detect a set of two or more streams application actors that is a potential fusion candidate. The two or more candidates may be identified based on one or more fusion candidate rules. These rules can be set by an administrator, determined by an individual programmer, or in another way. For example, a rule could be that if two operators are performing a similar function, and the input and output time of the first operator is a majority of the processing time for the operation, the first and second operators may be identified as the fusion candidate set. Once the set is identified as satisfying a fusion candidate rule, the identity of each of the streams application actors (e.g., operators one and two), along with the recommendation to fuse the two, may be displayed on a user interface. Other data useful to a developer could also be displayed on the user interface, as described herein. This user interface may be a part of an integrated development environment. It could also be a part of another type of performance management display. Thus, a developer could receive indications or “hints” as to how the distribution of process across an operator graph may be improved. The developer can also be provided with real-time and historical data, and analysis of this data, to determine the appropriateness of the fusion recommendation to the particular operator graph.
-
FIG. 1 illustrates oneexemplary computing infrastructure 100 that may be configured to execute a stream computing application, according to some embodiments. Thecomputing infrastructure 100 includes amanagement system 105 and two ormore compute nodes 110A—110D—i.e., hosts—which are communicatively coupled to each other using one ormore communications networks 120. Thecommunications network 120 may include one or more servers, networks, or databases, and may use a particular communication protocol to transfer data between thecompute nodes 110A-110D. Adevelopment system 102 may be communicatively coupled with themanagement system 105 and thecompute nodes 110 either directly or via thecommunications network 120. - The
communications network 120 may include a variety of types of physical communication channels or “links.” The links may be wired, wireless, optical, or any other suitable media. In addition, thecommunications network 120 may include a variety of network hardware and software for performing routing, switching, and other functions, such as routers, switches, or bridges. Thecommunications network 120 may be dedicated for use by a stream computing application or shared with other applications and users. Thecommunications network 120 may be any size. For example, thecommunications network 120 may include a single local area network or a wide area network spanning a large geographical area, such as the Internet. The links may provide different levels of bandwidth or capacity to transfer data at a particular rate. The bandwidth that a particular link provides may vary depending on a variety of factors, including the type of communication media and whether particular network hardware or software is functioning correctly or at full capacity. In addition, the bandwidth that a particular link provides to a stream computing application may vary if the link is shared with other applications and users. The available bandwidth may vary depending on the load placed on the link by the other applications and users. The bandwidth that a particular link provides may also vary depending on a temporal factor, such as time of day, day of week, day of month, or season. -
FIG. 2 is a more detailed view of acompute node 110, which may be the same as one of thecompute nodes 110A-110D ofFIG. 1 , according to various embodiments. Thecompute node 110 may include, without limitation, one or more processors (CPUs) 205, anetwork interface 215, aninterconnect 220, amemory 225, and astorage 230. Thecompute node 110 may also include an I/O device interface 210 used to connect I/O devices 212, e.g., keyboard, display, and mouse devices, to thecompute node 110. - Each
CPU 205 retrieves and executes programming instructions stored in thememory 225 orstorage 230. Similarly, theCPU 205 stores and retrieves application data residing in thememory 225. Theinterconnect 220 is used to transmit programming instructions and application data between eachCPU 205, I/O device interface 210,storage 230,network interface 215, andmemory 225. Theinterconnect 220 may be one or more busses. TheCPUs 205 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In one embodiment, aprocessor 205 may be a digital signal processor (DSP). One or more processing elements 235 (described below) may be stored in thememory 225. Aprocessing element 235 may include one or more stream operators 240 (described below). In one embodiment, aprocessing element 235 is assigned to be executed by only oneCPU 205, although in other embodiments thestream operators 240 of aprocessing element 235 may include one or more threads that are executed on two ormore CPUs 205. Thememory 225 is generally included to be representative of a random access memory, e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or Flash. Thestorage 230 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), or removable memory cards, optical storage, flash memory devices, network attached storage (NAS), or connections to storage area network (SAN) devices, or other devices that may store non-volatile data. Thenetwork interface 215 is configured to transmit data via thecommunications network 120. - A stream computing application may include one or
more stream operators 240 that may be compiled into a “processing element”container 235. Two ormore processing elements 235 may run on thesame memory 225, each processing element having one ormore stream operators 240. Eachstream operator 240 may include a portion of code that processes tuples flowing into a processing element and outputs tuples toother stream operators 240 in the same processing element, in other processing elements, or in both the same and other processing elements in a stream computing application.Processing elements 235 may pass tuples to other processing elements that are on thesame compute node 110 or on other compute nodes that are accessible viacommunications network 120. For example, aprocessing element 235 oncompute node 110A may output tuples to aprocessing element 235 oncompute node 110B. - The
storage 230 may include abuffer 260. Although shown as being in storage, thebuffer 260 may be located in thememory 225 of thecompute node 110 or in a combination of both memories. Moreover,storage 230 may include storage space that is external to thecompute node 110, such as in a cloud. - The
compute node 110 may include one ormore operating systems 262. Anoperating system 262 may be stored partially inmemory 225 and partially instorage 230. Alternatively, an operating system may be stored entirely inmemory 225 or entirely instorage 230. The operating system provides an interface between various hardware resources, including theCPU 205, and processing elements and other components of the stream computing application. In addition, an operating system provides common services for application programs, such as providing a time function. -
FIG. 3 is a more detailed view of themanagement system 105 ofFIG. 1 according to some embodiments. Themanagement system 105 may include, without limitation, one or more processors (CPUs) 305, anetwork interface 315, aninterconnect 320, amemory 325, and astorage 330. Themanagement system 105 may also include an I/O device interface 310 connecting I/O devices 312, e.g., keyboard, display, and mouse devices, to themanagement system 105. - Each
CPU 305 retrieves and executes programming instructions stored in thememory 325 orstorage 330. Similarly, eachCPU 305 stores and retrieves application data residing in thememory 325 orstorage 330. Theinterconnect 320 is used to move data, such as programming instructions and application data, between theCPU 305, I/O device interface 310,storage unit 330,network interface 315, andmemory 325. Theinterconnect 320 may be one or more busses. TheCPUs 305 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In one embodiment, aprocessor 305 may be a DSP.Memory 325 is generally included to be representative of a random access memory, e.g., SRAM, DRAM, or Flash. Thestorage 330 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), removable memory cards, optical storage, Flash memory devices, network attached storage (NAS), connections to storage area-network (SAN) devices, or the cloud. Thenetwork interface 315 is configured to transmit data via thecommunications network 120. - The
memory 325 may store astream manager 134. Additionally, thestorage 330 may store anoperator graph 132. Theoperator graph 132 may define how tuples are routed to processing elements 235 (FIG. 2 ) for processing. - The
management system 105 may include one ormore operating systems 332. Anoperating system 332 may be stored partially inmemory 325 and partially instorage 330. Alternatively, an operating system may be stored entirely inmemory 325 or entirely instorage 330. The operating system provides an interface between various hardware resources, including theCPU 305, and processing elements and other components of the stream computing application. In addition, an operating system provides common services for application programs, such as providing a time function. -
FIG. 4 is a more detailed view of thedevelopment system 102 ofFIG. 1 according to some embodiments. Thedevelopment system 102 may include, without limitation, one or more processors (CPUs) 405, anetwork interface 415, aninterconnect 420, amemory 425, andstorage 430. Thedevelopment system 102 may also include an I/O device interface 410 connecting I/O devices 412, e.g., keyboard, display, and mouse devices, to thedevelopment system 102. - Each
CPU 405 retrieves and executes programming instructions stored in thememory 425 orstorage 430. Similarly, eachCPU 405 stores and retrieves application data residing in thememory 425 orstorage 430. Theinterconnect 420 is used to move data, such as programming instructions and application data, between theCPU 405, I/O device interface 410,storage unit 430,network interface 415, andmemory 425. Theinterconnect 420 may be one or more busses. TheCPUs 405 may be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In one embodiment, aprocessor 405 may be a DSP.Memory 425 is generally included to be representative of a random access memory, e.g., SRAM, DRAM, or Flash. Thestorage 430 is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), removable memory cards, optical storage, flash memory devices, network attached storage (NAS), connections to storage area-network (SAN) devices, or to the cloud. Thenetwork interface 415 is configured to transmit data via thecommunications network 120. - The
development system 102 may include one ormore operating systems 432. Anoperating system 432 may be stored partially inmemory 425 and partially instorage 430. Alternatively, an operating system may be stored entirely inmemory 425 or entirely instorage 430. The operating system provides an interface between various hardware resources, including theCPU 405, and processing elements and other components of the stream computing application. In addition, an operating system provides common services for application programs, such as providing a time function. - The
memory 425 may store acompiler 136. Thecompiler 136 compiles modules, which include source code or statements, into the object code, which includes machine instructions that execute on a processor. In one embodiment, thecompiler 136 may translate the modules into an intermediate form before translating the intermediate form into object code. Thecompiler 136 may output a set of deployable artifacts that may include a set of processing elements and an application description language file (ADL file), which is a configuration file that describes the stream computing application. In some embodiments, thecompiler 136 may be a just-in-time compiler that executes as part of an interpreter. In other embodiments, thecompiler 136 may be an optimizing compiler. In various embodiments, thecompiler 136 may perform peephole optimizations, local optimizations, loop optimizations, inter-procedural or whole-program optimizations, machine code optimizations, or any other optimizations that reduce the amount of time required to execute the object code, to reduce the amount of memory required to execute the object code, or both. The output of thecompiler 136 may be represented by an operator graph, e.g., theoperator graph 132 ofFIG. 1 . - The
compiler 136 may also provide the application administrator with the ability to optimize performance through profile-driven fusion optimization. Fusing operators may improve performance by reducing the number of calls to a transport. While fusing stream operators may provide faster communication between operators than is available using inter-process communication techniques, any decision to fuse operators requires balancing the benefits of distributing processing across multiple compute nodes with the benefit of faster inter-operator communications. Thecompiler 136 may automate the fusion process to determine how to best fuse the operators to be hosted by one or more processing elements, while respecting user-specified constraints. This may be a two-step process, including compiling the application in a profiling mode and running the application, then re-compiling and using the optimizer during this subsequent compilation. The end result may, however, be a compiler-supplied deployable application with an optimized application configuration. -
FIG. 5 illustrates anexemplary operator graph 500 for a stream computing application beginning from one ormore sources 502 through to one ormore sinks FIG. 5 is abstracted to show connected processing elements PE1-PE10, theoperator graph 500 may include data flows between stream operators 240 (FIG. 2 ) within the same or different processing elements. Typically, processing elements, such as processing element 235 (FIG. 2 ), receive tuples from the stream as well as output tuples into the stream (except for a sink—where the stream terminates, or a source—where the stream begins). While theoperator graph 500 includes a relatively small number of components, an operator graph may be much more complex and may include many individual operator graphs that may be statically or dynamically linked together. - The example operator graph shown in
FIG. 5 includes ten processing elements (labeled as PE1-PE10) dispersed over and running on thecompute nodes 110A-110D. A processing element may include one or more stream operators fused together to form an independently running process with its own process ID (PID) and memory space. In cases where two (or more) processing elements are running independently, inter-process communication may occur using a “transport,” e.g., a network socket, a TCP/IP socket, or shared memory. Inter-process communication paths used for inter-process communications can be a critical resource in a stream computing application. However, when stream operators are fused together, the fused stream operators can use more rapid communication techniques for passing tuples among stream operators in each processing element. - The
operator graph 500 begins at asource 502 and ends at asink Compute node 110A includes the processing elements PE1, PE2, and PE3.Source 502 flows into the processing element PE1, which in turn outputs tuples that are received by PE2 and PE3. For example, PE1 may split data attributes received in a tuple and pass some data attributes in a new tuple to PE2, while passing other data attributes in another new tuple to PE3. As a second example, PE1 may pass some received tuples to PE2 while passing other tuples to PE3. Tuples that flow to PE2 are processed by the stream operators contained in PE2, and the resulting tuples are then output to PE4 oncompute node 110B. Likewise, the tuples output by PE4 flow to PE6 before being transmitted to asink 504. Similarly, tuples flowing from PE3 to PE5 also reach PE6 and are sent to asink 504. Thus, PE6 could be configured to perform a join operation, combining tuples received from PE4 and PE5, before sending the data to thesink 504. This example operator graph also shows tuples flowing from PE3 to PE7 oncompute node 110C, which itself shows tuples flowing to PE8 and looping back to PE7. Tuples output from PE8 flow to PE9 oncompute node 110D, which in turn outputs tuples to be processed by operators in PE10 before being sent to asink 506. - The tuple received by a particular processing element 235 (
FIG. 2 ) is generally not considered to be the same tuple that is output downstream. Typically, the output tuple is changed in some way. An attribute or metadata may be added, deleted, or changed. However, it is not required that the output tuple be changed in some way. Generally, a particular tuple output by a processing element may not be considered to be the same tuple as a corresponding input tuple even if the input tuple is not changed by the processing element. - Processing elements 235 (
FIG. 2 ) may be configured to receive or output tuples in various formats, e.g., the processing elements or stream operators could exchange data marked up as XML documents. Furthermore, eachstream operator 240 within aprocessing element 235 may be configured to carry out any form of data processing functions on received tuples, including, for example, writing to database tables or performing other database operations such as data joins, splits, reads, etc., as well as performing other data analytic functions or operations. - The
stream manager 134 ofFIG. 1 may be configured to monitor a stream computing application running on compute nodes, e.g., computenodes 110A-110D, as well as to change the deployment of an operator graph, e.g.,operator graph 132. Thestream manager 134 may move processing elements from onecompute node 110 to another, for example, to manage the processing loads of thecompute nodes 110A-110D in thecomputing infrastructure 100. Further,stream manager 134 may control the stream computing application by inserting, removing, fusing, un-fusing, or otherwise modifying the processing elements and stream operators (or what tuples flow to the processing elements) running on thecompute nodes 110A-110D. - Because a processing element may be a collection of fused stream operators, it is equally correct to describe the operator graph as one or more execution paths between specific stream operators, which may include execution paths to different stream operators within the same processing element.
FIG. 5 illustrates execution paths between processing elements for the sake of clarity. -
FIG. 6 illustrates a flow diagram for recommending streams application actors as fusion candidates, according to various embodiments. Themethod 600 can start, per 602, when profiles are generated for each streams application actor, per 604. A streams application actor could be, for example, a streams operator, a processing element, or another processing entity in a streams operator graph. Each profile can include data relevant to the particular streams application actor. For example, a profile could include historical central processing unit (CPU) usage, current or real-time CPU usage, processing speeds, tuple rates, and other data relevant to the data processing occurring over a particular streams application actor. A profile can be created for each streams application actor on a particular operating graph within a streams processing environment. - From the profiles, a set of fusion candidates may be identified, per 606. For example, three streams operators could have profiles with historical usage data that indicates that, when running on the same hardware configuration as in the past, the operators would be able to more quickly process data (e.g., with fewer delays) if they were fused together in a single processing element. This could be based on a parameter or set of parameters established by an application manager or a system administrator, or in another way. A determination that two or more streams application actors (e.g., stream operators) are fusion candidates may also indicate that the streams application actors can be separated from a same processing element (e.g., “un-fused”). Thus, a fusion candidate recommendation can recommend that the streams application actors be merged or separated.
- The identity of the streams application actors and the fusion candidate recommendation may then be displayed, per 608. The fusion candidate rule satisfied by the streams application actors can also be displayed. For example, identifying data about each of the three streams operators can be displayed. The recommendation to fuse the three operators into a single processing element can also be displayed. The rule or parameter that was met in order to qualify them as fusion candidates could also be displayed. Other data which could be helpful to a developer or other user in determining whether or not the fusion candidate recommendation should be accepted can also be displayed. As indicated herein, the display can be a part of a user interface that is part of a job execution recommendation of a performance management system. Or, for example, an integrated development environment (IDE) could display the results to a developer, in order to hint to the developer an appropriate configuration or reconfiguration for a particular portion of an operator graph. The method can then end, per 610. The system can also continue to monitor for fusion candidates, continue to update the streams application actor profiles, and generate new profiles as streams application actors are added to a graph.
-
FIG. 7 illustrates a block diagram of auser interface 700, according to various embodiments. Theuser interface 700 may be a graphical user interface and may be displayed on one or more I/O devices (e.g., I/O devices 212 inFIG. 2 ), and theuser interface 700 may be generated by one or more processors (e.g.,processors 205 inFIG. 2 ) executing instructions contained in a stream manager or stream computing application (e.g.,stream manager 134 inFIG. 1 or stream computing application 232 inFIG. 2 ). - A user presented with the
user interface 700 can monitor a stream computing application. Theuser interface 700 can also provide a visual representation of the processes of the various streams computing elements including streams application actors. Theuser interface 700 can also present a user with one or more fusion candidate recommendations and relevant metrics associated with the fusion candidate recommendation. - The
user interface 700 can include, individually or in combination, a graphical display (e.g., 702 and 704), a tabulated display (e.g., 718 b), or any other representation suitable for a user. For example,graphical display 702 can provide a graphical overview of acurrent operator graph 706 that is being developed, monitored, or edited. As described herein, within a streams computingenvironment operator graph 706 can comprise a number ofcompute nodes source 710 to asink graphical display 702 of theoperator graph 706 are individual stream operators, which may comprise one or more processing element (PE1-10). In embodiments, each processing element (e.g., PE1), may be a selectable option, which allows a user to select the processing element in order to see the one or more stream operators or other streams application actors that comprise the processing element. - The
operator graph 706 depicted herein may correspond to the graph illustrated inFIG. 5 . Data points or performance metrics for a compute node 708 or each particular processing element may also be included on theoperator graph 706. In this way, a user presented with theuser interface 700 can view particular operations in theoperator graph 706, while still maintaining a perspective of the overall data processing occurring over theentire operator graph 706. - The
user interface 700 can also include agraphical display 704 that includes a fusion candidate recommendation 718. Thegraphical display 704, which herein may be referred to as a “fusion recommendation window” can include both the fusion candidate recommendation 718 and adata window 716, which can include additional data related to the fusion candidate recommendation. - For example, the
fusion recommendation window 716 may include a graphical representation of the two or more streams operators for which fusion or separation is being recommended (e.g., OP4 and OP5). It may also include text or a graphical depiction which indicates that the operators are to be fused or separated. It may also include tabular data for each of these operators, here 718 a-b. An example of thistabular data 718 b is depicted for illustrative purposes. Thistabular data 718 b can include data specific to each particular streams application actor that has been identified as a fusion candidate (e.g. OP 5). Example non-limiting performance metrics can include: tuple flow rates, processing power demand, location of operator processing, hardware configurations, or other data. Further discussion of the performance metrics that may be included in atabular display 718 b may be found atFIG. 9 . - The graphical and tabular displays depicted in
FIG. 7 are intended to provide examples only and are not intended to restrict the appearance or values present on theuser interface 700. Rather, the various displays, arrangement of the displays, types of data shown, and other features may be configured to accommodate user needs or preferences. -
FIG. 8 illustrates a graphical display, according to embodiments. Thegraphical display 800 can be a part or the whole of a user interface. The user interface may be a part of an IDE for a streams computing application. Thegraphical display 800 may be phase-integrated into a more complex user interface (e.g.,graphical display 704 inFIG. 7 ). Like its counterpartgraphical display 704 inFIG. 7 ,graphical display 800 may be referred to as afusion recommendation window 800. - For example, the
fusion recommendation window 800 can include one or more graphical depictions ofstreams operators operators operator 804 c has not been selected as a candidate for fusion. However, one or more surrounding but not recommended streams operators (e.g., 804 c) may be included in thefusion recommendation window 800. These may be displayed based on a setting or based on a threshold value, or in another way. For example, a user may set a threshold that if reached, indicates that the recommended operators are extremely strong candidates for fusion. If that threshold is reached, then surrounding operators that are not candidates for fusion (here 804 c) may not be displayed. However, if the threshold is not reached based on various metrics, then the failure to meet the threshold could indicate that the recommended operators are only moderately strong candidates for fusion. In this case, where the threshold is not reached, one or more surrounding operators (and theperformance metrics 806 c of those one or more surrounding operators) may be displayed in thefusion recommendation window 800. This way, a user (e.g., a developer) can, with relative ease, access and use surrounding performance data in confirming or denying fusion recommendations. - Each streams operator 804 (or other streams application actor) may be displayed with performance metrics or other relevant data associated with the stream operator. This data may be displayed in one or more tables 806 a-c. Performance metrics can be displayed to assist a user in understanding the reasoning or confidence of a fusion recommendation.
FIG. 9 includes more details on this data. - A graphical representation may indicate to a user that the one or
more streams operators recommendation 802 may appear as a selectable option, a graphic, or in other manner. For example, therecommendation 802 may include aselectable option 808. This option could link to performance metrics, analysis, or other data that can help the user understand why the recommendation was made. The data inselectable option 808 could also indicate performance benefits from fusing or separating the operators, or other data. Elements described herein are provided for illustrative purposes only, and a graphical display orfusion recommendation window 800 may be configured to any appearance or design deemed useful. For example, if the IDE in which thefusion recommendation window 800 is contained is for a streams computing environment that is very complex and developed with little human interaction, thefusion recommendation window 800 may be configured to display fewer performance metrics, and the fusion may occur automatically (i.e., without additional user input) upon, for example, exceeding a particular threshold. However, in an IDE for a streams computing application that is less complex or where more precision or severability may be desired, the system may display the aforementioned data and more, in order to provide the developer with as much useful data as possible, so that the developer can be well-informed to make a decision relatively independent of the system's recommendation. -
FIG. 9 illustrates atabular display 900 according to embodiments. Thetabular display 900 may be a table like the example one shown here. The table is not limited to the categories given here, and may include performance metrics, analytics, fusion details, or other categories. The table shown here indicates performance metrics for a particular streams operator, OP1. As described herein, example performance metrics include: tuple flow rates, processing power demand, location of operator processing, hardware configurations of processing, or other data. These metrics may be displayed in a table for each particular operator of interest. - For example and as illustrated here, the performance metrics may be listed as an individual column, including: Performance metric A, Performance metric B, Performance metric C, and Performance metric D. Additional or fewer columns may be included, in order to suit the preferences of a particular user. Another column, here including “1”, “2”, “3”, . . . “current” indicates the particular processing cycle for which the performance metrics were calculated. Additional rows including projected or speculative metrics for future processing cycles could also be included. The column may be configured to display between one or all processing cycles, may be selectable in order to see additional data (e.g., hardware configurations) associated with the particular cycle, and may be updated in real-time to reflect the most current processing metrics. In this way, historical data can be observed and used by a user in analysis of a fusion recommendation. The table 900 could also be used in making other determinations. In other embodiments, performance metrics may be listed in another type of visual display, including a graph, chart, or other type of visual display.
- Additionally, the tabular data updated in real-time and displayed in table 900 can be used by a developer in order to monitor the effects of a fusion or separation, after the fact and in real-time. For example, if a fusion recommendation window (e.g., 800 in
FIG. 8 ) contained predictive data that indicated CPU usage would be decreased by x % upon a fusion of three particular operators, the developer could accept the fusion recommendation and monitor the processing consumption for the next y number of cycles to determine whether or not the predication was accurate. In this way, a developer can monitor in real-time the impact and efficacy of the fusion or separation of particular types of operators, and adjust thresholds or other recommendation configurations accordingly. -
FIG. 10 illustrates amethod 1000 for recommending fusion candidates in a streams computing environment, according to various embodiments. Themethod 1000 begins when a first streams application actor profile is generated, per 1002, and a second streams application actor profile is generated, per 1004. These profiles can be generated simultaneously or in any order. For ease of discussion, only two streams application actors, and only two streams application actor profiles are being described, however, profiles may be developed and maintained for each streams application actor (e.g., stream operators, processing elements, or other actors) in an operator graph. If two or more profiles satisfy a fusion candidate rule, per 1006, the streams application actors associated with the profiles may be identified as fusion candidates, per 1008. The fusion candidates' identities could then be displayed as recommended candidates for fusion or separation, along with the recommendation to fuse or separate the actors, per 1010. Other data could be displayed with this recommendation including data regarding the recommendation like detected or evaluated inefficiencies that could be alleviated by the fusion or other data. - The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
- The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
- Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
- Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
- Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
- These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
- The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
- The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
- It is understood in advance that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.
- Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.
- Characteristics are as follows:
- On-demand self-service: a cloud consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.
- Broad network access: capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).
- Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
- Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.
- Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service.
- Service Models are as follows:
- Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.
- Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.
- Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).
- Deployment Models are as follows:
- Private cloud: the cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on-premises or off-premises.
- Community cloud: the cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may exist on-premises or off-premises.
- Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.
- Hybrid cloud: the cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).
- A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.
- Referring now to
FIG. 11 , a schematic of an example of a cloud computing node is shown.Cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless,cloud computing node 10 is capable of being implemented and/or performing any of the functionality set forth hereinabove. - In
cloud computing node 10 there is a computer system/server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with computer system/server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like. - Computer system/
server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system/server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices. - As shown in
FIG. 11 , computer system/server 12 incloud computing node 10 is shown in the form of a general-purpose computing device. The components of computer system/server 12 may include, but are not limited to, one or more processors orprocessing units 16, asystem memory 28, and abus 18 that couples various system components includingsystem memory 28 toprocessor 16. -
Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus. - Computer system/
server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12, and it includes both volatile and non-volatile media, removable and non-removable media. -
System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/orcache memory 32. Computer system/server 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only,storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected tobus 18 by one or more data media interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention. - Program/
utility 40, having a set (at least one) ofprogram modules 42, may be stored inmemory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment.Program modules 42 generally carry out the functions and/or methodologies of embodiments of the invention as described herein. - Computer system/
server 12 may also communicate with one or moreexternal devices 14 such as a keyboard, a pointing device, adisplay 24, etc.; one or more devices that enable a user to interact with computer system/server 12; and/or any devices (e.g., network card, modem, etc.) that enable computer system/server 12 to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces 22. Still yet, computer system/server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) vianetwork adapter 20. As depicted,network adapter 20 communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. - Referring now to
FIG. 12 , illustrativecloud computing environment 50 is depicted. As shown,cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) orcellular telephone 54A, desktop computer 54B,laptop computer 54C, and/or automobile computer system 54N may communicate.Nodes 10 may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allowscloud computing environment 50 to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types ofcomputing devices 54A-N shown inFIG. 12 are intended to be illustrative only and thatcomputing nodes 10 andcloud computing environment 50 can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser). - Referring now to
FIG. 13 , a set of functional abstraction layers provided by cloud computing environment 50 (FIG. 12 ) is shown. It should be understood in advance that the components, layers, and functions shown inFIG. 13 are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided: - Hardware and
software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68. - Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.
- In one example,
management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering andPricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may comprise application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment for consumers and system administrators.Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA. - Workloads layer 90 provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation 91; software development and
lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and providing fusion hints 96. - The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (11)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/868,679 US9569178B1 (en) | 2015-08-28 | 2015-09-29 | Fusion recommendation for performance management in streams |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/838,585 US9582250B1 (en) | 2015-08-28 | 2015-08-28 | Fusion recommendation for performance management in streams |
US14/868,679 US9569178B1 (en) | 2015-08-28 | 2015-09-29 | Fusion recommendation for performance management in streams |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/838,585 Continuation US9582250B1 (en) | 2015-08-28 | 2015-08-28 | Fusion recommendation for performance management in streams |
Publications (2)
Publication Number | Publication Date |
---|---|
US9569178B1 US9569178B1 (en) | 2017-02-14 |
US20170060538A1 true US20170060538A1 (en) | 2017-03-02 |
Family
ID=57964244
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/838,585 Expired - Fee Related US9582250B1 (en) | 2015-08-28 | 2015-08-28 | Fusion recommendation for performance management in streams |
US14/868,679 Expired - Fee Related US9569178B1 (en) | 2015-08-28 | 2015-09-29 | Fusion recommendation for performance management in streams |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/838,585 Expired - Fee Related US9582250B1 (en) | 2015-08-28 | 2015-08-28 | Fusion recommendation for performance management in streams |
Country Status (1)
Country | Link |
---|---|
US (2) | US9582250B1 (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108073437A (en) * | 2017-12-20 | 2018-05-25 | 维沃移动通信有限公司 | Method and mobile terminal are recommended in a kind of application |
US10044569B2 (en) | 2016-09-22 | 2018-08-07 | International Business Machines Corporation | Operator fusion management in a stream computing environment |
US20190317822A1 (en) * | 2018-04-16 | 2019-10-17 | International Business Machines Corporation | Fusing and unfusing operators during a job overlay |
US20220137940A1 (en) * | 2020-10-30 | 2022-05-05 | International Business Machines Corporation | Using fusion to reduce encryption in streaming applications |
WO2023055363A1 (en) * | 2021-09-30 | 2023-04-06 | 17Live Japan Inc. | System, method and computer-readable medium for rendering a streaming |
US11870828B2 (en) | 2021-09-30 | 2024-01-09 | 17Live Japan Inc. | System, method and computer-readable medium for rendering a streaming |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10977017B2 (en) | 2016-04-23 | 2021-04-13 | International Business Machines Corporation | Warning data management for distributed application development |
US10831637B2 (en) * | 2016-04-23 | 2020-11-10 | International Business Machines Corporation | Warning data management with respect to an execution phase |
US10523724B2 (en) * | 2016-05-10 | 2019-12-31 | International Business Machines Corporation | Dynamic stream operator fission and fusion with platform management hints |
US20180321855A1 (en) | 2017-05-03 | 2018-11-08 | Samsung Electronics Co., Ltd. | Multistreaming in heterogeneous environments |
CN107562870A (en) * | 2017-08-30 | 2018-01-09 | 国信优易数据有限公司 | A kind of user recommends method and apparatus |
US11102258B2 (en) | 2018-07-10 | 2021-08-24 | International Business Machines Corporation | Stream processing without central transportation planning |
CN115167846B (en) * | 2022-09-07 | 2022-12-20 | 浙江大华技术股份有限公司 | Recommendation method of downstream operator, electronic device and computer-readable storage medium |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8769485B2 (en) * | 2006-12-04 | 2014-07-01 | Tibco Software, Inc. | Data parallelism and parallel operations in stream processing |
US8566793B2 (en) | 2007-04-27 | 2013-10-22 | International Business Machines Corporation | Detecting and displaying errors in database statements within integrated development environment tool |
US8201138B2 (en) | 2008-01-18 | 2012-06-12 | International Business Machines Corporation | Solution providing navigation-independent access to elements of a software integrated development environment (IDE) using uniform resource locators(URLs) |
US8601458B2 (en) * | 2009-05-14 | 2013-12-03 | International Business Machines Corporation | Profile-driven data stream processing |
US8490072B2 (en) * | 2009-06-23 | 2013-07-16 | International Business Machines Corporation | Partitioning operator flow graphs |
US9384005B2 (en) | 2009-08-27 | 2016-07-05 | Adobe Systems Incorporated | Systems and methods for dynamic configuration of client-side development environments through use of application servers |
US8806510B2 (en) * | 2009-09-10 | 2014-08-12 | International Business Machines Corporation | Determining operator partitioning constraint feasibility |
US9218566B2 (en) * | 2010-08-20 | 2015-12-22 | International Business Machines Corporation | Detecting disallowed combinations of data within a processing element |
US9514159B2 (en) * | 2010-10-27 | 2016-12-06 | International Business Machines Corporation | Database insertions in a stream database environment |
US8776010B2 (en) | 2010-10-27 | 2014-07-08 | Microsoft Corporation | Data type provider for a data store |
US9619211B2 (en) | 2010-12-30 | 2017-04-11 | International Business Machines Corporation | Code suggestion in a software development tool |
US8856764B2 (en) * | 2011-01-25 | 2014-10-07 | International Business Machines Corporation | Distributed static analysis of computer software applications |
US8869097B2 (en) * | 2011-03-23 | 2014-10-21 | Infosys Limited | Online integrated development environment with code assist |
US20120311562A1 (en) * | 2011-06-01 | 2012-12-06 | Yanlin Wang | Extendable event processing |
US9009007B2 (en) * | 2012-03-13 | 2015-04-14 | International Business Machines Corporation | Simulating stream computing systems |
US20130263090A1 (en) | 2012-03-30 | 2013-10-03 | Sony Online Entertainment Llc | System and method for automated testing |
CA2773721C (en) * | 2012-04-05 | 2019-03-05 | Ibm Canada Limited - Ibm Canada Limitee | Code validation using content assist |
US20140075364A1 (en) | 2012-09-13 | 2014-03-13 | Microsoft Corporation | Capturing Activity History Stream |
US8793653B2 (en) | 2012-12-06 | 2014-07-29 | International Business Machines Corporation | Program code library selection in an integrated development environment |
US9269076B2 (en) * | 2013-03-15 | 2016-02-23 | International Business Machines Corporation | Techniques to facilitate collaborative social coding |
-
2015
- 2015-08-28 US US14/838,585 patent/US9582250B1/en not_active Expired - Fee Related
- 2015-09-29 US US14/868,679 patent/US9569178B1/en not_active Expired - Fee Related
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10044569B2 (en) | 2016-09-22 | 2018-08-07 | International Business Machines Corporation | Operator fusion management in a stream computing environment |
US10341189B2 (en) | 2016-09-22 | 2019-07-02 | International Business Machines Corporation | Operator fusion management in a stream computing environment |
US10623269B2 (en) | 2016-09-22 | 2020-04-14 | International Business Machines Corporation | Operator fusion management in a stream computing environment |
CN108073437A (en) * | 2017-12-20 | 2018-05-25 | 维沃移动通信有限公司 | Method and mobile terminal are recommended in a kind of application |
US20190317822A1 (en) * | 2018-04-16 | 2019-10-17 | International Business Machines Corporation | Fusing and unfusing operators during a job overlay |
US20190317823A1 (en) * | 2018-04-16 | 2019-10-17 | International Business Machines Corporation | Fusing and unfusing operators during a job overlay |
US10977089B2 (en) * | 2018-04-16 | 2021-04-13 | International Business Machines Corporation | Fusing and unfusing operators during a job overlay |
US11010203B2 (en) * | 2018-04-16 | 2021-05-18 | International Business Machines Corporation | Fusing and unfusing operators during a job overlay |
US20220137940A1 (en) * | 2020-10-30 | 2022-05-05 | International Business Machines Corporation | Using fusion to reduce encryption in streaming applications |
US11675573B2 (en) * | 2020-10-30 | 2023-06-13 | International Business Machines Corporation | Using fusion to reduce encryption in streaming applications |
WO2023055363A1 (en) * | 2021-09-30 | 2023-04-06 | 17Live Japan Inc. | System, method and computer-readable medium for rendering a streaming |
US11870828B2 (en) | 2021-09-30 | 2024-01-09 | 17Live Japan Inc. | System, method and computer-readable medium for rendering a streaming |
Also Published As
Publication number | Publication date |
---|---|
US9569178B1 (en) | 2017-02-14 |
US9582250B1 (en) | 2017-02-28 |
US20170060536A1 (en) | 2017-03-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9569178B1 (en) | Fusion recommendation for performance management in streams | |
US9535734B2 (en) | Managing stream components based on virtual machine performance adjustments | |
US9971810B2 (en) | Multifusion of a stream operator in a streaming application | |
US20170063724A1 (en) | Asset arrangement management for a shared pool of configurable computing resources associated with a streaming application | |
US10545931B2 (en) | Removing sets of tuples in a streaming environment | |
US10042736B2 (en) | Breakpoint for predicted tuple processing time in a streaming environment | |
US9619518B2 (en) | Tracking tuples to reduce redundancy in a graph | |
US20150281313A1 (en) | Deploying a portion of a streaming application to one or more virtual machines | |
US9253070B2 (en) | Predicted outputs in a streaming environment | |
US10812406B2 (en) | Assigning processing elements to stream computing servers | |
US9996585B2 (en) | Supplementing a tuple with a derived value |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BARSNESS, ERIC L.;BRANSON, MICHAEL J.;SANTOSUOSSO, JOHN M.;REEL/FRAME:036679/0024 Effective date: 20150820 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20210214 |