US10055105B2

US10055105B2 - Method and system for enabling interaction with a plurality of applications using a single user interface

Info

Publication number: US10055105B2
Application number: US14/872,757
Authority: US
Inventors: Colin John Holmes; Monroe Milas Thomas; Glen Lehmann; Pierre Joseph Lemire
Original assignee: Calgary Scientific Inc
Current assignee: Calgary Scientific Inc
Priority date: 2009-02-03
Filing date: 2015-10-01
Publication date: 2018-08-21
Also published as: US20160054897A1

Abstract

An example method for providing simultaneous interaction with a plurality of application programs is described herein. The method includes providing a remote access framework for enabling a client to remotely access the application programs over a network, receiving a communication from the client through the remote access framework, and maintaining a state model within the remote access framework. The state model includes respective application states of the application programs. The method also includes, using the state model, linking interactive functionalities of the application programs, and generating display data indicative of a single user interface including the plurality of application programs. The method further includes communicating the display data to the client through the remote access framework.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent application Ser. No. 12/658,080, filed on Feb. 2, 2010, and entitled “METHOD AND SYSTEM FOR ENABLING INTERACTION WITH A PLURALITY OF APPLICATIONS USING A SINGLE USER INTERFACE,” which claims the benefit of U.S. provisional patent application No. 61/202,168, filed on Feb. 3, 2009, and entitled “METHOD AND SYSTEM FOR ENABLING INTERACTION WITH A PLURALITY OF APPLICATIONS USING A SINGLE USER INTERFACE,” the disclosures of which are expressly incorporated herein by reference in their entireties. This application is also a continuation-in-part of U.S. patent application Ser. No. 13/586,007, filed on Aug. 15, 2012, and entitled “NON-INVASIVE REMOTE ACCESS TO AN APPLICATION PROGRAM,” which claims the benefit of U.S. provisional patent application Nos. 61/523,632 and 61/523,644, each filed on Aug. 15, 2011, and each entitled “NON-INVASIVE REMOTE ACCESS TO AN APPLICATION PROGRAM,” the disclosures of which are expressly incorporated herein by reference in their entireties.

BACKGROUND

Techniques exist for the virtualization of single applications to allow users to interact with application programs while using a client computer remote to the computer on which the application is executed. Other techniques enable modeling and virtualizing of single applications such that an internal user interface and application state are available to a remote client computer.

Often multiple associated applications are used simultaneously in an overall workflow to accomplish a set of tasks. Currently these applications are launched individually with individual tools for their virtualization.

When a user has to interact simultaneously with two or more applications using a display monitor the applications typically employ display images that share the display area of the monitor. For example, a first application generates a first user interface using a first image window to enable user interaction with the first application. When launching a second application a second user interface using a second image window is generated to enable user interaction with the second application. The second image window is placed in the foreground overlaying the first image window when in response to user interaction with the second application. Thus the first image window is at least partially obscured. When the user then interacts with the first application the situation is vice versa. This is a non-user friendly process substantially impeding demanding user interaction such as viewing and diagnosing medical images.

It would be desirable to enable simultaneous user interaction with a plurality of application programs using a single user interface.

SUMMARY

In accordance with embodiments of the present invention there is provided a method simultaneously interacting with a plurality of application programs. Using a processor an application interaction program is executed. The application interaction program has a plurality of application programs associated therewith. Using the processor interactive functionalities of the plurality of application programs are associated with respective interaction zones, wherein an interactive functionality being a same for at least two application programs is associated with a single interaction zone. Using the processor display data indicative of a single user interface comprising the interaction zones are generated. The interaction zones enable user interaction with the plurality of application programs. Using a graphical display connected to the processor the display data are displayed in a human comprehensible fashion.

In accordance with embodiments of the present invention there is provided a storage medium having stored therein executable commands for execution on a processor of a computer system, the processor when executing the commands executes an application interaction program. The application interaction program has a plurality of application programs associated therewith. The processor associates interactive functionalities of the plurality of application programs with respective interaction zones, wherein an interactive functionality being a same for at least two application programs is associated with a single interaction zone. The processor generates display data indicative of a single user interface comprising the interaction zones. The interaction zones enable user interaction with the plurality of application programs and provides the display data to a graphical display connected to the processor for displaying the display data in a human comprehensible fashion.

An example method for recombination of a plurality of application programs is described herein. The method can include providing a remote access framework for enabling a client to remotely access the plurality of application programs over a network, receiving a communication (e.g., requesting access to the plurality of application programs) from the client through the remote access framework, and maintaining a state model within the remote access framework. The plurality of application programs can run on at least one server. Additionally, the state model can include respective application states of the plurality of application programs. The method can also include, using the state model, linking interactive functionalities of the plurality of application programs, and generating display data indicative of a single user interface including the plurality of application programs. The method can further include communicating the display data to the client through the remote access framework.

Optionally and additionally, the method can include receiving user input data from the client through the remote access framework, determining an association of the user input data to a respective interaction zone using the state model, and providing the user input data to at least one of the plurality of application programs associated with the respective interaction zone through the remote access framework. The user input data can optionally be simultaneously provided to the at least two of the plurality of application programs associated with the single interaction zone.

Optionally and additionally, the method can include receiving application program data from the at least one of the plurality of application programs through the remote access framework. The application program data can be dependent upon the user input data. Additionally, the display data can be generated in dependence upon the application program data for displaying the application program data using the single user interface.

Optionally and additionally, the method can include receiving first application program data and second application program data from a first application program and a second application program, respectively, through the remote access framework. Additionally, the display data can be generated in dependence upon the first application program data and the second application program data for displaying the first application program data and the second application program data using the single user interface. Optionally, the display data can be generated for simultaneously displaying the first application program data and the second application program data using the single user interface.

Optionally and additionally, the method can include receiving user input data from the client through the remote access framework, determining an association of the user input data to first application program data associated with a first application program and to second application program data associated with a second application program using the state model, and providing the user input data to the first and second application programs. Additionally, the first and second application program data can be modified according to the user input data.

Optionally and additionally, the association of the user input data to the respective interaction zone can be determined based on a location of the respective interaction zone on a graphical display image of the client. For example, the association of the user input data to the respective interaction zone can be determined in a pixel-wise fashion.

Optionally and additionally, the interactive functionalities can be a subset of a plurality of functionalities of at least one of the plurality of application programs.

It should be understood that the above-described subject matter may also be implemented as a computer-controlled apparatus, a computing system, or an article of manufacture, such as a computer-readable storage medium.

Other systems, methods, features and/or advantages will be or may become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features and/or advantages be included within this description and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding parts throughout the several views.

FIG. 1 is a simplified block diagram of a system for implementing a method for simultaneously interacting with a plurality of application programs according to implementations described herein;

FIG. 2A is a simplified flow diagram of the method for simultaneously interacting with a plurality of application programs according to implementations described herein;

FIG. 2B is another simplified flow diagram of the method for recombination of a plurality of application programs according to implementations described herein;

FIG. 3 is a simplified block diagram illustrating a single navigable window for use with the method for simultaneously interacting with a plurality of application programs according to implementations described herein;

FIG. 4 is a simplified block diagram illustrating operation of server and client remote access program(s) according to implementations described herein; and

FIG. 5 illustrates a state model according to implementations described herein.

FIG. 6A is a block diagram illustrating recombination of a plurality of application programs according to implementations described herein.

FIG. 6B is another block diagram illustrating recombination of a plurality of application programs according to implementations described herein.

DETAILED DESCRIPTION

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present disclosure. As used in the specification, and in the appended claims, the singular forms “a,” “an,” “the” include plural referents unless the context clearly dictates otherwise. The term “comprising” and variations thereof as used herein is used synonymously with the term “including” and variations thereof and are open, non-limiting terms. The terms “optional” or “optionally” used herein mean that the subsequently described feature, event or circumstance may or may not occur, and that the description includes instances where said feature, event or circumstance occurs and instances where it does not. While implementations will be described for simultaneously interacting with a plurality of application programs using a single user interface (sometimes referred to herein as “recombination”), it will become evident to those skilled in the art that the implementations are not limited thereto.

Referring to FIG. 1, an example 100 of a system for implementing a method for simultaneously interacting with a plurality of application programs according to implementations described herein below is shown. The system 100 comprises a client computer 102 such as, for example, a personal computer, workstation, laptop computer, tablet computer, or smart phone. Optionally, the client computer 102 can be used for accessing one or more application programs (e.g., remote application programs) such as applications for viewing medical images of a patient and for entering a diagnostic report into a respective patient application file. Optionally, the client computer 102 can be used for accessing application programs for viewing three-dimensional (3D) and two-dimensional (2D) computer-aided designs (CAD) images. It should be understood that these application programs are provided only as examples and that the client computer 102 can be used for accessing other types of application programs. The client computer 102 is connected via a communication network 112 to

server computers

120A and 120B. The

server computers

120A and 120B are, for example, running

different application programs

124A and 124B. The

application programs

124A and 124B can be stored in memory, for example. Optionally, the

server computers

120A and 120B can be running different application programs for performing, for example, different image rendering processes of image data captured by different imaging systems such as MRI and CT scanners, and application programs for managing patient files such as, for example, application programs implemented in a PACS system. Optionally, the

server computers

120A and 120B can be running different application programs for performing, for example, different image rendering processes of 2D CAD images and 3D CAD images, respectively. It should be understood that these application programs are provided only as examples and that the server computers 120A and 120 be can be used for running other types of application programs. Alternatively, in some implementations, one of the server computers (i.e., server computer 120A or server computer 120B) can run different application programs. The communication network 112 comprises, for example, a Local Area Network (LAN) connecting the client computer 102 with the

server computers

120A and 120B within an organization such as, for example, a hospital. Alternatively, the communication network 112 comprises a Wide Area Network (WAN) or the Internet connecting client computers and server computers of various organizations such as, for example, hospitals, diagnostic clinics, and family physician's offices. It should be understood that the networks connecting client computers with server computers of medical organizations are provided only as examples and that the networks for connecting client computers with server computers of other organizations including, but not limited to, public and/or private companies, governments, educational institutions, or combinations thereof, can be implemented.

The implementations of the method for recombination of and/or simultaneously interacting with a plurality of application programs described herein are performed, for example, by executing executable commands stored in memory 106 of the client computer 102 using processor 104. An application interaction program is then executed in dependence upon user requests. For example, the application interaction program provides commands to

processors

122A and 122B of the

server computers

120A and 120B to execute a first application program and a second application program, respectively. During execution the application interaction program enables simultaneous user interaction with both application programs while providing a single user interface which is displayed on graphical display 108 of the client computer 102. Alternatively, one or more application programs are executed using the processor 104 of the client computer 102. As described below, client and server remote access program(s) of FIG. 4 and a state model of FIG. 5 can be used to facilitate recombination of and/or simultaneous interaction with the plurality of application programs. The processor 104 for executing the application interaction program is, for example, a Central Processing Unit (CPU) of the client computer 102. Optionally, a Graphics Processing Unit (GPU) is employed executing at least a portion of the application interaction program, for example, for generating display data. The client computer 102 can also include input device(s) such as a keyboard 110. It should be understood that the client computer 102 can include other types of input devices including, but not limited to, a mouse, touch screen, or other device for receiving user input data.

While implementations will be described for recombination of and/or simultaneously interacting with a plurality of application programs using a single user interface in a medical application for the sake of simplicity, it will become evident to those skilled in the art that the implementations are not limited thereto, but are applicable in numerous other fields where simultaneous interaction with a plurality of application programs using a single user interface is desired.

Referring to FIG. 2A, an example implementation of a method for simultaneously interacting with a plurality of application programs is provided. At 10, the application interaction program having a plurality of application programs associated therewith is executed. The plurality of application programs is, for example, user selected with a user being presented with a selection menu comprising a plurality of buttons—each button associated with an application program. Alternatively, the plurality of application programs is already associated with the application interaction program, for example, application programs needed for performing a specific task by the user. Optionally and additionally, each of the plurality of application programs can be remotely accessed application programs (e.g.,

application programs

124A and 124B running on

server computers

120A and 120B). At 12, interactive functionalities of the plurality of application programs are associated with respective interaction zones. An interactive functionality that is the same for at least two application programs is associated with a single interaction zone. The interactive functionalities of the plurality of application programs are, for example, simultaneously associated with respective interaction zones after the plurality of application programs has been selected. Alternatively, the interactive functionalities of the plurality of application programs are associated with respective interaction zones separately for each of the plurality of application programs. For example, after the user has selected a first application program the interactive functionalities of the first application program are associated with respective interaction zones and after the user has selected a second application program the interactive functionalities of the second application program are associated with respective interaction zones. If a functionality is in common with the first application program it is associated with the respective interaction zone of the first application program. Here a user is enabled to select an additional application program at a later time. Optionally, interactive functionalities of the plurality of application programs can be associated with respective interaction zones using the client and server remote access application(s) and/or state model, for example, as described below with respect to FIGS. 4 and 5.

The associated interactive functionalities are, for example, a subset of the functionalities of an application program. For example, the user is presented with a menu for selecting the subset of the functionalities enabling the user to select functions, for example, related to viewing but to omit functions related to editing, thus avoiding clutter on the screen. Further optionally, the user is enabled to select additional functionalities during execution of the application program if a need arises. Alternatively, the subset of functionalities is predetermined, for example, depending on the task the user is going to use the plurality of application programs for or access limitations of the user.

At 14, display data indicative of a single user interface comprising the interaction zones are generated. The interaction zones enable the user to interact with the plurality of application programs. The display data are then displayed—16—in a human comprehensible fashion using the graphical display 108 connected to the processor 104. Optionally, display data (e.g., a single user interface including respective display areas for different application programs) can be generated using the client and server remote access application(s) and/or state model, for example, as described below with respect to FIGS. 4 and 5.

Referring to FIG. 3, an example outlay of a single user interface 200 is illustrated, but the implementations are not limited thereto. The single user interface 200 is presented as a single navigable window which comprises a command field 202 with a plurality of interaction zones 204—for example, in the form of buttons—and

display areas

206 and 208. The interaction zones 204 are, for example, clicked on by a mouse click, using keyboard 110 commands, or touched using a touch screen. The display areas are for displaying, for example, imaging data provided by a first application program—206—and a second application program—208. The interaction zones 204 are grouped, for example, according to the functionality they are associated with—for example, one group related to commands used for viewing and another group related to commands used for editing—and/or the respective application program. Optionally, the interaction zones 204 when clicked on open a scroll down menu for selection of specific functions or open a field for receiving user information, for example, a patient's name typed by the user.

At 18, user input data are received, i.e. user has clicked on an interaction zone 204. An association of the received user input data to the respective interaction zone is then determined—20—and provided to the application program associated with the respective interaction zone—22. Optionally, user input data can be associated with the respective interaction zone and provided to the application program(s) using the client and server remote access application(s) and/or state model, for example, as described below with respect to FIGS. 4 and 5. If an interaction zone is associated with a functionality shared by at least two application programs the user input data are simultaneously provided to the at least two application programs associated with the single interaction zone. For example, the user is then enabled while viewing two sets of image data provided by two application programs—for example, a first application program for viewing CT scans and a second application program for viewing MRI data—which are simultaneously displayed in the

display areas

206 and 208 to simultaneously move up and down in both images. Optionally, the user is enabled and to select one application program for provision of the user input data thereto.

At 24, first application program data and second application program data are received from a first application program and a second application program, respectively, and display data are generated—26—in dependence upon the first application program data and the second application program data which are then simultaneously displayed in the

display areas

206 and 208 of the single user interface 200. Optionally, the display data in dependence upon the first and second application program data can be generated using the client and server remote access application(s) and/or state model, for example, as described below with respect to FIGS. 4 and 5.

The implementations described herein enable simultaneous interaction with a plurality of application programs—including programs utilizing Graphic Processing Units (GPU)—in the overall workflow to accomplish a set of tasks using a single user interface. The simultaneous interaction is realized by monitoring events such as, for example, key strokes or mouse clicks with respect to the location of the event—the respective interaction zone—on the graphical display image. The received user input data are associated to a respective interaction zone—which is associated to a functionality of an application program—based on the location of the interaction zone in the graphical display image. The location is determined, for example, in a pixel-wise fashion, i.e. is based on a pixel by pixel understanding of the user interface. The application interaction program is implemented using standard programming languages, for example, programming languages based on PUREWEB technology of CALGARY SCIENTIFIC of Calgary, AB, Canada.

Further optionally, the user is enabled to request linking of application programs using the single user interface in order to directly link the operation of the first application program to the operation of the second application program. For example, the operation of the second application program depends on a result produced by the first application program or during operation of the second application program data produced by the first application program are used.

Referring now to FIG. 4, a simplified block diagram illustrating operation of server and client remote access program(s) are described, which can be used to facilitate remote access to application programs. For example, as described above, client computer 102 can remotely access and simultaneously interact with

application programs

124A and 124B, which are different application programs running on

server computers

120A and 120B, respectively. Alternatively, client computer 102 can remotely access different application programs running on one of server computer 120A or server computer 120B.

To provide remote access to the

application programs

124A and 124B, communication is established between, e.g., the client computer 102 and the

server computers

120A and 120B at respective enumerated URLs associated with each of

application programs

120A and 120B. In particular, the communication is enabled by a server remote access program 107A of server computer 120A, a server remote access program 107B of server computer 120B, and a client remote access program 111 of client computer 102, as shown in FIG. 4. The server

remote access programs

107A and 107B can be stored in respective memories of

server computers

120A and 120B and executed by

processors

122A and 122B, respectively. Additionally, the client remote access program 111 can be stored in memory 106 and executed by processor 104. The communication may be over any type of network, as noted above.

During remote communication sessions with the client computer 102, the server

remote application programs

107A and 107B receive application data from

application programs

124A and 124B, respectively. Upon receipt, the server

remote application programs

107A and 107B generate

presentation data

109A and 109B (e.g., in accordance with a state model described with regard to FIG. 5), respectively, and transmits the same to the client remote access program 111.

Optionally or additionally, the display of the

application programs

124A and 124B can be presented through a process called recombination, which can be based on information contained in the state model of FIG. 5, described below. As will be described in greater detail, the state model includes an association of logical elements of the application programs with corresponding states of the application programs. With recombination, the logical elements in the state model associated with each of the

application programs

124A and 124B are seamlessly integrated into the same user interface (e.g., a single user interface) of client computer 102 such that views of both

application programs

124A and 124B can be presented simultaneously. For example, as described above, a single user interface including respective display areas for different application programs can be provided. Optionally or additionally, functionalities and controls associated with the

application programs

124A and 124B can be combined within the same user interface (e.g., a single user interface). For example, as described above, a plurality of interaction zones, which correspond to functionalities of a plurality of application programs, can be provided. In some implementations, a single interaction zone is associated with the same functionality of a plurality of application programs. For example, a plurality of image viewing application programs (e.g., application programs for viewing MRI and CT images or application programs for viewing 2D CAD and 3D CAD images) can share a common interaction zone (e.g., an arrow button). When a user selects this common interaction zone (e.g., the arrow button), the respective images of the application programs are manipulated (e.g., pan, scroll, zoom, etc.) simultaneously. This disclosure contemplates that recombination also optionally provides the ability to identify and coordinate common data shared by a plurality of application programs. In other words, by understanding the functionality of the application programs, it is possible to identify and synchronize underlying common data used by the application programs. For example, a plurality of application programs such as an application program for viewing a patient's medical images and a billing application program, respectively, can share common data (e.g., biographic data such as the patient's name). Each application can maintain the common data independently of the other application. For example, the image viewing application displays biographic data (e.g., the patient's name) as recorded in the image data file, while the billing application program displays the same biographic data (e.g., the patient's name) as recorded in a billing database. Recombination can be used to ensure that when a change is made to an instance of the common data in one application program, an event can be automatically generated (e.g., using the state model described herein) to cause the same change to be made to instance(s) of the common data in the other application program(s). For example, using recombination, the image viewing application and billing application (both including the biographic data) can be simultaneously displayed in a single user interface. In other words, recombination can be used to display the biographic data for the image viewing application and the billing application in a single user interface. This can be used for quality assurance (QA) applications, e.g., to ensure that the biographic data is correctly stored in both the image viewing application and the billing application. Additionally, in the event the patient's name is changed in one of the applications (e.g., in the billing application, “J Smith” is changed to “John P. Smith”), an event can be automatically generated that causes the same change to occur in the other application (e.g., on noting the change in the patient's name in the billing application, and using the state model, an “update patient name” event is generated with “John P. Smith” as the new information in the image viewing application). It should be understood that this saves effort and ensures data synchronization between the application programs. Optionally, a user can be prompted to confirm the change in the other application.

In some implementations, the operation of the remote access program (server and client) is performed in cooperation with a state model 300, as illustrated in FIG. 5. When executed, the client remote access program updates the state model 300 in accordance with user input data received from a user interface program. As described above, the user input data can include a user's selection of an interaction zone, which can be associated with an interactive functionality of one or more application programs. Alternatively or additionally, the user input can include a user's entry of, or modification to, application data. The remote access program may generate control data in accordance with the updated state model, and provide the same to the

application programs

124A and 124B running on the

server computers

120A and 120B. The

application programs

124A and 124B may be any application running on the

server computers

120A and 120B with which an end user interacts using the client computer 102.

As described above, upon receipt of application data from the respective application programs, the server remote access programs update the state model 300 in accordance with the screen or application data, generate presentation data in accordance with the updated state model, and provide the same to the user interface program on the client computer 102. The state model 300 comprises an association of logical elements of the application programs with corresponding states of the application programs, with the logical elements being in a hierarchical order. In some implementations, the state model 300 may represent the logical elements within an eXtensible Markup Language (XML) document. For example, the logical elements may be a screen, a menu, a submenu, a button, etc. that make up the application programs user interfaces. This enables the client computer 102, for example, to natively display the logical elements. As such, a menu of the application program 124A and/or 124B that is presented on a mobile phone will look like a native menu of the mobile phone. Similarly, the menu of the application program 124A and/or 124B that is presented on desktop computer will look like a native menu of the desktop computer operating system. In accordance with aspects of the present disclosure, the screen image may be communicated as a single logical element in a separate channel to the client. The screen image may be referenced in the state model to enable the client remote access program to retrieve the screen image from the separate channel and display it on the client. As such, the screen will have a substantially similar appearance on different client computers, although it may be scaled according to the characteristics of each device.

The state model 300 is determined such that each of the logical elements is associated with a corresponding state of the application program. The state model 300 may be determined such that the logical elements are associated with user interactions. For example, the logical elements of the application program are determined such that the logical elements comprise transition elements with each transition element relating a change of the state model 300 to one of control data and application representation data associated therewith.

In some implementations, the state information may include a reference or references to one or more rectangular regions that comprise the screen image generated by the application program. A control may be provided by the client remote access program to enable a user to select among the one or more rectangular regions for display at the client device. In some implementations, the control may provide a selection among one or more rectangular regions of screen image data created by plural application programs (e.g.,

application programs

124A and 124B) for selection by a user.

In some implementations, the state information may include information about a state of a display of the client computer 102. For example, display resolution may be maintained in the state model 300, which may be used as part of the refactoring of the presentation data in accordance with the client computing device's capabilities. The state model may also include information that may be used for recombination purposes.

Referring to FIG. 2B, an example implementation of a method for recombination of a plurality of application programs is provided. At 30, a remote access framework is provided. The remote access framework can allow a client (e.g., client computer 102) to remotely access the plurality of application programs (e.g.,

application programs

124A and 124B running on at least one server such as server computer 120A and/or server computer 120B) over a network (e.g., network 112). For example, the remote access framework can be implemented as described with respect to FIG. 4 (e.g., using client and server remote access application(s)) and FIG. 5 (e.g., using a state model). At 32, a communication from the client is received through the remote access framework. The communication can be a request to access the plurality of application programs. For example, as described above, the client and the server(s) can establish connections at respective URLs associated with each of the plurality of application programs. At 34, a state model can be maintained within the remote access framework. The state model can include respective application states of the plurality of application programs, for example, as described above with regard to FIG. 5.

At 36, using the state model, interactive functionalities of the plurality of application programs can be linked. As described above, the respective interaction zones enable user interaction with the plurality of application programs. For example, for image viewing application programs (e.g., applications for viewing medical images, CAD images, etc.), the interactive functionalities can include a subset of functionalities enabling a user to view images (e.g., pan, scroll, zoom, etc.). The interactive functionalities can optionally be associated with buttons, for example, which can be selected by a user's keystroke, mouse, touch, etc. at the client. In some implementations, a same interactive functionality for at least two application programs can be associated with a single interaction zone. For example, as described above, two (or more) application programs can share the same functionality. By providing a single interaction zone for the shared functionality, user input data can be simultaneously provided to at least two application programs. In some implementations, interactive functionalities for at least two application programs can be linked by linking views and/or application state information of the respective application programs. At 38, display data indicative of a single user interface (e.g., user interface 200) including the plurality of applications can be generated. At 40, the display data can be communicated to the client through the remote access framework. The display data can then be displayed, for example, using a display device of the client (e.g., graphical display 108).

Optionally and additionally, as described above, a user can input data (e.g., user input data) at the client, for example, using a user interface program. This can optionally include selection of an interaction zone, for example by keystroke, mouse, touch, etc. at the client. Using the remote access framework, the client (e.g., using the client remote access program 111) can update the state model (e.g., state model 300) in accordance with the user input data. Using the state model, the interaction zone can be associated with an interactive functionality of at least one application program. Then, using the remote access framework, the server (e.g., using the server

remote access program

107A or 107B) can generate control data in accordance with the updated state model and provide the same to the at least one application program. When a single interaction zone is associated with a functionality shared by two or more applications programs, the control data can be simultaneously provided to the two or more application programs. Alternatively or additionally, when views and/or application state are linked, control data can be automatically generated and provided in one application program based on a user input in the other application program.

Optionally and additionally, as described above, the user input data can modify first application program data associated with a first application program and second application program data associated with a second application program. Using the remote access framework, the client (e.g., using the client remote access program 111) can update the state model (e.g., state model 300) in accordance with the user input data. Then, using the state model, the user input data can be associated with the first application program data and the second application program data. For example, the first and second application program data can optionally be data common to two application programs (e.g., patient biographical data). Then, using the remote access framework, the server (e.g., using the server

remote access program

107A or 107B) can generate control data in accordance with the updated state model and provide the same to the first and second application programs, which causes the first and second application program data to be modified. Accordingly, common data can be synchronized by two (or more) application programs.

Optionally and additionally, as described above, the user input data can cause the at least one application program to change application states. Using the remote access framework, the server (e.g., using the server

remote access program

107A or 107B) can receive application program data from the at least one application program. In other words, the application program data depends on the user input data (e.g., the change of application state caused by the user input data). The server (e.g., using the server

remote access program

107A or 107B) can update the state model (e.g., state model 300) in accordance with the application data. Then, the server (e.g., using the server

remote access program

107A or 107B) can generate display data (e.g., presentation data 109A and/or 109B) in dependence upon the application program data for displaying the application program data using the single user interface (e.g., user interface 200). As described above, the display data can be communicated to the client through the remote access framework. The display data can then be displayed, for example, using a display device of the client (e.g., graphical display 108). Optionally, as described above, the server (e.g., using the server

remote access program

107A or 107B) can receive first and second application program data from first and second application programs, respectively. Optionally, the display data can be generated for simultaneously displaying the first application program data and the second application program data using the single user interface (e.g., in

display areas

206 and 208, respectively, of user interface 200).

Examples

The systems and methods described herein enable a plurality of application programs to interoperate together on the server-side such that the application programs can be recombined into a single workflow on the client side. The systems and methods described herein seamlessly integrate logical elements in a state model associated with a plurality of applications programs into a single user interface. The systems and methods described herein enable simultaneous interaction with a plurality of application programs using a single user interface. The systems and methods described herein enable display of a plurality of applications programs in a single user interface. The systems and methods described herein provide the ability to identify and coordinate common data shared by a plurality of application programs. Each of these aspects, alone or in combination with other aspects, of the systems and methods is referred to herein as “application recombination” or “recombination.” Recombination allows a plurality of application programs (e.g., applications of a company or enterprise that has grown through merger and acquisition) to interact and be presented with a common user interface client-side. In particular, a logic layer can be created that recombines application logic of the application programs into a new workflow. In some implementations, recombination can be implemented using PUREWEB technology of CALGARY SCIENTIFIC of Calgary, AB, Canada.

Referring now to FIGS. 6A and 6B, block diagrams illustrating recombination of a plurality of application programs according to implementations described herein are shown. As shown in FIG. 6A, a client (e.g., client computer 102) remotely accesses a plurality of applications programs A and B (e.g.,

application programs

124A and 124B running on at least one server such as server computer 120A and/or server computer 120B) over a network (e.g., network 112). The client can remotely access the application programs A and B using a remote access framework such as described with respect to FIG. 4 (e.g., using client and server remote access application(s)) and FIG. 5 (e.g., using a state model). As described above, the client can remotely access the application programs via respective URLs associated with the application programs. As shown in FIG. 6B, a plurality of clients (e.g., two mobile devices and a desktop client) remotely accesses a plurality of applications programs A and B over a network. In addition, each of the clients in FIG. 6B is capable of natively displaying application programs A and B. Optionally, in FIGS. 6A and 6B, PUREWEB technology of CALGARY SCIENTIFIC of Calgary, AB, Canada can provide the remote access framework. For example, in FIGS. 6A and 6B, remote access framework A and remote access framework B are provided for application programs A and B, respectively. As shown in FIG. 6B, the remote access framework can be used to pass commands, views, and/or state information between the client and the application programs A and B. The remote access framework can be used to recombine application programs A and B into a single user interface as described herein.

Recombination of application programs A and B can include linking views (e.g., display data including but not limited to interactive zones) and/or application state. This is shown by the dotted line in FIG. 6A. Recombination can also include implementing commands in each of application programs A and B. For example, recombination enables an understanding of the application functionality of application programs A and B when views and/or application state are linked, and therefore, commands entered in application program A or B can be executed in the other application program B or A, for example, by automatically generating the same command in the other program. Example pseudocode for linking views of application programs A and B is provided below:


// create first view...
var view1 = new rasframework.client.View({id: ‘myView1’, viewName:
‘myView1’});
// create second view...
var view2 = new rasframework.client.View({id: ‘myView2’, viewName:
‘myView2’});
// now link the two views.
rasframework.client.linkViews(view1, view2);
//...now all input operations (e.g., mouse/keyboard operations) in one view
will be
//automatically reflected in the other view
// now unlink the two views...
rasframework.client.unlinkViews(view1, view2);

This disclosure contemplates that when application program views (e.g., View1 and View2 above) are linked, it is possible to generate operations received in one linked application program in the other linked application program. For example, with reference to FIGS. 6A and 6B, application programs A and B are linked. When a user performs an operation in the view of application program A on the single user interface (e.g., a scroll operation), the user input is passed through the remote access framework (e.g., remote access framework A) to application program A. Since application programs A and B are linked, a corresponding command can be automatically generated and passed, for example by the remote access framework (e.g., remote access framework B), to perform the same operation (e.g., a scroll operation) in application program B. Thereafter, respective application data and/or presentation data for each of application programs A and B can be generated by the remote application frameworks and then integrated into the single user interface. Thus, it is the understanding of application functionality provided by recombination that allows the linking of interactive functionalities of application programs A and B. It should be understood that the scroll operation is described as the interactive functionality as an example only and that other interactive functionalities of the application programs can be linked using the systems and methods described herein.

Example pseudocode for linking application state of application programs A and B is provided below:


	// specify application state variable in first framework
	var framework1 = rasframework.getFramework(0);
	framework1.getState().setValue(‘ScribbleColor’, ‘red’);
	// specify application state variable in second framework
	var framework2 = rasframework.getFramework(1);
	framework2.getState().setValue(‘lineColor’, ‘blue’);
	// now specify link the application state
	rasframework.frameworks.linkState(framework1, ‘ScribbleColor’,
	framework2, ‘lineColor’);

Alternatively or additionally, this disclosure contemplates linking application state information of the application programs. For example, with reference to FIGS. 6A and 6B, application programs A and B are linked. In some implementations, this includes linking logical elements (e.g., a menu, submenu, button, etc.) of application programs A and B that correspond to the same interactive functionality (e.g., linking the same interaction zone of application programs A and B). In other implementations, this includes linking logical elements that are similar, but different, in application programs A and B. In other words, this includes linking logical elements that are related but not necessarily the same such as the red ScribbleColor and the blue lineColor in the pseudocode above. For example, the red ScribbleColor and the blue lineColor may optionally correspond to mark-ups in respective images stored and displayed by application programs A and B. Changes to the linked application state information (e.g., the red ScribbleColor and the blue lineColor) can be passed to through the remote access framework (e.g., remote access framework A and/or remote access framework B) to the application programs A and/or B. Thereafter, respective application data and/or presentation data for each of application programs A and B can be generated by the remote application frameworks and then integrated into the single user interface. Thus, it is the understanding of application functionality provided by recombination that allows the linking of application state information of application programs A and B.

Example Use Cases

As described above, using the systems and methods described herein, it is possible to recombine a plurality of application programs. In some implementations, the application programs can be applications for viewing medical images of a patient and for entering a diagnostic report into a respective patient application file. In other implementations, the application programs can be applications for viewing three-dimensional (3D) and two-dimensional (2D) computer-aided designs (CAD) images. It should be understood that these application programs are provided only as examples. Other non-limiting examples for the use of application program recombination as described herein include:

a. Recombination can be used for enterprise application integration (i.e., integrating application programs across an enterprise).

b. Two (or more) application programs running simultaneously for displaying a prior study and a current study using simultaneously two displays.

c. When data common to two (or more) application programs (e.g., patient name data) is changed in one application, an event is automatically generated in the other application(s) that causes the common data (e.g., patient name data) to also be changed in the other application(s).

d. Commonality can be purely user interaction, e.g., two viewer application programs that both use the same input command (e.g., the arrow keys) to scroll through documents and/or images.

e. A user (e.g., physician) launches a recombined viewer application to access two different servers at two different hospitals, where the same patient's images are displayed in two views within a single client browser window. In response to pan or scroll operation, the recombined viewer application sends appropriate commands to each independent application at the independent servers.

f. A quality assurance (QA) application that displays and affects multiple silo/databases simultaneously without requiring health level seven (HL7) messaging to perform updates. This may be useful for QA of data maintained in separate applications of the same enterprise.

g. A PACS workstation that can also pop-up windows from the same vendor's nuclear medicine viewer.

h. A universal worklist, e.g., a worklist coming from different vendors for a common worklist display for a single radiologist reading from multiples hospitals. For example, a viewer that displays the worklist from a modality in the same browser as the worklist from the radiology information system (RIS), both coming from different vendors, each required to be correct as it was generated by the modality or RIS directly. The worklist can be displayed in a window, tile, area, etc. of the user interface or display. It should be understood that there would be no delay or opportunity for database mistakes during worklist collection into a third party environment using the systems and methods described herein.

i. A universal worklist display for a single radiologist reading from multiple hospitals, where the radiologist's credentials are stored and passed to each of the relevant hospital's RIS viewers. Each of the RIS viewers are stripped down and then displayed in a respective window, tile, box, area, etc. of the user interface or display. In this example, the worklists are not merged (as in example (h) above) and instead are just displayed.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

What is claimed:

1. A method for providing recombination of a plurality of application programs, comprising:

providing a remote access framework for enabling a client to remotely access the plurality of application programs over a network, the plurality of application programs running on at least one server;

receiving a communication from the client through the remote access framework, the communication requesting access to the plurality of application programs;

maintaining a state model within the remote access framework, the state model comprising respective application states of the plurality of application programs;

using the state model, linking interactive functionalities of the plurality of application programs;

using the state model, generating display data indicative of a single user interface including the plurality of application programs; and

communicating the display data to the client through the remote access framework.

2. The method of claim 1, further comprising:

receiving user input data from the client through the remote access framework;

using the state model, determining an association of the user input data to a respective interaction zone; and

providing the user input data to at least one of the plurality of application programs associated with the respective interaction zone through the remote access framework.

3. The method of claim 2, further comprising:

receiving application program data from the at least one of the plurality of application programs through the remote access framework, the application program data being in dependence upon the user input data, wherein the display data is generated in dependence upon the application program data for displaying the application program data using the single user interface.

4. The method of claim 2, wherein the user input data are simultaneously provided to the at least two of the plurality of application programs associated with the single interaction zone.

5. The method of claim 2, further comprising:

receiving first application program data and second application program data from a first application program and a second application program, respectively, through the remote access framework, wherein the display data is generated in dependence upon the first application program data and the second application program data for displaying the first application program data and the second application program data using the single user interface.

6. The method of claim 5, wherein the display data are generated for simultaneously displaying the first application program data and the second application program data using the single user interface.

7. The method of claim 1, further comprising:

receiving user input data from the client through the remote access framework;

using the state model, determining an association of the user input data to first application program data associated with a first application program and to second application program data associated with a second application program; and

providing the user input data to the first and second application programs, wherein the first and second application program data are modified according to the user input data.

8. The method of claim 2, wherein the association of the user input data to the respective interaction zone is determined based on a location of the respective interaction zone on a graphical display image of the client.

9. The method of claim 8, wherein the association of the user input data to the respective interaction zone is determined in a pixel-wise fashion.

10. The method of claim 1, wherein the interactive functionalities are a subset of a plurality of functionalities of at least one of the plurality of application programs.

11. A non-transitory computer-readable medium having computer-executable instructions stored thereon for recombination of a plurality of application programs that, when executed by a processor, cause the processor to:

provide a remote access framework for enabling a client to remotely access the plurality of application programs over a network, the plurality of application programs running on at least one server;

receive a communication from the client through the remote access framework, the communication requesting access to the plurality of application programs;

maintain a state model within the remote access framework, the state model comprising respective application states of the plurality of application programs;

using the state model, link interactive functionalities of the plurality of application programs;

using the state model, generate display data indicative of a single user interface including the plurality of application programs; and

communicate the display data to the client through the remote access framework.

12. The non-transitory computer-readable medium of claim 11, having further computer-executable instructions stored thereon that, when executed by the processor, cause the processor to:

receive user input data from the client through the remote access framework;

using the state model, determine an association of the user input data to a respective interaction zone; and

provide the user input data to at least one of the plurality of application programs associated with the respective interaction zone through the remote access framework.

13. The non-transitory computer-readable medium of claim 12, having further computer-executable instructions stored thereon that, when executed by the processor, cause the processor to:

receive application program data from the at least one of the plurality of application programs through the remote access framework, the application program data being in dependence upon the user input data, wherein the display data is generated in dependence upon the application program data for displaying the application program data using the single user interface.

14. The non-transitory computer-readable medium of claim 12, wherein the user input data are simultaneously provided to the at least two of the plurality of application programs associated with the single interaction zone.

15. The non-transitory computer-readable medium of claim 12, having further computer-executable instructions stored thereon that, when executed by the processor, cause the processor to:

receive first application program data and second application program data from a first application program and a second application program, respectively, through the remote access framework, wherein the display data is generated in dependence upon the first application program data and the second application program data for displaying the first application program data and the second application program data using the single user interface.

16. The non-transitory computer-readable medium of claim 15, wherein the display data are generated for simultaneously displaying the first application program data and the second application program data using the single user interface.

17. The non-transitory computer-readable medium of claim 11, having further computer-executable instructions stored thereon that, when executed by the processor, cause the processor to:

receive user input data from the client through the remote access framework;

using the state model, determine an association of the user input data to first application program data associated with a first application program and to second application program data associated with a second application program; and

provide the user input data to the first and second application programs, wherein the first and second application program data are modified according to the user input data.

18. The non-transitory computer-readable medium of claim 12, wherein the association of the user input data to the respective interaction zone is determined based on a location of the respective interaction zone on a graphical display image of the client.

19. The non-transitory computer-readable medium of claim 18, wherein the association of the user input data to the respective interaction zone is determined in a pixel-wise fashion.

20. The non-transitory computer-readable medium of claim 11, wherein the interactive functionalities are a subset of a plurality of functionalities of at least one of the plurality of application programs.