WO2007070225A1

WO2007070225A1 - Alternative graphics pipe

Info

Publication number: WO2007070225A1
Application number: PCT/US2006/044927
Authority: WO
Inventors: Jeremy De Souza; Matthew B. Karr
Original assignee: Microsoft Corporation
Priority date: 2005-12-12
Filing date: 2006-11-17
Publication date: 2007-06-21
Also published as: US20070132753A1; CN101326513A; US7773096B2; KR101331337B1; RU2008123838A; KR20080076938A; BRPI0618551A2; RU2433462C2; EP1960900A4; JP2009519542A; EP1960900A1; CN101326513B; JP4928558B2

Abstract

Various technologies and techniques are disclosed that improve the operation of accessibility applications. A graphics pipe is provided that can be called in user mode from multiple accessibility programs. A request is received from an accessibility application to access the graphics pipe, and a connection is established. The accessibility application listens to the graphics pipe for particular content of interest and builds a model based on that content. The model is used to deliver content to an end user appropriately. Screen captures can be performed on at least part of the content and then rendered onto another surface.

Description

ALTERNATIVE GRAPHICS PIPE BACKGROUND

[001] Assistive technologies are software or hardware products that make software applications or operating systems accessible to individuals with a range of disabilities, such as impaired mobility, sight, hearing, etc. Examples of assistive technologies include magnifiers, screen readers, and Braille displays. These products use a variety of data interception techniques throughout the operating system in order to operate. Generally, assistive technologies intercept graphics primitive function calls at the display driver interface (DDI) level and use the operating system kernel state to build off-screen models. Such techniques often cause system instability and crashes.

SUMMARY

[002] Various technologies and techniques are disclosed that improve the operation of accessibility applications. A graphics pipe is provided that can be called in user mode from multiple accessibility programs simultaneously and/or separately. A request is received from an accessibility application to access the graphics pipe, and a connection is established. The accessibility application listens to the graphics pipe for particular content of interest and builds a model based on that content. The model is used to deliver content in an accessibility application to an end user appropriately. Screen captures can be performed on at least part of the content and then rendered onto another surface.

[003] This Summary was provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[004] FIG. 1 is a diagrammatic view of parts of a graphics pipe system. [005] FIG. 2 is a diagrammatic view of a computer system of one implementation of the system of FIG. 1. [006] FIG. 3 is a diagrammatic view of an accessibility graphics pipe application operating on the computer system of FIG. 2.

[007] FIG. 4 is a high-level process flow diagram for one implementation of the system of FIGS. 1 and 2. [008] FIG. 5 is a process flow diagram for one implementation of the system of FIGS. 1 and 2 illustrating the stages involved in performing screen captures on the graphics pipe and drawing the screen captures to another surface.

[009] FIG. 6 is a process flow diagram for one implementation of the system of FIGS. 1 and 2 illustrating the stages involved in screen readers or Braille displays accessing the graphics pipe and building a content model.

[010] FIG. 7 is a process flow diagram for one implementation of the system of FIGS. 1 and 2 illustrating the stages involved in magnifiers accessing the graphics pipe and building a content model.

DETAILED DESCRIPTION [011] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope is thereby intended. Any alterations and further modifications in the described embodiments, and any further applications of the principles as described herein are contemplated as would normally occur to one skilled in the art.

[012] The system may be described in the general context as an application that improves the operation of accessibility applications and their related assistive technologies, such as screen readers, screen magnifiers, and Braille displays. One or more of the techniques described herein can be implemented as features within a graphics pipe application, or from any other type of program or service that facilitates accessibility scenarios. As described in further detail herein, in one implementation of the system, a graphics pipe is provided that can be called in user mode from multiple accessibility programs simultaneously. In another implementation, the accessibility application listens to the graphics pipe for particular content of interest and builds a model based on that content. The model is used to deliver content in an accessibility application to an end user appropriately.

[013] As shown in Figure I₅ graphics pipe system 20 includes graphics pipe 21 and accessibility applications (22, 24, and 26, respectively). Graphics pipe 21 allows accessibility applications 22, 24, and/or 26 to intercept graphics primitives (e.g. geometry calls, text calls) 18, and/or information related to custom owner drawn controls 19. Graphics pipe 21 serves as a central location for accessibility applications to retrieve graphic display information that can be modeled and used in rendering content (and modifying content, if appropriate) in accessibility scenarios. In one implementation, connections through graphics pipe 21 are in user mode, instead of kernel mode, thereby providing a more reliable operating environment.

[014] In one implementation, accessibility application 22 is coupled to graphics pipe 21 in read-only user mode over communication pathway 28, and serves as screen reader 34. Accessibility application 24 is coupled to graphics pipe 21 in read and/or update user mode over communication pathway 30, and serves as a screen magnifier 36. Similarly, accessibility application 26 is coupled to graphics pipe 21 in read-only user mode over communication pathway 32, and serves as a Braille display. In one implementation, screen readers and Braille displays do not need to alter the content of graphics pipe 21, so their respective connections to graphics pipe 21 are read-only. Numerous other accessibility applications and assistive technologies could be used instead of or in addition to those shown in Figure 1.

[015] As shown in Figure 2, an exemplary computer system to use for implementing one or more parts of the system 20 includes a computing device, such as computing device 100. In its most basic configuration, computing device 100 typically includes at least one processing unit 102 and memory 104. Depending on the exact configuration and type of computing device, memory 104 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in Figure 2 by dashed line 106. [016] Additionally, device 100 may also have additional features/ functionality. For example, device 100 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in Figure 2 by removable storage 108 and non-removable storage 110. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory 104, removable storage 108 and non-removable storage 110 are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by device 100. Any such computer storage media may be part of device 100.

[017] Computing device 100 contains one or more communications interface (s) 114 that allow the device to communicate with other devices. For example, communications interface(s) 114 allows computing device 100 to communicate with one or more other computers and/or applications 115, where applicable. Examples of communications interfaces are serial ports, Universal Serial Bus (USB) ports, parallel ports, wireless communication adapters, network adapters, etc. Communications interface(s) 114 are used by computer 100 to exchange information such as communication media with external devices. Some examples of communication media are computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct- wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.

[018] Device 100 may also have input device(s) 112 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 111 such as a display, screen reader, Braille display, magnifier, speakers, printer, etc. may also be included. These devices are well known in the art and need not be discussed at length here.

[019] Turning now to Figure 3 with continued reference to Figure 2, an accessibility graphics pipe application 200 operating on computing device 100 is illustrated. In one implementation, accessibility graphics pipe application 200 is included as part of the resident operating system on system memory 104, such as MICROSOFT® WINDOWS® or Linux. In another embodiment, accessibility graphics pipe application 200 is one of the application programs that reside on computing device 100. Alternatively or additionally, one or more parts of accessibility graphics pipe application can be part computers and/or applications 115, or other such variations as would occur to one in the computer software art. [020] Accessibility graphics pipe application 200 includes business logic 204, which is responsible for carrying out some or all of the techniques described herein. Business logic may include logic 206 for allowing read and/or updates to the graphics pipe by accessibility applications, logic 208 for supporting legacy content primitives, logic 210 for tagging content in the pipe with a handle to the control window, logic 212 for making off- screen content available as a bitmap, logic 214 for providing an indication that one or more assistive technologies are connected to the graphics pipe, logic 216 for allowing multiple clients to access the pipe concurrently and/or asynchronously, logic 218 for allowing owner drawn controls to be accessed through the graphics pipe, logic 220 for forcing applications to repaint upon new client connection to the graphics pipe, and other logic 222 for operating accessibility graphics pipe application 200.

[021] In one implementation, accessibility graphics pipe application 200 resides on computing device 100. It will be understood that business logic 204 of graphics pipe application 200 can alternatively or additionally be embodied as computer-executable instructions on one or more computers and/or in different variations than shown on Figures 2 and 3. As one non-limiting example, one or more parts of business logic 204 could alternatively or additionally be implemented as a service that resides on an external computer that is called when needed. [022] Turning now to Figures 4-7 with continued reference to Figures 1-3, the stages for implementing one or more implementations of accessibility graphics pipe application 200 are described in further detail. It will be appreciated that some, all, or fewer of these stages can be performed, and that they can be performed in a variety of different orders than as described in Figures 4-7. Figure 4 is a high level process flow diagram of one implementation of accessibility graphics pipe 200. In one form, the process of Figure 4 is at least partially implemented in the operating logic of computing device 100 and is executed as part of business logic 204.

[023] The process begins at start point 240 with accessibility application opening a connection to graphics pipe in user mode (stage 242). In one implementation, when the accessibility application connects, the graphics pipe tells all applications to repaint (stage 244) so they will have the most current content. The graphics pipe provides content (stage 246), and the accessibility application listens to the pipe for that content (stage 248). The content can include a client status flag indicating whether or not assistive technology is connected (stage 246), and/or the content can include off-screen content rendered in bitmaps (stage 246). The accessibility application builds a model to use at least part of the pipe (stage 250). The accessibility application closes the connection to the graphics pipe when finished (stage 252). The stages are repeated for each accessibility application (one or more of 22, 24, and/or 26) that accesses the graphics pipe, which can be simultaneously and/or separately (stage 254). The process then ends at end point 256.

[024] Turning now to Figure 5, a process flow diagram for one implementation of the system of Figure 1 illustrates the stages involved in performing screen captures on the graphics pipe and drawing the screen captures to another surface. In one form, the process of Figure 5 is at least partially implemented in the operating logic of computing device 100. The process begins at start point 260 with accessibility application opening a connection to the graphics pipe in user mode (stage 262). Accessibility application listens to the graphics pipe and performs screen captures on at least a portion of the content (stage 264). Accessibility application then draws at least some of the screen captures to another surface, such as to a file or video for vision assistance and/or training (stage 266). The accessibility application then closes the connection to the graphics pipe (stage 268). The process then ends at end point 269.

[025] Turning now to Figure 6, a process flow diagram for one implementation of the system of Figure 1 illustrates the stages involved in screen readers or Braille displays accessing the graphics pipe and building a content model. In one form, the process of Figure 6 is at least partially implemented in the operating logic of computing device 100. The process begins at start point 270 with a screen reader or Braille display client application opening a connection to the graphics pipe, such as in a read-only fashion (stage 272). The screen reader or Braille display client application listens to the graphics pipe for relevant information (stage 274). The screen reader or Braille display builds off-screen models and uses these models to output spoken voice or tactile feedback (stage 276). The screen reader or Braille display client application closes the connection to the graphics pipe (stage 278). The process then ends at end point 280.

[026] Turning now to Figure 7, a process flow diagram for one implementation of the system of Figure 1 illustrates the stages involved in magnifiers accessing the graphics pipe and building a content model. In one form, the process of Figure 7 is at least partially implemented in the operating logic of computing device 100. The process begins at start point 300 with the magnification application opening a connection to the graphics pipe such as in read-only and/or update mode (stage 302). The magnification application listens to the graphics pipe for relevant information (stage 304). The magnification application removes the client window from magnified content, if applicable (stage 306). [027] Alternatively or additionally, magnification application rescales content that it obtains from the graphics pipe, such as primitives and/or surfaces (stage 308). Any pre-composed filtering is also performed if applicable (stage 310). Magnification application composes visuals and renders the data that have been magnified (stage 312). Post-composition filtering is performed by magnification application, if applicable (stage 314). When finished, magnification application closes the connection to the graphics pipe (stage 316). The process then ends at end point 318.

[028] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. All equivalents, changes, and modifications that come within the spirit of the implementations as described herein and/or by the following claims are desired to be protected. [029] For example, a person of ordinary skill in the computer software art will recognize that the client and/or server arrangements, user interface screen content, and/or data layouts as described in the examples discussed herein could be organized differently on one or more computers to include fewer or additional options or features than as portrayed in the examples.

Claims

What is claimed is:

1. A computer-readable medium having computer-executable instructions for causing a computer to perform steps comprising: providing a graphics pipe that is operable to be called from a plurality of accessibility applications (21); receiving a request from a first accessibility application to access the graphics pipe (242); establishing a connection between the first accessibility application and the graphics pipe (242); and providing content from the graphics pipe to the first accessibility application (246).

2. The computer-readable medium of claim 1, further comprising the steps of: receiving a request from a second accessibility application to access the graphics pipe during at least a portion of a same time period as the first accessibility application (254); establishing a connection between the second accessibility application and the graphics pipe (242); and providing content from the graphics pipe to the second accessibility application (246).

3. The computer-readable medium of claim 1, wherein the graphics pipe communicates with each running application and tells each running application to repaint after the connection is established with the first accessibility application (244).

4. The computer-readable medium of claim 1 , wherein the graphics pipe is operable to be called in a user mode (242).

5. The computer-readable medium of claim 1, wherein the graphics pipe is operable to be called asynchronously (216).

6. The computer-readable medium of claim 1, wherein the accessibility application is operable to communicate with at least one output device selected from the group consisting of a screen reader, a Braille display, and a magnifier (111).

7. The computer-readable medium of claim 1 , wherein the providing content step further comprises the step of: providing a client status flag to indicate whether an assistive technology is connected to the graphics pipe at a particular moment (246).

8. The computer-readable medium of claim 1 , wherein the providing content step further comprises the step of: providing off-screen content in a bitmap format (246).

9. A computer-readable medium having computer-executable instructions for causing a computer to perform steps comprising: opening a connection to a graphics pipe from an accessibility application (262); from the accessibility application, listening to a set of content received from the graphics pipe and performing a set of screen captures on at least a portion of the content (264); drawing at least some of the screen captures to another surface (266); and closing the connection between the accessibility application and the graphics pipe (268).

10. The computer-readable medium of claim 9, wherein the screen captures are drawn to another surface for vision assistance (266).

11. The computer-readable medium of claim 9, wherein the screen captures are written to a file (266).

12. The computer-readable medium of claim 11 , wherein the screen captures are written to a file for use in a training video (266).

13. A method for using an accessibility graphics pipe comprising the steps of: opening a connection between a graphics pipe and an accessibility application in a user mode (242); from the accessibility application, listening to the graphics pipe for relevant information (248); and from the accessibility application, building an off-screen model with at least some of the information (250).

14. The method of claim 13, wherein the off-screen model is used to output a spoken voice (276).

15. The method of claim 13, wherein the off-screen model is used to output tactile feedback (276).

16. The method of claim 13 , wherein the off-screen model is used to output tactile feedback (276).

17. The method of claim 13, wherein the accessibility application is a screen reader (274).

18. The method of claim 13 , wherein the accessibility application is a Braille provider (272).

19. The method of claim 13, wherein the accessibility application is a magnifier (302).

20. A computer-readable medium having computer-executable instructions for causing a computer to perform the steps recited in claim 13 (200).