WO2017142391A1

WO2017142391A1 - Screen reader with visual and system interpreters for non-textual information

Info

Publication number: WO2017142391A1
Application number: PCT/MY2016/050080
Authority: WO
Inventors: Ngip Khean CHUAN; Ashok SIVAJI
Original assignee: Mimos Berhad
Priority date: 2016-02-17
Filing date: 2016-11-25
Publication date: 2017-08-24

Abstract

Screen reader with Visual and System Interpreters are provided for interpreting non-textual information. The screen reader (101a) comprising at least one Main Controller (110); at least one Listener (102); at least one Impairment Adjuster (104); at least one Element Analyzer (103); at least one set of Interpreter Modules (105, 106); at least one Outputer (107); at least one Archiver (108); and at least one Contextual Command Module (109). The at least one set of Interpreter Modules (105, 106) comprising at least one System Interpreter and at least one Visual Interpreter for interpreting non-textual information of said screen reader for visually impaired. The Element Analyzer module identifies and collects non-textual information of the user interface elements that users interact with. File information, table layout, and text pattern are example of the non-textual information that can be collected. This information can then be output in pre-configured manner and be archived for contextual command purposes. The Impairment Adjuster used to include and exclude non-textual attributes that needs to be obtained for each instance of use. Each attribute will have a default impairment value whereby users can easily select their visual impairment condition (e.g., 0, 10, 20... 100%). The screen reader will only collect attributes with the appropriate impairment value which allows users to edit to include and exclude the required attributes by adjusting the impairment value of the attribute individually.

Description

SCREEN READER WITH VISUAL AND SYSTEM INTERPRETERS FOR NON-TEXTUAL

INFORMATION

FIELD OF INVENTION The present invention relates to a system and method for interpreting non-textual information through a screen reader in communication with a machine readable framework. In particular, the present invention provides a screen reader for interpreting non-textual information by utilization of visual and system interpreters. BACKGROUND ART

Existing accessibility framework in mainstream operating system (e.g., Windows, Mac OS X, and Linux) does not provide sufficient practical coverage to collect non-textual information. Example of accessibility frameworks that have extensive accessibility of application programming interface, APIs is Windows User Interface, Ul Automation. However, developers are required to implement these APIs into their programs. Due to lack of monetization value versus the huge effort needed to implement the accessibility of APIs, the accessibility of APIs is not implemented. Further, there are various levels of visual impairments whereby types of impairment are different for different causes of visual impairment. Different visually impaired users would have different needs of non-textual information.

United States Patent No. US 8826137 B2 (US 137 Patent) entitled "Screen Reader having Concurrent Communication of Non-Textual Information" having a filing date of 12 August 2004 (Applicant: Freedom Scientific, Inc.) relates to a screen reader application to read documents through a word processor or read web pages from a web browser; said screen reader also provides information relating to graphic user interface (GUI) and menu selections available to the end user. The screen reader as provided in the US 137 Patent is applicable for product for low-vision users having a reader module that is able to collect the textual and non-textual display information generated by a web browser or web processor. The US 137 Patent utilized various way of outputting the non-textual information. For example, a broadcast module communicates the display information collected by reader module to an output device. The broadcast module includes controls and logic to determine what display information is sent to the output device as well as how it is presented to the user through the output device. Non-textual display information sounds may be generated by the screen reader application or may be prerecorded digital audio. Non-textual information may be communicated through a Braille display by altering the speed at which the pins move by modifying the distance or level by which pins protrude or retract from the surface of the Braille display. Further, in the US 137 Patent, speech synthesizer may be combined with a Braille display wherein the speech synthesizer audibly broadcasts textual display information and the Braille display outputs non-textual information.

United States Patent No. US 6345389 B1 entitled "Interactive Television System and Method for Converting Non-Textual Information to Textual Information by a Remote Server" having a filing date of 21 October 1998 (Applicant: OpenTV, Inc.) relates to interactive television systems particularly to methods for using a server to convert user-provided information into data format which can be used by an interactive television application. In the US 389 Patent, when a user provides non-textual information to the interactive television system, this information is converted by a server to textual information. Further, the US 389 Patent provides for user's handwriting (non-textual) being digitized (textual) prior to convey the same to the set-top box (where it is inputted to an application).

In an IEEE publication entitled "Synchronizing Audio and Haptic Feedback to Read Webpage" by Sulaiman et. al. (the Sulaiman et. al. publication) provides non-textual analysis. The Sulaiman et. al. publication provides a study that examines the significance of text attributes to support web accessibility and improve understanding of the content. Further, synchronization of audio and haptic is provided to read webpage and represent text attributes while providing limitation of non-textual information being analyse (i.e. header, sub-header, bold, underline and italic).

As different visually impaired users have different needs of non-textual information and non-textual information available is currently limited to what the API can support, there is a need to provide a screen reader for interpreting non-textual information by utilization of visual and system interpreters. SUMMARY OF INVENTION

The present invention relates to a system and method for interpreting non-textual information through a screen reader in communication with a machine readable framework. In particular, the present invention provides a screen reader for interpreting non-textual information by utilization of visual and system interpreters.

One aspect of the present invention provides a system (100) for interpreting non-textual information through a screen reader (101 a) in communication with a machine readable framework (101 ). The screen reader (101 a) comprising at least one Main Controller (1 10) for executing query instruction; at least one Listener (102) for receiving at least one user input event from a machine readable framework (101 ), retrieving user interface element object from said machine readable framework (101 ) and forwarding said element object to the at least one Main Controller (1 10); at least one Impairment Adjuster (104) for including and excluding non-textual attributes that needs to be obtained for each instance of use; at least one Element Analyzer (103) for receiving user interface element object, retrieving a list of attributes from at least one Impairment Adjuster (104) for extraction and filtering said list of attributes and extracting attributes according to type of element object; at least one set of Interpreter Modules (105, 106) for interpreting non-textual information of said screen reader (101 a) for visually impaired; at least one Outputer (107) for outputting information in a non-textual information package based on pre-defined configuration to a single or plurality of devices and presenting contextual command query results; at least one Archiver (108) for storing element object and NTI package with a time stamp and for retrieving context-sensitive information; and at least one Contextual Command Module (109) for receiving and processing contextual command from users by converting speech into text command. The at least one set of Interpreter Modules (105, 106) comprising at least one System Interpreter (105) and at least one Visual Interpreter (106) for interpreting non-textual information of said screen reader for visually impaired. Another aspect of the invention provides that the at least one System Interpreter (105) further comprising Dynamic Link Library; said Dynamic Link Library is a set of machine readable instructions wherein said Dynamic Link Library is used to hold a plurality of machine readable instructions that represent objects based on instructions. Still another aspect of the invention provides that the at least one Visual Interpreter (106) further comprising at least one Image Processing Module. The said Image Processing Module comprises of at least one Image Processor for processing image of the User Interface; at least one Optical Character Recognizer for converting images into text; and at least one Image Recognizer for determining similarities between signatures and User Interface element.

Another aspect of the invention provides a method (200) for interpreting non-textual information through a screen reader in communication with a machine readable framework. The method (200) comprising steps of connecting at least one Listener to user input events (202); receiving at least one user input event by said Listener from a machine readable framework (204); retrieving user interface element object from said machine readable framework and forwarding said element object by said Listener to at least one Main Controller (1 10), said element object is further forwarded to at least one Element Analyzer (206); retrieving a list of attributes to be extracted by said Element Analyzer from at least one Impairment Adjuster (208); filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer (210); triggering Interpreter Modules to retrieve attributes of said Interpreter Modules to obtain details of attributes (212); combining results obtained upon retrieving all attributes by creating non-textual information package and further forwarding said non-textual information package to the Main Controller (214); and displaying information of the non-textual information package and element object based on pre-defined configuration to at least one device (216) and archiving simultaneously said non-textual information package and element object. The step for triggering Interpreter Modules to retrieve attributes of said Interpreter Modules to obtain details of attributes (212) further comprising steps of (400) obtaining attribute object from list of attributes (402); obtaining list of retrieval methods inside attribute object (404); obtaining retrieval methods with highest priority retrieval methods (406); executing retrieval methods of same priority with Interpreter Modules (408); receiving results for each retrieval methods (410); determining if at least one result obtained is valid (412); and saving result into non-textual information package if at least one result is found to be valid (414); else determining whether all retrieval methods are executed if there is no result found to be valid (416) and obtain next highest priority retrieval method (418) and iterating steps 408 onwards until at least one valid result is obtained. A further aspect of the invention provides that the list of attributes includes non-textual details that need to be extracted. Yet another aspect of the invention provides that the step for filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer (210) further comprises steps of (300) obtaining a first attribute from said Impairment Adjuster (302); retrieving list of element signatures of attribute (304); obtaining a first element signature (308); determining if elements object matches said element signature (310); if element object matches said first element signature (310); determine if it is a last element signature(314); and storing attribute into filtered list if it is a last element signature (318); else obtaining next element signature (316) and iterate step 310 if element object does not match said first element signature (310);determining if it is a last attribute in list (312); and end process if it is the last attribute in the list (306); else obtaining next attribute (306) and iterate step 304 onwards.

Still another aspect of the invention provides that the step of triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes (212) further comprising steps of searching file or folder by name in designated directories based on input of element name (602); retrieving file type for each possible file location (604); saving result for next similar attribute (606); and determining if conflicting results occurs based on output obtained (606); generating conflicting result report if conflicting results is observed (612); else generating result report if no conflicting results (610).

Another aspect of the invention provides that the method further comprising converting speech into text command through at least one Contextual Command Module which comprises steps of (700) receiving contextual command from users (702); converting said contextual command into query (704); searching in Archiver through query (706); determine if results obtained from the search is valid (708); displaying results in a predefined manner if results are valid (712); else displaying fail message if results are invalid (710). A further aspect of the invention provides that the step for archiving simultaneously said non-textual information package and element object further comprising storing said nontextual information with time stamp as archiving is used for context-sensitive information retrieval. The present invention consists of features and a combination of parts hereinafter fully described and illustrated in the accompanying drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention.

BRIEF DESCRIPTION OF ACCOMPANYING DRAWINGS

To further clarify various aspects of some embodiments of the present invention, a more particular description of the invention will be rendered by references to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the accompanying drawings in which: FIG. 1 .0 illustrates a general architecture of the system of the present invention.

FIG. 1 .0a illustrates the non-textual information screen reader of the present invention.

FIG. 2.0 is a flowchart illustrating the general methodology of the present invention.

FIG. 3.0 is a flowchart illustrating the steps for filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer.

FIG. 4.0 is a flowchart illustrating the steps for triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes.

FIG. 5.0 illustrates an example of a system diagram of Visual Interpreter. FIG. 6.0 is a flowchart illustrating further steps for triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes.

FIG. 7.0 is a flowchart illustrating the steps for converting speech into text command through at least one Contextual Command Module. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides a system and method for interpreting non-textual information through a screen reader in communication with a machine readable framework. In particular, the present invention provides a screen reader for interpreting non-textual information by utilization of visual and system interpreters whereby the screen reader provides for vision-impaired users. Hereinafter, this specification will describe the present invention according to the preferred embodiments. It is to be understood that limiting the description to the preferred embodiments of the invention is merely to facilitate discussion of the present invention and it is envisioned without departing from the scope of the appended claims.

The present invention provides interpreter modules for vision-impaired users together with an element analyzer module to identify and collect non-textual information of the user interface, Ul elements that users interact with. File information, table layout, and text pattern are examples of non-textual information that can be collected. This information can then be output in pre-configured manner and be archived for contextual command purposes. FIG. 1 .0 illustrates a general architecture of the system of the present invention and FIG. 1 .0a illustrates the non-textual information screen reader of the present invention. The system of the present invention for interpreting non-textual information through a screen reader (101 a) in communication with a machine readable framework (101 ), the screen reader (101 a) comprising at least one Main Controller (1 10) for executing query instruction; at least one Listener (102) for receiving at least one user input event from a machine readable framework, retrieving user interface element object from said machine readable framework and forwarding said element object to the at least one Main Controller (1 10); at least one Impairment Adjuster (104) for including and excluding non-textual attributes that needs to be obtained for each instance of use; at least one Element Analyzer (103) for receiving user interface element object, retrieving a list of attributes from at least one Impairment Adjuster (104) for extraction and filtering said list of attributes and extracting attributes according to type of element object; at least one set of Interpreter Modules (105, 106) for interpreting non-textual information of said screen reader (101 a) for visually impaired; at least one Outputer (107) for outputting information in a non-textual information package based on pre-defined configuration to a single or plurality of devices and presenting contextual command query results; at least one Archiver (108) for storing element object and NTI package with a time stamp and for retrieving context-sensitive information; and at least one Contextual Command Module (109) for receiving and processing contextual command from users by converting speech into text command. The at least one set of Interpreter Modules (105, 106) comprising at least one System Interpreter (105) and at least one Visual Interpreter (106) for interpreting non-textual information of said screen reader for visually impaired. The at least one System Interpreter (105) further comprising Dynamic Link Library; said Dynamic Link Library is a set of machine readable instructions wherein said Dynamic Link Library is used to hold a plurality of machine readable instructions that represent objects based on instructions. The at least one Visual Interpreter (106) is made out of machine readable instructions for classifying a plurality of items which includes Desktopltem, StartMenultem; Taskbar Item; ConflictReport; Signature; SystemAPI; and Result. The at least one Visual Interpreter (106) further comprising at least one Image Processing Module; said Image Processing Module further comprises of at least one Image Processor for processing image of the User Interface; at least one Optical Character Recognizer for converting images into text; and at least one Image Recognizer such as hidden Markov model for determining similarities between signatures and User Interface element. As illustrated in FIG. 1 .0 and FIG. 1 .0a, the Main Controller (1 10) that resides within the screen reader having the Listener (102) that connects to user inputs events. User input events include mouse movement and keyboard navigation. The Listener (102) is connected to user input events by using available application programming interfaces, (APIs) from the main framework (101 ). When a desired event is triggered, the Listener (102) will receive the event object from the main framework (101 ) and proceed to retrieve the user interface element object from the main framework (101 ) based on event object and forwards the element object to the Main Controller (1 10). Subsequently, the Main Controller (1 10) forwards the element object to Element Analyzer (103). The Element Analyzer (103) retrieves a list of 'relevant non-textual details that needs to be extracted' (i.e. attributes) from the Impairment Adjuster (104).

Impairment Adjuster (104) includes and excludes non-textual attributes' that needs to be obtained for each instance of use. Each attribute will have a default impairment value. Users can easily select the visual impairment condition (e.g., 0, 10, 20 ... 100%) and the Impairment Adjuster will only gather attributes with the appropriate impairment value. Users can also edit to include and exclude the attribute required by adjusting the attribute 'impairment value' individually. For each attributes, the Element Analyzer (103) will evaluate whether the element object matches the compatible element signatures of the attribute. If the element object does not match the element signature or attributes, the attribute cannot be obtained from the current element. Subsequently, the Element Analyzer (103) will create a list of attributes that can be extracted from element object from the matched results. Further, Element Analyzer (103) will obtain retrieval methods for each attributes. Element Analyzer (103) then triggers the appropriate interpreters (105, or 106) to use the retrieval instructions inside each of the retrieval method in order to obtain the attribute detail. Upon retrieving all attributes, the results (non-textual information) are combined into non-textual information (NTI) package and Element Analyzer (103) will forward the NTI package to the Main Controller (1 10) and the Main Controller (1 10) will forward the element object and NTI package to the Outputer (107). The Outputer (107) will output the information in the NTI package based on pre-defined configuration to a single or plurality of devices. Further, the Main Controller (1 10) will also forward the element object and NTI package to Archiver (108) simultaneously. Archiver (108) will store each of the objects with a time stamp. Archiver (108) is used for context-sensitive information retrieval.

Contextual Command Module (109) could be enabled from the start to receive and process contextual command from users. Contextual Command Module (109) can be connected to a microphone to listen to the appropriate command. Once a valid command is received, Contextual Command Module (109) computes and sends query instructions to Main Controller (1 10). Main Controller (1 10) executes the query instructions to query the Archiver (108) database. The Main Controller (1 10) sends the contextual command query results to Outputer (107). Outputer (107) presents the contextual command query results in a pre-defined manner.

FIG. 2.0 illustrates a general methodology of the present invention for interpreting nontextual information through a screen reader in communication with a machine readable framework. As illustrated in FIG. 2.0, the method is initiated by first connecting at least one Listener to user input events (202) and said Listener receives at least one user input event from a machine readable framework (204). Upon receiving at least one user input event, said Listener retrieves user interface element object from said machine readable framework and forwards said element object to at least one Main Controller. The element object is further forwarded to at least one Element Analyzer (206) and said Element Analyzer retrieves a list of attributes to be extracted from at least one Impairment Adjuster (208). Subsequently, Element Analyzer (210) filters said list of attributes and extract attributes according to type of element object. Thereafter, Interpreter Modules are triggered to retrieve attributes of said Interpreter Modules to obtain details of attributes (212). Upon retrieving all attributes, the results (non-textual information) are combined by creating nontextual information, NTI package and Element Analyzer further forwards said non-textual information package to the Main Controller (214). The Outputer displays information of the non-textual information, NTI package and element object based on pre-defined configuration to at least one device (216) and archiving simultaneously said non-textual information package and element object. Archiving simultaneously of said non-textual information package and element object includes storing said non-textual information with time stamp as archiving is used for context-sensitive information retrieval.

Reference is now made to FIG. 3.0 which illustrates the steps for filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer. In filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer (210), a first attribute is obtained from said Impairment Adjuster (302). Thereafter, list of element signatures of attribute is retrieved (304). A first element signature is obtained (308) and it is further determined if elements object matches said element signature (310). If element object matches said first element signature (310); it is determined if it is a last element signature (314); and attribute is stored into filtered list if it is a last element signature (318). Otherwise, next element signature is obtained and (316) and iterate step 310. Alternatively, if element object does not match said first element signature (310); it is determined if it is a last attribute in the list (312); and process ends if it is the last attribute in the list (306). Otherwise, next attribute is obtained (306) and iterate step 304 onwards.

Reference is now made to FIG. 4.0 which illustrates the steps for triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes. In interpreting non-textual information, attribute object is first obtained from list of attributes (402). List of attributes includes non- textual details that need to be extracted. There are various retrieval methods available. Therefore, the best retrieval method is to be selected. A list of retrieval methods inside attribute object is obtained (404) and retrieval methods with highest priority retrieval methods is obtained (406) followed by executing retrieval methods of same priority with Interpreter Modules (408). Results for each retrieval methods is obtained (410) and it is determined if at least one result obtained is valid (412). The said result obtained is saved into non-textual information package if at least one result is found to be valid (414). Otherwise, it is determined whether all retrieval methods are executed if there is no result found to be valid (416) and obtain next highest priority retrieval method (418) and iterate steps 408 onwards until at least one valid result is obtained. FIG. 5.0 illustrates an example of a system diagram of Visual Interpreter. As illustrated in FIG. 5.0, the current element object is a desktop item while the current attribute that is intended for retrieval is "IsFolder" and the information retrieval method is called "Standard Folder Visual Cue". Element Analyzer (103) sends the instructions to execute retrieval method "Standard Folder Visual Cues" to Visual Interpreter (106). In 501 , Visual Interpreter (106) takes a screenshot of the "desktop item" element object. The area and screen information is required to take screenshot provided by element object. In 502, Visual Interpreter (106) applies mask to screenshot while in 503, Visual Interpreter (106) applies matching algorithm to screenshot. If matching probability is over a certain value (e.g., 80%), a positive result is obtained (IsFolder = true) and the current element object is a desktop item while the current attribute retrieved is "IsShortcut" and the information retrieval method is named "Standard Shortcut Visual Cue". The process in 504, 505 and 506 is the same as 501 , 502 and 503 except that the Visual Interpreter is searching for arrow visual cue instead of folder visual cue. Reference is made to FIG. 6.0 which illustrates the further steps for triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes. As illustrated in FIG. 6.0, file or folder is searched by name in designated directories based on input of element name (602) and file type for each possible file location is retrieved (604). The result for next similar attribute is saved (606) and it is determined if conflicting results occurs based on output obtained (608). Conflicting result report is generated if conflicting results is observed (612). Otherwise, result report is generated if no conflicting results (610).

Reference is now made to FIG. 7.0 which illustrates the steps for converting speech into text command through at least one Contextual Command Module. In converting speech into text command, contextual command is received from users (702) and said contextual command is converted into query (704). It is searched in Archiver through query (706) and it is determined if results obtained from the search is valid (708). Results are displayed in a pre-defined manner if results are valid (712). Otherwise, fail message is displayed if results are invalid (710). The screen reader of the present invention which is developed based on machine readable instructions by overcoming the limitation of the current API, Application Programming Interface. In short, the screen reader is able to interpret non-textual information by utilization of interpreter modules whereby the interpreter module, the System Interpreter is coupled to an Impairment Adjuster. The Impairment Adjuster used to include and exclude non-textual attributes that needs to be obtained for each instance of use. Each attribute will have a default impairment value whereby users can easily select their visual impairment condition (e.g., 0, 10, 20 ... 100%). The screen reader will only collect attributes with the appropriate impairment value which allows users to edit to include and exclude the required attributes by adjusting the impairment value of the attribute individually.

Unless the context requires otherwise or specifically stated to the contrary, integers, steps or elements of the invention recited herein as singular integers, steps or elements clearly encompass both singular and plural forms of the recited integers, steps or elements.

Throughout this specification, unless the context requires otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated step or element or integer or group of steps or elements or integers, but not the exclusion of any other step or element or integer or group of steps, elements or integers. Thus, in the context of this specification, the term "comprising" is used in an inclusive sense and thus should be understood as meaning "including principally, but not necessarily solely".

Claims

1 . A system (100) for interpreting non-textual information through a screen reader (101 a) in communication with a machine readable framework (101 ), the screen reader (101 a) comprising:

at least one Main Controller (1 10) for executing query instruction;

at least one Listener (102) for receiving at least one user input event and retrieving user interface element object from said machine readable framework (101 ), and forwarding said element object to the at least one Main Controller (1 10);

at least one Impairment Adjuster (104) for including and excluding nontextual attributes that needs to be obtained for each instance of use;

at least one Element Analyzer (103) for receiving user interface element object, retrieving a list of attributes from the at least one Impairment Adjuster (104) for extraction and filtering said list of attributes and extracting attributes according to type of element object;

at least one set of Interpreter Modules (105, 106) for interpreting nontextual information of said screen reader (101 a) for visually impaired;

at least one Outputer (107) for outputting information in a non textual information, package based on pre-defined configuration to a single or plurality of devices and presenting contextual command query results;

at least one Archiver (108) for storing element object and NTI package with a time stamp and for retrieving context-sensitive information; and at least one Contextual Command Module(109) for receiving and processing contextual command from users by converting speech into text command; characterized in that

the at least one set of Interpreter Modules (105, 106) comprising at least one System Interpreter (105) and at least one Visual Interpreter (106) for interpreting non-textual information of said screen reader (101 a) for visually impaired.

2. The system (100) according to Claim 1 , wherein the at least one System Interpreter (105) further comprising Dynamic Link Library to hold a plurality of machine readable instructions that represent objects based on instructions.

The system (100) according to Claim 2, wherein the Dynamic Link Library is a set of machine readable instructions.

The system (100) according to Claim 1 , wherein the at least one Visual Interpreter (106) further comprising at least one Image Processing Module comprises of:

at least one Image Processor for processing image of a User Interface; at least one Optical Character Recognizer for converting images into text; and

at least one Image Recognizer for determining similarities between signatures and User Interface element.

A method (200) for interpreting non-textual information through a screen reader in communication with a machine readable framework, the method (200) comprising steps of:

connecting at least one Listener to user input events (202);

receiving at least one user input event by said Listener from a machine readable framework (204);

retrieving user interface element object from said machine readable framework and forwarding said element object by said Listener to at least one Main Controller, said element object is further forwarded to at least one Element Analyzer (206);

retrieving a list of attributes to be extracted by said Element Analyzer from at least one Impairment Adjuster (208);

filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer (210);

triggering Interpreter Modules to retrieve attributes of said Interpreter Modules to obtain details of attributes (212);

combining results obtained upon retrieving all attributes by creating nontextual information package and further forwarding said non-textual information package to the Main Controller (214); and

displaying information of the non-textual information package and element object based on pre-defined configuration to at least one device (216) and archiving simultaneously said non-textual information package and element object;

characterized in that triggering Interpreter Modules to retrieve attributes of said Interpreter Modules to obtain details of attributes (212) further comprising steps of (400):

obtaining attribute object from list of attributes (402); obtaining list of retrieval methods inside attribute object (404);

obtaining retrieval methods with highest priority retrieval methods (406);

executing retrieval methods of same priority with Interpreter Modules (408);

receiving results for each retrieval methods (410);

determining if at least one result obtained is valid (412); and saving result into non-textual information package if at least one result is found to be valid (414); else

determining whether all retrieval methods are executed if there is no result found to be valid (416) and obtain next highest priority retrieval method (418) and iterating steps 408 onwards until at least one valid result is obtained.

The method (200) according to Claim 5, wherein the list of attributes includes nontextual details that needs to be extracted.

The method (200) according to Claim 5, wherein filtering said list of attributes and extracting attributes according to type of element object by said Element Analyzer (210) further comprises steps of (300):

obtaining a first attribute from said Impairment Adjuster (302);

retrieving list of element signatures of attribute (304);

obtaining a first element signature (308);

determining if elements object matches said element signature (310);

if element object matches said first element signature ;

determine if it is a last element signature(314); and

storing attribute into filtered list if it is a last element signature (318); else obtaining next element signature (316) and iterate step 310; if element object does not match said first element signature ;

determining if it is a last attribute in list (312); and

end process if it is the last attribute in the list (306); else obtaining next attribute (306) and iterate step 304 onwards.

8. The method (200) according to Claim 5, wherein triggering Interpreter Modules to retrieve attributes by using retrieval instructions in retrieval method of said Interpreter Modules to obtain details of attributes (212) further comprising steps of (600):

searching file or folder by name in designated directories based on input of element name (602);

retrieving file type for each possible file location (604);

saving result for next similar attribute (606); and

determining if conflicting results occurs based on output obtained (606); generating conflicting result report if conflicting results is observed (612); else

generating result report if no conflicting results (610).

9. The method (200) according to Claim 5, further comprising converting speech into text command through at least one Contextual Command Module which comprises steps of (700):

receiving contextual command from users (702);

converting said contextual command into query (704);

searching in Archiver through query (706);

determine if results obtained from the search is valid (708);

displaying results in a pre-defined manner if results are valid (712); else displaying fail message if results are invalid (710).

10. The method according to Claim 5, wherein archiving simultaneously said non- textual information package and element object further comprising storing said nontextual information with time stamp as archiving is used for context-sensitive information retrieval.