US8943468B2 - Wireframe recognition and analysis engine - Google Patents
Wireframe recognition and analysis engine Download PDFInfo
- Publication number
- US8943468B2 US8943468B2 US13/597,329 US201213597329A US8943468B2 US 8943468 B2 US8943468 B2 US 8943468B2 US 201213597329 A US201213597329 A US 201213597329A US 8943468 B2 US8943468 B2 US 8943468B2
- Authority
- US
- United States
- Prior art keywords
- wireframe
- features
- components
- images
- software application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000004458 analytical method Methods 0.000 title abstract description 8
- 238000000034 method Methods 0.000 claims abstract description 64
- 238000013461 design Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 8
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 238000012015 optical character recognition Methods 0.000 claims description 6
- 230000009471 action Effects 0.000 description 11
- 239000003550 marker Substances 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 101100277337 Arabidopsis thaliana DDM1 gene Proteins 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 3
- 230000002730 additional effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 210000001072 colon Anatomy 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011960 computer-aided design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
Definitions
- the present invention generally relates to the field of software development, and more particularly, relates to a method and system for dynamically producing source code for a software application by means of recognition and analysis of the wireframes depicting the software application.
- wireframes depicting the software application that is to be constructed.
- Wireframes can contain information regarding the layout of the software application's user interface, navigational structure, and available functionality.
- UI user interface
- UX user experience
- the present invention comprises of a computer-implemented method, a non-transitory computer readable storage medium, and a computer system for dynamically producing source code for a software application from a set of wireframes.
- FIG. 1 is a high-level block diagram illustrating a system for dynamically producing source code for a software application from a set of wireframe images, according to one embodiment.
- FIG. 2 is a flowchart illustrating a method for dynamically producing source code for a software application from a set of wireframe images, according to one embodiment.
- FIG. 3 is a flowchart illustrating a method of feature extraction performed on each wireframe image in an input set, according to one embodiment.
- FIG. 4 is a flowchart illustrating a method to formulate a collection of wireframe components from the extracted features found in a set of wireframes, according to one embodiment.
- FIG. 5 is a flowchart illustrating a method to fabricate a software application from a collection of wireframe components, according to one embodiment.
- FIG. 6 is a high-level block diagram illustrating an example of a computing system for use as a wireframe recognition and analysis engine (WRAE) server, according to one embodiment.
- WRAE wireframe recognition and analysis engine
- FIG. 7 is an illustration of a wireframe depicting a software application, according to one embodiment.
- FIG. 8 is an example of a source template used to generate source code for a drop down menu, according to one embodiment.
- FIG. 9A is an exemplary class declaration for a wireframe component object, according to one embodiment.
- FIG. 9B is an exemplary class declaration for a drop down menu wireframe component object, according to one embodiment.
- FIG. 9C is an example of a serialized drop down menu wireframe component object, according to one embodiment.
- FIG. 10 is exemplary output source code generated by a template engine for a drop down menu, according to one embodiment.
- FIG. 11B is an example of a drop down menu rule defined in the knowledge base, according to one embodiment.
- FIG. 12A is an example of a wireframe annotation marker rule defined in the knowledge base, according to one embodiment.
- FIG. 1 is a high-level block diagram illustrating a system 100 for dynamically producing source code for a software application from a set of wireframes, according to one embodiment of the present invention.
- the system 100 includes the wireframe recognition and analysis engine 108 (WRAE) in communication with an end user's device 118 , which could be a smartphone, tablet, or personal computer via a network 116 (e.g., intranet or Internet).
- the end user's device communicates with the wireframe recognition and analysis engine through a TCP/IP connection over the network 116 .
- the wireframe recognition and analysis engine 108 is referred to as the WRAE.
- the WRAE 108 is coupled to a library of models 102 , a knowledge base 104 , and a data store 106 .
- the library of models 102 , the knowledge base 104 , and the data store 106 are flat-file databases.
- a wireframe image can be created by scanning a sheet of paper containing a wireframe with a desktop (or flatbed) scanner, taking a photo of a wireframe with a camera or mobile device (e.g., smartphone or tablet), or using a computer aided design tool to draw a wireframe and save the wireframe as an image file.
- the input, a set of wireframe images depicting the software application to be fabricated, is sent via the network 116 from an end user's device 118 to the WRAE 108 for processing.
- a set of user configurable parameters is transmitted along with the input set of wireframe images. These parameters include, a parameter to select the target platform of the software application.
- target platform refers to the device and operating system (including the operating system's major release version) that the finished software application is expected to run on.
- a parameter exists to specify the user's e-mail address, which will be used to alert the user when the source code for the software application has been generated.
- An “assets” directory for the job is dynamically created as a subdirectory whose parent directory is the current job id.
- the uploaded wireframe image files are moved over to the assets directory.
- the following assets directory will be created on the WRAE server:/wrae_jobs/bm93jzrfn/assets.
- An additional directory is created to serve as the data store 106 , where the data store 106 is unique to a user session and applies only to the input set of wireframes for a particular job id.
- This directory will be named “data_store” and will exist as a subdirectory of the directory named with the current job id.
- data_store For example, for job id “bm93jzrfn”, the following data store 106 directory will be created on the WRAE server:/wrae_jobs/bm93jzrfn/data_store.
- a “source code output” directory for the job is dynamically created as a subdirectory whose parent directory is the current job id. All source code produced by the WRAE system for the current job id is stored at this location. For example, for job id “bm93jzrfn”, the following output source code directory will be created on the WRAE server:/wrae_jobs/bm93jzrfn/output
- the term “module” refers to computer program instructions and/or other logic used to provide the specified functionality.
- a module can be implemented in software, hardware, and/or firmware.
- the WRAE 108 consists of a discovery module 110 for performing feature extraction on each wireframe contained in the set of wireframes, an inference module 112 for formulating the collection of wireframe components, and a fabrication module 114 for dynamically producing the source code for the software application.
- features refers to shapes and text contained in a wireframe.
- a base feature object defines properties shared across all features, such as the feature's screen coordinates.
- a shape feature object extends the base feature object and may provide additional properties such as the type of shape the feature depicts.
- a variety of shapes, such as a rectangle, triangle, circle, etc. can be implemented as subclasses of the base shape object. These individual shape feature objects would have properties that were custom to the type of shape they represent.
- a text object extends the base feature object and has a text value property which stores the value of the text string for the text feature it represents. All feature objects have a rectangle bounding box property, which represents a bounding box rectangle that bounds the feature.
- FIG. 7 is an illustration of a wireframe depicting a software application.
- FIG. 7 comprises a wireframe set used as input to the WRAE.
- the term “wireframe component” refers to an element contained in a wireframe.
- Wireframe components can be classified into two distinct types. The first type, a user interface wireframe component, depicts a user interface element (e.g., text field, drop down menu, button) which will be represented in the finished software application's graphical user interface. The second type, a structural wireframe component, depicts an element that is specific to the wireframe's structure (e.g., the title of the wireframe, the page number of the wireframe, wireframe annotation markers).
- Structural wireframe components are not displayed in the finished software application.
- the wireframe components depicted in illustration 700 include the following user interface wireframe components: application window frame 714 , text label 708 , and drop down menu field 710 .
- the following structural wireframe components are depicted in illustration 700 : the title of the wireframe 702 , user interface container 704 , wireframe annotations container 706 , a wireframe annotation marker 712 , and the page number of the wireframe 716 .
- wireframe components are also represented in the WRAE as objects.
- a base wireframe component object defines properties shared across all wireframe components such as the wireframe component's screen coordinates.
- a wireframe component object can have a type property specifying if the wireframe is of type user interface or structural.
- a variety of wireframe component objects, such as a drop down menu, text label, button, etc. can be implemented as a subclass of the base wireframe component object.
- a wireframe component object representing a drop down menu can have the following properties: width, height, key list (a list of items displayed in the drop down menu's list), and default key (the item that should be displayed by default in the drop down menu).
- FIG. 2 is a flow chart of a method 200 for dynamically producing source code for a software application from a set of wireframe images, in accordance with embodiments of the present invention.
- the method 200 begins with the WRAE 108 receiving a set of wireframes via a network from an end user's device 202 .
- the three aforementioned modules 110 , 112 , 114 of the WRAE 108 are self-contained and each module is executed in the following sequential fashion: first, the discovery module 110 is executed corresponding to step 204 , where feature extraction is performed on each wireframe in the input set and the extracted features are stored in the data store 106 . Second, the inference module 112 is executed corresponding to step 206 , where the collection of wireframe components is formulated.
- the fabrication module 114 is executed corresponding to step 208 , where the software application's source code is dynamically produced using the collection of wireframe components.
- the output of the fabrication module 114 is the source code for the software application which is delivered to an end users device in step 210 via a network 116 .
- FIG. 3 is a flow chart illustrating a method 300 of feature extraction performed by the discovery module 110 for each wireframe in the input set according to one embodiment. Other embodiments may perform the steps in different orders and may include different and/or additional steps.
- the discovery module 110 is responsible for identifying and extracting features from the input set of wireframe images.
- the OpenCV Library (available at http://opencv.willowgarage.com/) is used by the discovery module for all computer vision related needs.
- the library of models 102 is pre-populated with data obtained from a training set.
- Each record in the library of models 102 consists of the name of the shape being described, its computed contour moments, and a serialized contour object representing the shape.
- the system can be trained to identify common shapes found in wireframe images such as triangles, rectangles, and circles. The placement and arrangement of these features are used in the inference module 112 to recognize structural and user interface wireframe components.
- Method 300 corresponds to step 204 in method 200 .
- the data store 106 is empty.
- the Canny Edge Detector is applied on the wireframe to determine edges and form contours.
- the Canny algorithm forms contours through the application of hysteresis thresholding.
- the considered embodiment of the invention conducts edge detection by passing each input wireframe image through OpenCV's cvCanny( ) function to create an output binary image for each wireframe image in the input set.
- Each of the output binary images are passed to OpenCV's cvFindContours( ) function, which creates contours from the binary images.
- the contour moments are computed for each contour found.
- a common way to describe a contour is by computing its contour moments.
- the calculated contour moments provide a means to compare contours.
- the computed contour moments can include the central moments, normalized moments, and rotation invariant moments such as the Hu invariant moments.
- Each contour found is passed to OpenCV's cvContourMoments( ) function which computes the contour's moments.
- the central moments can be obtained by using OpenCV's cvGetCentralMoment( ) function. Normalized moments can be obtained by using OpenCV's cvGetNormalizedCentralMoment( ) function.
- shapes are extracted from the wireframe by comparing each contour's computed moments with those contained in the library of models 102 and finding the best match. This can be accomplished by comparing each found contour with the contours defined in the library of models 102 using OpenCV's cvMatchShapes( ) function. A contour defined in the library of models 102 would first have to be deserialized before performing the matching test using OpenCV's cvMatchShapes( ) function. Upon successfully identifying a shape feature in a wireframe image, an instance, of a shape feature object is instantiated for the shape that is found. The factory design pattern can be utilized by the WRAE system to dynamically instantiate the shape feature object at run-time.
- the newly instantiated shape feature object is initialized with the screen coordinates, width, and height properties obtained from the matched contour.
- the shape feature object's bounding box rectangle property is initialized to the rectangle bounding box bounding the matched contour.
- OpenCV's cvMinAreaRect2( ) function can be used to find the rectangle that bounds the matched contour.
- This bounding box rectangle is represented using OpenCV's CVBox2D data type.
- an e-mail is generated by the system and sent to the system administrator(s).
- the e-mail would have the coordinates in the image where the undefined shape is located in the message body and would have an attachment containing the wireframe image with the unidentified shape.
- the system administrator(s) may then choose to identify the shape, compute its contour moments and add it to the training set by populating it in the library of models 102 resulting in the system being trained to recognize the hitherto unknown shape.
- step 308 text content contained in the wireframe is extracted using an optical character recognition (OCR) system, such as OCRopus (available at http://code.google.com/p/ocropus/).
- OCRopus optical character recognition
- Each wireframe image is passed through to the OCR system.
- OCRopus Upon identifying a line of text, a text feature object is instantiated and initialized with the value of the text found, the screen coordinates for the line of text, and the bounding box property is set to the bounding box bounding the text feature object.
- the factory design pattern can be utilized by the WRAE system to instantiate the text feature object at run-time.
- step 310 the features extracted from the wireframe are saved to the data store 106 .
- Each feature object would also contain a property that specified the sheet number that the feature was found on. The value of the sheet number property, begins at 1, and increments by one for each wireframe image in the input set. For example, wireframe image 700 , comprising an input set, would have a sheet number property value equal to 1 when input into the system. If another wireframe image had also been included in the input set, it would have been assigned a sheet number property value equal to 2.
- FIG. 4 is a flow chart illustrating a method 400 performed by the inference module to formulate a collection of wireframe components from the features found in a set of wireframes, according to one embodiment.
- the inference module is responsible for dynamically producing a collection of wireframe component objects that represent the wireframe components that are present in the input set of wireframes.
- Method 400 corresponds to step 206 in method 200 .
- Method 400 implements an inference engine that processes the contents of the data store 106 using a knowledge base 104 of wireframe design rules.
- the knowledge base 104 of wireframe design rules is a repository of rules related to the design of wireframes which provide the WRAE system a means to identify wireframe components from a subset of text and shape feature objects.
- the data store 106 holds text and shape feature objects populated by the discovery module.
- Steps 402 to 408 of method 400 comprise the inference engine's recognize-act cycle.
- step 402 all rules in the knowledge base 104 that satisfy the current contents of the data store 106 are found.
- the system would match feature objects in the data store 106 to the wireframe design rules defined in the knowledge base 104 .
- FIGS. 11A , 11 B, 12 A, 12 B are examples of wireframe design rules used to recognize wireframe components, according to one embodiment.
- Wireframe design rules specify the criteria to identify wireframe components from a subset of text and shape feature objects in the data store 106 . These rules are defined in the knowledge base 104 and are stored in condition-action form, where each rule consists of a condition section and an action section.
- the condition section of a rule specifies a list of conditions that must be met for the rule to be applied.
- the pair of a rule and a subset of matching items from the data store 106 that match the rule is known as an instantiation of the rule.
- the action section of a rule specifies the actions that should be taken when all conditions specified in the condition section of the rule have been met. When a rule is executed, the actions specified in the action section of the rule are executed.
- the subset of matching items from the data store 106 are considered to be instantiation data and are provided as input parameters to a rule when a rule is executed.
- the condition section of the wireframe design rules specify how to recognize a wireframe component by testing for the existence of a particular subset of features from the data store 106 .
- the condition section of a rule also takes into consideration the disposition of a feature to recognize wireframe components.
- disposition of a feature refers to the placement of the feature and the arrangement of the feature in relation to other features found in the data store 106 .
- the bounding box rectangle that bounds the feature object, and is stored within the feature object's rectangle bounding box property is used to ascertain the disposition of a feature.
- the placement of a feature can be determined using the screen coordinates of the rectangle bounding box of that feature.
- the arrangement of the feature in relation to other features can be determined by testing for rectangle containment of the rectangle bounding box of the feature object with the rectangle bounding box of other feature objects in the data store 106 .
- a containment test can be conducted, such as the separating axis test for rectangle containment, to determine if one feature is contained within another feature.
- Rules are represented in the WRAE as objects which subclass a base rule object.
- all rules must implement a testConditions( ) method which represents the condition section of the rule, and an executeActions( ) method, which represents the action section of the rule.
- Rule objects have a property named “instantiation data” which is an associative array that is empty when the rule object is first initialized by the WRAE system.
- the base rule object implements a clearInstantiationData( ) method which removes all elements in the rule object's instantiation data property when called.
- the knowledge base 104 is a directory on the WRAE server, purposed to store serialized rule objects.
- the system administrator(s) implement rule objects to represent the wireframe design rules that need to be supported by the WRAE system.
- the wireframe design rules are defined in the knowledge base 104 by saving the serialized rule objects as flat files in the knowledge base 104 directory.
- each condition specified in the condition section of the rule has a respective if-else conditional construct in the rule object's testConditions( ) method. Should the conditional expression specified in the if portion of the construct evaluate to a boolean value of true, the code specified in the if block of the construct is executed.
- a shape or text feature object from the data store 106 that matches a condition is designated with a unique identifier, known as a tag name, that has a uniqueness at the rule object level.
- Populating a feature in a rule object's instantiation data property involves adding a new element to the array where the key is set to the tag name of the feature object and the value is set to the full path of the serialized feature object file.
- flow of execution continues to the next condition specified in the testConditions( ) method.
- the testConditions( ) method returns a boolean value of true. This informs the WRAE system that all conditions specified by the rule object have been satisfied and the rule object's instantiation data property is populated with the subset of features from the data store 106 that satisfy all conditions in the condition section of the rule.
- the WRAE system adds the rule object to a queue of rule objects that are to be executed. Should the conditional expression specified in the if portion of the construct evaluate to a boolean value of false, the code specified in the else block of the construct is executed. When this occurs, the rule object's clearinstantiationData( ) method is called, clearing all elements in the rule object's instantiation data property. In addition to this, the testConditions( ) method returns a boolean value of false, which informs the WRAE system that not all conditions specified by the rule object have been satisfied and that the rule object should be discarded.
- testConditions( ) method Since the testConditions( ) method will immediately return a boolean value of false upon a failed condition, it is ensured that no subsequent conditions in a rule object's testConditions( ) method are evaluated. Thus, one failed condition in a rule object's testConditions( ) method, disqualifies the rule object from being added to the queue of rule objects to be executed.
- the WRAE system deserializes all rule objects from the knowledge base 104 directory, and determines if each rule's conditions are satisfied by calling the rule object's testConditions( ) method in step 402 .
- the rule object's instantiation data property which is an associative array, will contain the following keys: TF 1 , R 1 , R 2 .
- Each key in the associative array will be paired to a value, which is the full path to the serialized feature object file in the data store 106 directory for the respective key.
- the drop down menu rule shown in FIG. 11B , has a condition section that specifies how to recognize a drop down menu wireframe component from a subset of matching text and shape feature objects from the data store 106 where each feature object is designated with a tag name.
- a triangle shape feature object, designated as T 1 is contained with a rectangle shape feature object, designated as R 1 .
- R 1 represents the rectangle containing the triangle in FIG. 7 .
- R 1 is contained within a rectangle shape feature object, R 2 .
- R 2 's width is at least 7 times greater than that of R 1 .
- R 1 's x-coordinate is at least 1.5 times greater than R 2 's x-coordinate.
- a text feature object, TF 1 is contained within R 2 .
- TF 1 represents the default value that the drop down menu is initialized to. In this case, that would be “Select a color”.
- R 2 is contained within a rectangle shape feature object, designated as R 3 .
- R 3 has the second largest width in the wireframe image, and is contained inside another rectangle shape feature object, designated as R 4 .
- R 3 represents the application window frame 714 shown in FIG. 7 .
- R 4 has the largest width in the wireframe image, and represents the user interface container 704 shown in FIG. 7 .
- the rule object's instantiation data property which is an associative array, will contain the following keys: T 1 , TF 1 , R 1 , R 2 , R 3 , R 4 .
- Each key in the associative array will be paired to a value, which is the full path to the serialized feature object file in the data store 106 directory for the respective key.
- the wireframe annotation marker rule shown in FIG. 12A , has a condition section that specifies how to recognize a wireframe annotation marker wireframe component from a subset of matching text and shape feature objects from the data store 106 where each feature object is designated with a tag name.
- a text feature object, designated as TF 1 is contained within a circle shape feature object, designated as C 1 .
- C 1 is contained within a rectangle shape feature object designated as R 1 .
- R 1 has the second largest width in the wireframe image, and represents the application window frame 714 show in FIG. 7 .
- R 1 is contained within a rectangle shape feature object, designated as R 2 .
- R 2 has the largest width in the wireframe image, and represents the user interface container 704 shown in FIG. 7 .
- the rule object's instantiation data property which is an associative array, will contain the following keys: TF 1 , C 1 R 1 , R 2 . Each key in the associative array, will be paired to a value, which is the full path to the serialized feature object file in the data store 106 directory for the respective key.
- the wireframe annotation rule shown in FIG. 12B , has a condition section that specifies how to recognize a wireframe annotation wireframe component from a subset of matching text and shape feature objects from the data store 106 where each feature object is designated with a tag name.
- a text feature object, designated as TF 1 is contained within a rectangle shape feature object designated as R 1 .
- R 1 represents the wireframe annotation container 706 shown in FIG. 7 .
- R 1 contains a text feature object, designated as TF 2 , that has a text value property of “Wireframe Annotations”.
- TF 1 's text value property should not be equal to “Wireframe Annotations”.
- the rule object's instantiation data property which is an associative array, will contain the following keys: TF 1 , TF 2 , R 1 , R 2 .
- Each key in the associative array will be paired to a value, which is the full path to the serialized feature object file in the data store 106 directory for the respective key.
- this requirement ensures that that wireframe components produced by the inference module are segmented by the sheet number of the wireframe image that the subset of features were found in. This prevents the WRAE system's inference module from erroneously producing wireframe components through the matching of features spread out across multiple wireframe images.
- a wireframe design rule is matched to a subset of features in the data store 106 , each feature in the subset of features have the same sheet number property value.
- the rules to be applied on the rule matchings are selected.
- the rules to be applied on the rule matchings are simply those rule objects that are in the queue of rule objects to be executed. For example, based off the rule matchings found for the input wireframe image 700 in the previous examples, a rule object representing the text label rule shown in FIG. 11A , a rule object representing the drop down menu rule shown in FIG. 11B , a rule object representing the wireframe annotation marker rule shown in FIG. 12A , and a rule object representing the wireframe annotation rule shown in FIG. 12B would be included in the queue of rule objects to be executed.
- the selected rules are executed with instantiation data as parameters. This is accomplished by dequeuing each rule object from the queue of rule objects to be executed, and calling each dequeued rule object's executeActions( ) method.
- the instantiation data for a rule can be accessed from the instantiation data property, an associative array, of the rule object that represents the rule. For a given feature object, referred to by a tag name in the action section of a rule, the respective path to the serialized object file in the data store 106 directory is accessed from the associative array by using the tag name for the feature object as the key.
- the feature object is deserialized and becomes available for use by the rule object's executeActions( ) method.
- a rule object's testConditions( ) method is responsible for populating the contents of the rule object's instantiation data property beforehand in step 402 .
- the instantiation data available to a rule object's executeActions( ) method is dictated by how the rule object's instantiation data property was populated by the testConditions( ) method in step 402 .
- FIGS. 11A , 11 B, 12 A are examples of rules that contain an action section that produces wireframe component objects.
- the action section of these rules specify what type of wireframe component object the system must create, as well as the properties that should be initialized for the newly created wireframe component object.
- the system Upon successfully creating and initializing the wireframe component object, the system saves the wireframe component object to the data store 106 , which involves serializing the wireframe component object and saving it as a flat file in the data store 106 directory.
- the text label rule shown in FIG. 11A , will instantiate a text label wireframe component object when executed.
- the instantiation data would comprise of the text feature object, designated as TF 1 , that was matched in the condition section of the rule.
- TF 1 would be sent as an input parameter to the action section of the text label rule.
- a text label wireframe component object designated as TL 1
- TL 1 will be instantiated with a text value property set to TF 1 's text value property.
- TL 1 will have its screen coordinate properties set to the screen coordinate properties of TF 1 .
- TL 1 will have its width and height properties set to the width and height properties of TF 1 .
- TL 1 is serialized and saved to the data store 106 .
- the drop down menu rule shown in FIG. 11B
- the instantiation data would comprise of the subset of extracted features from the data store 106 that matched the condition section of the rule. In this case, that would consist of four rectangle shape feature objects, one triangle shape feature object, and one text feature object.
- a drop down menu wireframe component object, DDM 1 will be instantiated having its width and height properties set to the width and height properties of the rectangle shape feature object, R 2 .
- DDM 1 's screen coordinate properties would be set to the screen coordinates of R 2 .
- DDM 1 's default value property would be set to the text value property of the text feature object, TF 1 .
- the DDM 1 is serialized and saved to the data store 106 .
- the wireframe annotation marker rule shown in FIG. 12A , will instantiate a wireframe annotation marker wireframe component object when executed.
- the instantiation data would comprise of the subset of extracted features from the data store 106 that matched the condition section of the rule. In this case, that would consist of a text feature object, TF 1 , and a circle shape feature object, C 1 .
- a wireframe annotation marker object, WAM 1 will be instantiated having its width and height properties set to the width and height properties assigned to C 1 .
- WAM 1 's screen coordinate properties will be set to the screen coordinate properties of C 1 .
- WAM 1 's annotation index property will be set to the text value assigned to the text value property of TF 1 .
- the annotation index property is a unique identifier which indicates which wireframe annotation applies to the wireframe annotation marker.
- WAM 1 is serialized and saved to the data store 106 .
- FIG. 12B is an example of a rule that contains an action section that modifies an existing wireframe component in the data store 106 .
- Modification of a wireframe component in the data store 106 involves the use of a wireframe component handler, which is conceptually analogous to an event handler handling user events.
- a wireframe component handler executes a predefined set of instructions upon a user event being triggered
- a wireframe component handler executes a predefined set of instructions when a wireframe annotation rule is executed.
- the name of the handler to be applied on a wireframe component object and the arguments that should be passed to the handler are embedded within a wireframe annotation object's text value property.
- the wireframe annotation rule shown in FIG. 12B , will apply changes to an existing wireframe component object that is referenced by a wireframe annotation marker when executed.
- the instantiation data would comprise of the subset of extracted features from the data store 106 that matched the condition section of the rule.
- Instantiation data is accessed using the rule object's instantiation data property. In this case, that would consist of two text feature objects and a rectangle feature object.
- the text value property of TF 1 is parsed to retrieve the annotation reference index, which will be used to identify the respective wireframe annotation marker object, WAM 1 .
- a search is conducted through the data store 106 to find the wireframe component object, WCO 1 , that is closest to WAM 1 .
- This search is performed by iterating through all wireframe component objects in the data store 106 and calculating the distance for each wireframe component object's screen coordinates to those screen coordinates assigned to WAM 1 .
- the appropriate handler to apply on WCO 1 is determined by parsing the text value property of TF 1 . By splitting TF 1 's text value property using the colon character as a delimiter, the name of the handler is ascertained to be “Drop Down Menu List” and the arguments passed to the handler are ascertained to be “Blue, Green, Orange, Red, Yellow”.
- the “Drop Down Menu List” is a wireframe component handler that is used to populate a drop down menu object's key list property, an array, with the arguments provided to the handler. Finally, the handler is applied to WCO 1 with the arguments resulting in the drop down menu object's key list property being populated with the list of values specified in the wireframe annotation.
- the method includes determining, in step 408 , whether the data store 106 has arrived at a quiescent state. If the data store 106 has not arrived at a quiescent state, the method returns to step 402 . If the data store 106 has arrived at a quiescent state, the method ends.
- the data store 106 has arrived at a quiescent state. Once the quiescent state is reached, the list of wireframe components that exist in the data store 106 are considered to be the collection of wireframe components that are used as input to the fabrication module. Other embodiments of the invention may use different criteria for specifying when the data store 106 has reached a quiescent state.
- FIG. 5 is a flow chart illustrating a method 500 performed by the fabrication module to dynamically produce source code from a collection of wireframe components. Method 500 corresponds to step 208 in method 200 .
- a set of source templates is selected based on the target platform.
- Source templates are platform-specific, as the code that is specified within them is used to generate source code for a particular target platform.
- the value for the target platform is obtained from the user-configured option specifying the target platform.
- all source templates may be stored at the following path:/wrae_templates.
- Source templates for a desktop Mac OS X version 10.7 application may reside at the following path:/wrae_templates/mac-os-x_v10.7.
- Source templates for an iOS version 6 application for an iPhone may reside at the following path:/wrae_templates/iphone-ios_v6.
- the user-configured option specifying the target platform determines which source template directory to use for generating source code.
- FIG. 8 is an example of a source template implemented in the PHP language as a shell script.
- the file name for the source template is “DropDownMenuTemplate” (since it is an executable PHP shell script, it doesn't require a “.php” extension).
- This source template produces source code in the Objective-C language for a drop down menu to be used in a Mac OS X desktop application.
- the path to a serialized wireframe component object file is provided as a command line argument to the “DropDownMenuTemplate” shell script.
- the source template deserializes the serialized wireframe component object file's contents, and the wireframe component object's properties are used in conjunction with the code specified in the source template to produce Objective-C source code for a drop down menu when processed through a template engine.
- the PHP interpreter's built-in template engine is used for processing the source template.
- each wireframe component from the collection of wireframe components is associated to its corresponding source template.
- Each wireframe component that exists in the data store 106 is deserialized and the wireframe's classname is obtained.
- the source template to be used on a particular wireframe component is derived from the class name of the wireframe component appended with the string “Template”. For example, if the class name of the wireframe component was “DropDownMenu”, the associated source template's file name would be “DropDownMenuTemplate”.
- a validation is performed to verify that the source template file exists and that it is an executable file. If the source template file exists and the file is executable, the wireframe component is associated with the source template file. If these conditions are not met, the wireframe component is discarded.
- FIG. 9A is an exemplary class declaration for a wireframe component object implemented in the PHP language with the file name “WireframeComponent.php”, according to one embodiment.
- FIG. 9B is an exemplary class declaration for a drop down menu wireframe component object implemented in the PHP language with the file name of “DropDownMenu.php”, according to one embodiment.
- the drop down menu class is a subclass of the wireframe component class shown in FIG. 9A .
- FIG. 9C is an example of a serialized drop down menu wireframe component object that exists in the data store 106 .
- source code is produced using a template engine to process each source template associated with a wireframe component.
- PHP's built-in template engine is used to process the code specified in the template in conjunction with the source template's associated serialized wireframe component.
- FIG. 11 is the exemplary output, according to one embodiment, after processing the code specified in the source template shown in FIG. 8 through a template engine in conjunction with the serialized wireframe component object shown in FIG. 9C .
- step 508 the produced source code is compressed into a single archive file.
- this step can be implemented using the tar command with the “czf” flags to create a gzipped Unix tarball file (.tgz file extension) on the output source code directory for the current job id.
- the compressed file may be created using a different archive file format such as PK Zip.
- the WRAE can send a notification to the end user's device that the software application has been generated.
- the archive file containing the source code for the software application can be transmitted using a persistent TCP/IP connection that had been established when the end user's device first connected to the WRAE server.
- the WRAE system is designed such that wireframe components are agnostic to a target platform, but source templates are platform-specific. This allows the WRAE system to be extensible. When a new platform needs to be supported, one simply needs to implement a set of source templates for the platform that needs to be supported, ensuring that a source template would be implemented for each type of wireframe component that needs to be supported by the WRAE system.
- FIG. 6 is a high-level block diagram illustrating an example of a computer 600 for use as a WRAE server 108 and implements the method illustrated in FIG. 2 , in accordance with embodiments of the present invention.
- the computer comprises a memory 602 , at least one central processing unit (CPU) 604 , a memory controller 606 , a network interface 608 , an I/O (input/output) interface 610 , and a storage device 612 .
- the executable computer program instructions implementing the method illustrated in FIG. 2 are stored on the storage device, loaded into memory, and executed by the CPU.
- the I/O interface 610 exchanges information to or from devices, which can be either internal or external to the computer 600 .
- the I/O interface 610 allows the computer 600 to retrieve and store program instructions and data, from a storage device 612 .
- the storage device 612 comprises one or more non-transitory computer-readable storage media such as a magnetic disk drive, optical disk drive (e.g. CD-ROM, DVD), or a solid-state memory device.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Stored Programmes (AREA)
Abstract
Description
-
- defining a plurality of shapes common to wireframe images in a library of models;
- defining a plurality of wireframe design rules in a knowledge base;
- receiving, from an end user's device, a set of wireframe images depicting the software application to be fabricated and a set of user-configured options;
- extracting the plurality of features contained in each wireframe image in the set;
- formulating a collection of wireframe components from the plurality of extracted features;
- generating the source code for the software application from the collection of wireframe components;
- providing to the end user's device, an archive file containing the source code for the software application.
-
- defining a plurality of shapes common to wireframe images in a library of models;
- defining a plurality of wireframe design rules in a knowledge base;
- receiving, from an end user's device, a set of wireframe images depicting the software application to be fabricated and a set of user-configured options;
- extracting the plurality of features contained in each wireframe image in the set;
- formulating the collection of wireframe components from the plurality of extracted features;
- generating the source code for the software application from the collection of wireframe components;
- providing to the end user's device, an archive file containing the source code for the software application.
-
- memory;
- input/output (I/O) device;
- at least one non-transitory computer-readable storage medium storing executable computer program instructions comprising instructions for:
- defining a plurality of shapes common to wireframe images in a library of models;
- defining a plurality of wireframe design rules in a knowledge base;
- receiving, from an end user's device via the input/output (I/O) device, a set of wireframe images depicting the software application to be fabricated and a set of user-configured options;
- extracting the plurality of features contained in each wireframe image in the set;
- formulating the collection of wireframe components from the plurality of extracted features;
- generating the source code for the software application from the collection of wireframe components;
- providing to the end user's device, via the input/output (I/O) device, an archive file containing the source code for the software application.
- and at least one CPU for executing the computer program instructions.
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/597,329 US8943468B2 (en) | 2012-08-29 | 2012-08-29 | Wireframe recognition and analysis engine |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/597,329 US8943468B2 (en) | 2012-08-29 | 2012-08-29 | Wireframe recognition and analysis engine |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140068553A1 US20140068553A1 (en) | 2014-03-06 |
US8943468B2 true US8943468B2 (en) | 2015-01-27 |
Family
ID=50189315
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/597,329 Active 2033-10-03 US8943468B2 (en) | 2012-08-29 | 2012-08-29 | Wireframe recognition and analysis engine |
Country Status (1)
Country | Link |
---|---|
US (1) | US8943468B2 (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10489126B2 (en) | 2018-02-12 | 2019-11-26 | Oracle International Corporation | Automated code generation |
US10733754B2 (en) | 2017-01-18 | 2020-08-04 | Oracle International Corporation | Generating a graphical user interface model from an image |
US10838699B2 (en) | 2017-01-18 | 2020-11-17 | Oracle International Corporation | Generating data mappings for user interface screens and screen components for an application |
US11080025B2 (en) | 2019-07-08 | 2021-08-03 | International Business Machines Corporation | Back-end application code stub generation from a front-end application wireframe |
US20220100639A1 (en) * | 2020-09-25 | 2022-03-31 | UiPath, Inc. | Computer-implemented method and system for test automation of an application under test |
US11662874B2 (en) | 2019-05-09 | 2023-05-30 | Tata Consultancy Serviced Limited | Method and system for transforming wireframes to as-is screens with responsive behaviour |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9268535B2 (en) * | 2013-03-12 | 2016-02-23 | Zheng Shi | System and method for computer programming with physical objects on an interactive surface |
US10664647B2 (en) * | 2014-10-28 | 2020-05-26 | Kuang-Yao Lee | Visual web page analysis system and method |
EP3403199A4 (en) * | 2016-01-12 | 2019-09-18 | Kavi Associates, LLC | Multi-technology visual integrated data management and analytics development and deployment environment |
US10515001B2 (en) * | 2016-11-02 | 2019-12-24 | Cognizant Technology Solutions India Pvt. Ltd. | System and method for automating testing of software applications |
US11379655B1 (en) * | 2017-10-16 | 2022-07-05 | BlueOwl, LLC | Systems and methods for automatically serializing and deserializing models |
US10831704B1 (en) | 2017-10-16 | 2020-11-10 | BlueOwl, LLC | Systems and methods for automatically serializing and deserializing models |
CN109739492B (en) * | 2019-01-09 | 2022-06-17 | 武汉瓯越网视有限公司 | Method, terminal, equipment and medium for generating script code |
US11487973B2 (en) * | 2019-07-19 | 2022-11-01 | UiPath, Inc. | Retraining a computer vision model for robotic process automation |
CN112860234A (en) * | 2021-01-18 | 2021-05-28 | 北京神州数字科技有限公司 | Front-end page code generation method and system |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060008151A1 (en) * | 2004-06-30 | 2006-01-12 | National Instruments Corporation | Shape feature extraction and classification |
US20090273597A1 (en) | 2008-05-05 | 2009-11-05 | International Business Machines Corporation | User interface screen layout analysis using hierarchical geometric features |
US7886222B2 (en) | 2006-02-09 | 2011-02-08 | Sap Ag | Converter-tool for a non-programmer user, to generate a web based application |
US7937665B1 (en) | 2000-06-13 | 2011-05-03 | National Instruments Corporation | System and method for automatically generating a graphical program to implement a prototype |
US20110145783A1 (en) * | 2009-12-16 | 2011-06-16 | Infosys Technologies Limited | System and method for representing and validating functional requirements of a software system |
US7987448B2 (en) | 2000-12-20 | 2011-07-26 | National Instruments Corporation | Conversion of a first diagram having states and transitions to a graphical data flow program using an intermediate XML representation |
US8214805B2 (en) | 2006-12-21 | 2012-07-03 | International Business Machines Corporation | Method and system for graphical user interface testing |
US8219969B2 (en) * | 2003-08-28 | 2012-07-10 | Ricoh Company, Ltd. | Data structure used for directory structure navigation in a skeleton code creation tool |
US8438495B1 (en) * | 2009-08-17 | 2013-05-07 | Adobe Systems Incorporated | Methods and systems for creating wireframes and managing containers |
-
2012
- 2012-08-29 US US13/597,329 patent/US8943468B2/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7937665B1 (en) | 2000-06-13 | 2011-05-03 | National Instruments Corporation | System and method for automatically generating a graphical program to implement a prototype |
US7987448B2 (en) | 2000-12-20 | 2011-07-26 | National Instruments Corporation | Conversion of a first diagram having states and transitions to a graphical data flow program using an intermediate XML representation |
US8219969B2 (en) * | 2003-08-28 | 2012-07-10 | Ricoh Company, Ltd. | Data structure used for directory structure navigation in a skeleton code creation tool |
US20060008151A1 (en) * | 2004-06-30 | 2006-01-12 | National Instruments Corporation | Shape feature extraction and classification |
US7886222B2 (en) | 2006-02-09 | 2011-02-08 | Sap Ag | Converter-tool for a non-programmer user, to generate a web based application |
US8214805B2 (en) | 2006-12-21 | 2012-07-03 | International Business Machines Corporation | Method and system for graphical user interface testing |
US20090273597A1 (en) | 2008-05-05 | 2009-11-05 | International Business Machines Corporation | User interface screen layout analysis using hierarchical geometric features |
US8438495B1 (en) * | 2009-08-17 | 2013-05-07 | Adobe Systems Incorporated | Methods and systems for creating wireframes and managing containers |
US20110145783A1 (en) * | 2009-12-16 | 2011-06-16 | Infosys Technologies Limited | System and method for representing and validating functional requirements of a software system |
Non-Patent Citations (9)
Title |
---|
Alvarado et al., SketchRead: A Multi-Domain Sketch Recognition Engine, UIST '04, Oct. 24, 2004, pp. 23-32, ACM, NY. |
Caetano et al., JavaSketchIt: Issues in Sketching the Look of User Interfaces, 2002, Proc. of the 2002 AAAI Spring Symposium, USA. |
Huot et al., The MaggLite post-WIMP toolkit: draw it, connect it and run it, UIST '04, 2004, pp. 257-266, ACM NY. |
Kieffer et al., User interface design by sketching: a complexity analysis of widget representations, EICS '10, Jun. 19, 2010, pp. 57-66, ACM, NY. |
Landay et al., Sketching Interfaces: Toward More Human Interface Design, Journal Computer, Mar. 2001, pp. 56-64 vol. 34 Issue 3, IEEE Computer Society Press, Los Alamitos, CA. |
Lin, A visual language for a sketch-based UI prototyping tool, CHI EA '99, May 1999, pp. 298-299, ACM, NY. |
Meyer, Disposable Sketch Based Interfaces, ACM CHI '96, Apr. 13, 1996, pp. 195-196, ACM, NY. |
Plimmer et al., A toolkit approach to sketched diagram recognition, BCS-HCI '07, 2007, pp. 205-213, BCS, Swinton, UK. |
Schmieder et al., Cross-Domain Diagram Sketch Recognition, VL/HCC Workshop, Sep. 15, 2008, pp. 64-73, Herrsching am Ammersee, Germany. |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10733754B2 (en) | 2017-01-18 | 2020-08-04 | Oracle International Corporation | Generating a graphical user interface model from an image |
US10838699B2 (en) | 2017-01-18 | 2020-11-17 | Oracle International Corporation | Generating data mappings for user interface screens and screen components for an application |
US11119738B2 (en) | 2017-01-18 | 2021-09-14 | Oracle International Corporation | Generating data mappings for user interface screens and screen components for an application |
US10489126B2 (en) | 2018-02-12 | 2019-11-26 | Oracle International Corporation | Automated code generation |
CN110785736A (en) * | 2018-02-12 | 2020-02-11 | 甲骨文国际公司 | Automatic code generation |
CN110785736B (en) * | 2018-02-12 | 2021-04-06 | 甲骨文国际公司 | Automatic code generation |
US11662874B2 (en) | 2019-05-09 | 2023-05-30 | Tata Consultancy Serviced Limited | Method and system for transforming wireframes to as-is screens with responsive behaviour |
US11080025B2 (en) | 2019-07-08 | 2021-08-03 | International Business Machines Corporation | Back-end application code stub generation from a front-end application wireframe |
US20220100639A1 (en) * | 2020-09-25 | 2022-03-31 | UiPath, Inc. | Computer-implemented method and system for test automation of an application under test |
Also Published As
Publication number | Publication date |
---|---|
US20140068553A1 (en) | 2014-03-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8943468B2 (en) | Wireframe recognition and analysis engine | |
US10970097B2 (en) | Adaptive web-based robotic process automation | |
US11119738B2 (en) | Generating data mappings for user interface screens and screen components for an application | |
EP3692438B1 (en) | Automatic generation of a graphic user interface (gui) based on a gui screen image | |
US10579344B2 (en) | Converting visual diagrams into code | |
US10943106B2 (en) | Recognizing text in image data | |
US10430180B2 (en) | System and method for resilient automation upgrade | |
US9015666B2 (en) | Updating product documentation using automated test scripts | |
US11513670B2 (en) | Learning user interface controls via incremental data synthesis | |
US20200387553A1 (en) | Digitization of technical documentation driven by machine learning | |
CN111523021B (en) | Information processing system and execution method thereof | |
US11221833B1 (en) | Automated object detection for user interface generation | |
CN106339479A (en) | Picture naming method and terminal | |
EP4040311A1 (en) | Utilizing machine learning and natural language processing to extract and verify vaccination data | |
US10614125B1 (en) | Modeling and extracting elements in semi-structured documents | |
US9898467B1 (en) | System for data normalization | |
US10445290B1 (en) | System and method for a smart configurable high performance interactive log file viewer | |
WO2022259561A1 (en) | Identification device, identification method, and identification program | |
US20160224918A1 (en) | Business influenced part extraction method and business influenced part extraction device based on business variation | |
US10656921B2 (en) | Sparse object instantiation | |
CN112685056A (en) | Script updating method and device | |
CN115878649A (en) | Data extraction method and electronic equipment | |
CN114816400A (en) | Method and system for building page based on JSON configuration, intelligent terminal and storage medium | |
CN114511380A (en) | Method and apparatus for determining an attribute of an item | |
WO2024049796A1 (en) | Systems and methods for legacy mbse diagram data importation using element specific ml models and schema templates |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO MICRO (ORIGINAL EVENT CODE: MICR) |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, MICRO ENTITY (ORIGINAL EVENT CODE: M3551) Year of fee payment: 4 |
|
AS | Assignment |
Owner name: WIRECOG, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BALASUBRAMANIAN, KAMESH SIVARAMAN;REEL/FRAME:044832/0331 Effective date: 20180205 |
|
AS | Assignment |
Owner name: BALASUBRAMANIAN, KAMESH, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WIRECOG, LLC;REEL/FRAME:052680/0502 Effective date: 20200517 |
|
AS | Assignment |
Owner name: KUBELY, LLC, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BALASUBRAMANIAN, KAMESH S;REEL/FRAME:057646/0101 Effective date: 20210929 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, MICRO ENTITY (ORIGINAL EVENT CODE: M3552); ENTITY STATUS OF PATENT OWNER: MICROENTITY Year of fee payment: 8 |