US20150012905A1 - System and method for streamlining user interface development - Google Patents

System and method for streamlining user interface development

Info

Publication number
US20150012905A1
US20150012905A1 US13/933,351 US201313933351A US2015012905A1 US 20150012905 A1 US20150012905 A1 US 20150012905A1 US 201313933351 A US201313933351 A US 201313933351A US 2015012905 A1 US2015012905 A1 US 2015012905A1
Authority
US
United States
Prior art keywords
user interface
assets
platform
controls
design
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US13/933,351
Other versions
US9858050B2 (en
Inventor
Andrew T. Emmons
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WarnerMedia Direct LLC
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US13/933,351 priority Critical patent/US9858050B2/en
Assigned to YOUI LABS INC. reassignment YOUI LABS INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: EMMONS, Andrew T.
Priority to CA2922860A priority patent/CA2922860C/en
Priority to PCT/CA2014/000530 priority patent/WO2015000054A1/en
Priority to EP14819710.6A priority patent/EP3017364B1/en
Priority to EP22196528.8A priority patent/EP4130978A1/en
Publication of US20150012905A1 publication Critical patent/US20150012905A1/en
Assigned to COMERICA BANK reassignment COMERICA BANK SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YOU I LABS INC.
Priority to US15/856,465 priority patent/US20180203675A1/en
Application granted granted Critical
Publication of US9858050B2 publication Critical patent/US9858050B2/en
Assigned to YOU I LABS INC. reassignment YOU I LABS INC. CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED ON REEL 030955 FRAME 0749. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: EMMONS, Andrew T.
Assigned to YOU I LABS INC. reassignment YOU I LABS INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: COMERICA BANK
Assigned to YOU I LABS INC. reassignment YOU I LABS INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: YOU I LABS INC.
Assigned to WM INTERACTIVE MEDIA HOLDINGS INC. reassignment WM INTERACTIVE MEDIA HOLDINGS INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YOU I LABS INC.
Assigned to YOU I LABS INC. reassignment YOU I LABS INC. MERGER AND CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: YOU I LABS INC
Assigned to WarnerMedia Direct, LLC reassignment WarnerMedia Direct, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: WM INTERACTIVE MEDIA HOLDINGS INC.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/30Creation or generation of source code
    • G06F8/38Creation or generation of source code for implementing user interfaces
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00Animation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/24Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]

Definitions

  • the present invention relates generally to user interface development.
  • the present invention further relates to solutions for aiding in the development of natural user interfaces (NUI).
  • NUI natural user interfaces
  • UI User interface
  • cameras, DVD players, music players, set-top boxes, mobile devices, and so on all have UI components.
  • UX user experience
  • UI elements are designed in part around user workflows driven when a user interacts with system features that are part of the overall UX.
  • Natural user interface is the common parlance used by designers and developers of human-machine interfaces to refer to a user interface that is (1) effectively invisible, or becomes invisible with successive learned interactions, to its users, and (2) is based on “natural” elements. “Natural” in this context means that the elements need not be learned, or require less learning, as they mimic known or already learned processes. The natural elements are contrasted for example with the artificial user elements used for example in control interfaces that are not created based on natural user interface principles.
  • NUI elements are created using processes involving both designers and programmers.
  • Designers use such tools as AFTER EFFECTSTM to compose the look and feel of a UI or UX.
  • a computer network implemented system provides a development platform for streamlining the creation of graphical user interface having natural user interface (NUI) attributes (the “platform”).
  • NUI natural user interface
  • the platform is based on a novel and innovative development framework and workflow that enables the creation of executable GUI elements.
  • a display utility provides a series of screens that provide access to functions that capture a designer's user interface design intent.
  • a composition panel allows a designer to capture the various elements and information indicative of their intent relative to the UI design, to enable creation of a series of UI assets that are encoded with sufficient information to enable their rendering with significantly reduced development effort than in prior art solutions.
  • the assets referred to herein incorporate coding elements corresponding to relevant effects such as position, animation, and “look”.
  • the elements together represent the design intent that is important to convey the important elements for accurately rendering a NUI.
  • the platform is designed to encode the assets with state to state transitions such that the assets themselves can be manipulated (to change order and make modifications for example), and yet these assets are also executable.
  • a computer implemented method comprising: a first computing device hosting a collaborative user interface development environment; a second computing device associated with a designer, that connects to the collaborative user interface development environment for composing a user interface using a plurality of user interface assets that encode the designer's intent; and a third computing device rendering an executable user interface, by compiling the using interface assets.
  • a computer system for streamlining the design of user interface elements comprising: one or more computers, executing one or more computer programs, that define a collaborative user interface development environment (the “platform”), wherein: the platform includes a composition utility that enables one or more users to create a series of user interface assets that are encoded with sufficient information to enable their rendering by a rendering utility; and the rendering utility renders user interface elements using the user interface assets in a what that related effects are rendered accurately.
  • the platform includes a composition utility that enables one or more users to create a series of user interface assets that are encoded with sufficient information to enable their rendering by a rendering utility; and the rendering utility renders user interface elements using the user interface assets in a what that related effects are rendered accurately.
  • the computer system comprises an extractor that decomposes a user interface design into a series of user interface components, and wherein the composition utility enables one or more users to (A) create one or more user interface controls for a user interface design of the video content, and (B) add behavior class names for each behavior control.
  • the platform enables the creation of the assets by (A) exporting UI controls to an asset file, (B) importing UI controls into source code, and accessing or generating one or more code components to execute business logic related to user interface functions (“business logic”) by importing the user interface controls into source code, and writing or importing specific code into the source code to execute the business logic.
  • business logic business logic related to user interface functions
  • the computer system enables one or more users to encode the user interface assets with state to state transitions, such that the assets can be manipulated in connection with a user interface design workflow and also executed by the rendering utility.
  • FIG. 1 illustrates an example of a computer network implementation of the system of the present invention.
  • FIG. 2 is a high level workflow diagram illustrating a workflow representing an example of the computer implemented method of the present invention.
  • FIG. 3 is a system resource diagram illustrating at a high level a possible implementation of the system of the present invention.
  • FIG. 4 is a more detail system resource diagram illustrating resources of the rendering system.
  • FIG. 5 illustrates the process of importation of files to the platform of the present invention.
  • FIG. 6 illustrates the tree structure presented by the platform of the present invention.
  • FIGS. 7 a , 7 b , 7 c , 7 d , 7 e and 7 f illustrate aspects of the development workflow of the present invention, while illustrating representative screens presented by the platform.
  • FIG. 8 illustrate a generic computer system which may provide a suitable operating environment for various embodiments.
  • a computer network implemented system provides a development platform for streamlining the creation of graphical user interface having natural user interface (NUI) attributes (the “platform”).
  • NUI natural user interface
  • the platform is based on a novel and innovative development framework and workflow that enables the creation of executable GUI elements.
  • the platform ( 10 ) includes a display utility ( 11 ) that presents a series of screens made accessible to platform users to access the functions and features described herein.
  • the platform ( 10 ) includes or links to a project management utility ( 13 ).
  • the project management utility ( 13 ) may incorporate standard features for creating and managing UI development related projects.
  • the display utility ( 11 ) may present a project panel (to access features of the project management utility ( 13 )), a composition panel, and a timeline panel.
  • the project panel acts as a bin to import stills, video, and audio footage items. Footage items from the project panel are used in the timeline panel, where layer order and timing can be adjusted.
  • the items visible at the current time marker are displayed in the composition panel, which is used to add transitions and other effects.
  • the composition panel allows a designer to capture the various elements and information indicative of their intent relative to the UI design, to enable creation of a series of UI assets that are encoded with sufficient information to enable their rendering with significantly reduced development effort than in prior art solutions.
  • the assets referred to herein incorporate coding elements corresponding to relevant effects such as position, animation, and “look”.
  • the elements together represent the design intent that is important to convey the important elements for accurately rendering a NUI.
  • the platform is designed to encode the assets with state to state transitions such that the assets themselves can be manipulated (to change order and make modifications for example), and yet these assets are also executable.
  • the platform includes a 3D rendering system ( 12 ) that is programmed to display and animate UI elements.
  • the rendering system ( 12 ) renders UI elements using the assets in a way that related effects are accurately rendered.
  • a designer uses an existing motion graphics/visual graphics platform ( 11 ) (“visual graphics platform”), such as AE, and the output from the visual graphic platform is imported to the platform ( 10 ) of the present invention.
  • the output generally consists of video content that shows UI elements organized in an animated workflow that illustrates interaction with the UI elements.
  • the 3D rendering system ( 12 ) includes: (A) a scene manager ( 16 ), (B) a scene renderer ( 17 ), and (C) a timeline utility ( 18 ).
  • FIG. 2 illustrates one example of implementation of the workflow described herein.
  • a representative implementation of the invention is shown in FIG. 2 as a possible workflow for practicing the invention.
  • a platform user who is a designer creates a UI layout and associated animations for example using AE or a similar platform.
  • a platform user who is a developer loads an AE project file into AE, wherein AE includes a plug-in in accordance with the present invention.
  • C Using the plug-in the platform user who is a developer adds class names to compositions, based on the present invention.
  • the platform user who is a developer optionally selects “Export all compositions and timelines” or equivalent from the plug-in and chooses a directory.
  • the platform user who is a developer loads root composition layout files using source code.
  • E The platform user who is a developer adds business logic and data to the application.
  • a computer implemented method for designing and implementing a series of UI elements in a streamlined way.
  • One implementation of the method consists of: (A) decomposing a UI design into a series of UI components (usually UI designer); (B) creating UI controls for the UI design, using a design tool or composition tool (usually UI designer); (C) add behavior class names for each behavior control (usually UI designer); (D) export UI controls to an asset file (usually UI designer); (E) import UI controls into source code (UI designer or UI developer); (F) access or generate specific code to handle business logic (for example what happens when a specific button is pushed) (usually UI developer); (G) Export UI controls to asset files (UI Developer); (H) import UI controls into source code (UI Developer); and (I) write or import specific code to handle business logic—e.g. what happens when a button is pushed (UI Developer).
  • the present invention provides a collaborative user interface development environment.
  • user interface development usually involves both designers and developers. Possible steps of a development workflow are explained, including which steps may be executed (using the platform of the present invention) by a designer, and which steps by a developer.
  • One aspect of the present invention is that user interface is simplified and therefore it should be understood that steps previously requiring a developer may be executed by a designer, and possibly vice versa.
  • the overall workflow is streamlined thereby allowing new collaborative methods and ways of working, including for example making modifications to a user interface development quickly or possibly in real time upon receipt of feedback from a designer, allowing tighter and more efficient working relationships between collaborators.
  • the platform also allows larger teams to work together more effectively by providing a consistent way of completing work elements, allowing multiple individuals to pick up project tasks, with minimal effort required for a new person to a project to get up to speed.
  • an additional step consists of iterating the UI controls using the design tool.
  • a “UI control” is an element of a GUI that displays an information arrangement changeable by the user, such as a window or a text box.
  • UI controls can have both presentation and behavior.
  • the input to the computer system and method of the present invention may consist of output files from a number of systems such as: (A) graphics files such as PNG or JPEG; (B) scene trees such as those generated by such platforms as AFTER EFFECTSTM (AE) of Adobe; or (C) animation files generated by AE or FLASHTM.
  • A graphics files such as PNG or JPEG
  • B scene trees such as those generated by such platforms as AFTER EFFECTSTM (AE) of Adobe
  • C animation files generated by AE or FLASHTM.
  • the present invention handles both time-based animation files and frame-based animation files.
  • the platform receives as input files from a motion graphics/visual effects platform such as AE.
  • AE is used to develop sophisticated videos using a large number of effects and a 3D stage.
  • Adobe publishes an API which allows developers to extract information within the content.
  • There are many commercial graphical design tools which help designers create 3D content in this way such as MayaTM, 3D StudioTM, BlenderTM, etc., and the present invention may interoperate with such platforms as well.
  • the platform includes an extractor ( 15 ) that analyzes the output files.
  • the computer system provides a number of tools that allows platform users to add class names to as to identify components that are UI controls.
  • the extractor ( 15 ) analyzes, and identifies, components that constitute UI controls.
  • the platform categorizes UI controls into “presentation” related UI controls and “behavior related” UI controls.
  • UI controls can have both presentation and behavior.
  • the extractor ( 15 ) is operable to differentiate between presentation related attributes and behavior related attributes, and generate UI assets based on the input in a way that attributes are encoded based on their type, i.e. whether they are presentation or behavior related.
  • This aspect allows the UI assets to be manipulated as required by the designer, and also for UI assets to encode the designer's intent for aspects of the computer system operations managed by developer users.
  • the presentation may consist of: (A) location (position, scale, rotation), (B) opacity (how translucent it is), (C) bitmaps (the graphics to display the control), (D) visual effects (lighting, shadows, blur), and (D) animations (how parts of the control move in response to events).
  • the behavior consists of: states of the control (pushed up, pushed down, scrolling, etc) information on how to transition from one state to another.
  • the presentation of a UI control is developed in a commercial tool that UI designers know how to use, and then exported as asset files.
  • This workflow is for UI controls that are described in 3D space, using a 3D coordinate system and a camera which determines how they are rendered on the screen. These controls are described visually using a tree structure, where each node in the tree can be a graphical element to display, as shown in FIG. 6 . This graphical element can be a single quad of triangles, or a large mesh of many thousands of triangles.
  • the behavior may be defined in C++ classes within the platform ( 10 ), by instantiating appropriate classes for the UI control when the asset file is loaded.
  • presentation and behavior are separated, they need to communicate with each other. For example, at some point a change in the behavior may trigger a particular visual effect, and so this effect has to be communicated to the behavior code.
  • presentation content can communicate to the behavior code, for example specific presentation related names can be used to name each element. These names can then be used in behavior code to identify a particular element. Markers can be added to timelines thereby indicating for example for an animation, a start, end and name for a particular portion of an animation. The behavior code can then use this to extract a specific animation based on a name.
  • Assigning a class name to a control each UI control can have a string associated with it which indicates the C++ class that will be instantiated when the asset file is loaded. This is the mechanism used to bind the presentation to behavior.
  • a single UI control can export two files—a layout file and a timeline file.
  • the layout files describe the attributes of each node in the control such as position, rotation, anchor point, scale, orientation opacity source—either a bitmap file, a solid color, or another UI control.
  • the export process becomes recursive and writes files for each embedded control. During this process, animation information is saved within a separate timeline file.
  • the animation information contains data needed to change attributes of nodes within the control over time.
  • Importing layout files results in creating C++ classes that both render to the screen and provide UI behavior.
  • Rendering is achieved by using a 3D scene graph data model, which allows the user interface to be hardware accelerated by a GPU.
  • Behavior is achieved by classes which are derived from a base view class. This base view class is in turn derived from a base scene node, as shown in FIG. 6 for example.
  • Importing .timeline files results in creating instances of the timeline class.
  • This class stores all information required to animate attributes of the scene tree that was created when importing the .layout file.
  • the timeline markers are used to create named timelines of a given duration that will be started and stopped based on UI behavior and state.
  • the scene manager ( 16 ) allows a user to define a 3D scene for a UI project using a tree structure, wherein each component corresponds to a UI element, and its position in the tree structure is indicative of the UI elements position in 3D space.
  • FIG. 6 illustrates the operations of the scene manager ( 16 ), wherein a tree structure comprising a plurality of nodes that are called scene nodes.
  • the scene manager ( 16 ) assists platform users in being able to organize UI scenes and their relationships with one another.
  • the scene manager ( 16 ) also is configured to enable transformations of scenes through the UI that it presents. For example application of a transform to a specific node applies the same transformation to all of its child nodes, if any.
  • the scene renderer ( 17 ) includes a data modeller ( 20 ) to drive the rendering of UI elements.
  • the scene nodes contain not only their position data, but other data related to rendering such as for example vertex, mesh and material data, thereby enabling the scene renderer ( 17 ) to render on a node by node basis if this is requested by a user.
  • the scene renderer ( 17 ), in one implementation, is programmed to render 3D graphics on a wide range of rendering APIs such as OpenGL ES 2.0, DirectX, PS3, etc.
  • the scene renderer ( 17 ) requests a display list from the scene manager ( 16 ).
  • the display list can consist of an optimized data stream that compiles information for each scene node.
  • the scene renderer ( 17 ) uses abstract constructs for rendering one or more UI graphical objects, including meshes, materials, and uniform buffers. Significantly, these graphical objects represent sufficient information for rendering as 3D objects.
  • the UI graphical objects are not however a concrete implementation; these using the sub-classes explained below. This way, however, the platform users can interact with 3D content without the need to use underlying rendering APIs that would be required in order to execute to a graphics processing unit (“GPU”).
  • GPU graphics processing unit
  • the design of the scene renderer ( 17 ) allows a user to generate generic 3D drawings and a range of effects. This allows a designer to compose UI elements, review and make changes to UI elements. In effect, this allows a designer to advance the development of UI and UX elements, without the need to wait for work product from programmers. This speeds up the development process and saves resources. It also allows designers to engage in a development process that is more natural and therefore allows designers to be more creative by following inspiration and testing particular design ideas. The disintermediation of the design process that is involved in prior art solutions, where the workflow is divided between designers and programmers is far less advantageous than the present invention.
  • a key insight of the present invention is the particular level of rendering capability provided, and the UI design features that are associated with the rendering capability.
  • the timeline utility ( 18 ) is used to animate scene nodes (transform, opacity) and their contents (vertices, shader variables, etc).
  • the timeline utility ( 18 ) provides a list of “tracks”, where each track corresponds to one attribute for animation.
  • Each track also contains any number of key times, which are used to identify at what point a value changes to a new value.
  • the key times can also be expressed as Bezier curves which indicate how the values change over time.
  • the rendering system workflow consists of: (A) current time is evaluated and updated; (B) user input is received and handled; (C) timelines are updated with a new time; (D) this results in an update to affected scene nodes and their content; (E) the scene manager is processed; (F) any dirty scene nodes re-calculate their composite world matrices; (G) a draw list is rendered; and (H) the draw list is destroyed.
  • the rendering system ( 22 ) of the present invention can be implemented over a middleware layer ( 24 ), as shown in FIG. 1 .
  • the computer system also includes one or more libraries ( 26 ).
  • the libraries ( 26 ) may include for example templates for different UI development projects.
  • the libraries ( 26 ) may also include routines for items such as image decoding, text glyph rendering, etc.
  • a template may include a look for a view, or layout.
  • the layout may contain one or more scene nodes that together define the graphical elements for a view.
  • a template can also define the animations for a view for example how items move and relate together when responding to specific states.
  • the platform ( 10 ) may include: (A) an import utility ( 30 ); (B) scene templates ( 32 ) repository; (C) a views utility ( 34 ); (D) a framework component ( 36 ); (E) an animation utility ( 38 ); and (F) the rendering system ( 12 ).
  • the framework component ( 36 ) acts as connector or communication layer between the various components in order to enable the workflows described.
  • the import utility ( 30 ) extracts information from one or more tools and makes the information available to the other components.
  • the views utility ( 34 ) may include common UI behaviors, for example ListView, PushButtonView, etc
  • a representative implementation of the rendering system ( 12 ) may be implemented with the components shown in FIG. 4 . These may include: (A) a component referred to as a CYIDisplay, which is a display component, which abstracts access to the OpenGL hardware; (B) a component referred to as the CYIBuffer, which contains bitmap data and blitting routines for both hardware and software buffers; (C) a component referred to as a CYIRenderer, which contains a list of scene node data, renders using OpenGL; (D) a component referred to as the CYISceneNode, which contains a building block of graphics sub-components, and provides a scene tree that includes positional information (this is no the component where drawing occurs); (E) a component referred to as the CYISceneObject, which is what the user draws; and finally (F) a component referred to as the CYISceneManager, which returns a render list of scene nodes and provides scene node data to the
  • the platform ( 10 ) uses After EffectsTM from Adobe. What follows is an example of the operation of the platform ( 10 ), working with After Effects (“AE”).
  • a UI design tool such as AE can communicate with the server application ( 52 ) shown in FIG. 1 .
  • a plug-in may be provided (not shown) which incorporates aspects of the features described into a UI design tool such as AE.
  • UI design features such as those available through AE, as an example, may be implemented to the server application ( 52 ).
  • AE files consist of nested compositions. Each composition can have layers and/or other compositions.
  • the platform ( 10 ) may include a plug-in that extracts files from AE, and specifically layout files for each composition from AE.
  • the layout files extracted from AE may include: (A) a list of layers used in the composition, the layers containing 3D positional data such as position, rotation, anchor point, scale, orientation; and (B) a list of sources used in the composition, which may include bitmaps, NULL objects, solid colors, or compositions.
  • a platform user can name a composition. Class names are assigned for the purposes of the scene view. The various elements for the composition are defined. Width/height for the scene may be selected using 2D coordinates.
  • the plug-in exports timeline related information from AE (although AE does not contain a timeline concept similar to that used by the platform.
  • the user selects values for multiple tracks for each element.
  • Each track an element's property—such as position rotation.
  • key time information is obtained. This includes for example what a particular value was at a given time, i.e. that at time X track Y had Z value. Rate of change can be determined using for example Bezier control points.
  • Marker information is obtained from AE that represents logical or functional elements. For example, one marker may indicate ‘pushdown’, another ‘pushup’. These have a start time and duration and allow the platform ( 10 ) to automatically bind behavior state to the animation. For example, if the composition class name is ‘CYIPushButtonView’, and the timeline contained a marker for ‘ondown’, then that part of the timeline would be used when the user pushes down on the button
  • the platform aligns automatically the various elements with the timeline automatically.
  • the platform loads the layout file for the root composition and automatically loads other layout files for any embedded compositions. These other layout files are created as CYIAETemplate instances and stored in the asset manager. Timeline files are automatically loaded as well, one per composition. The timeline data is stored in the CYIETTemplate instance.
  • a scene tree of CYISceneView (or derived) instances are created, complete with timeline animation data, from the CYIAETemplates.
  • FIGS. 7 a , 7 b , 7 c , 7 d , 7 e , and 7 f illustrate the operation of the present invention by referring to a series of representative screenshot views of the platform of the present invention, in one representative implementation of the present invention.
  • FIGS. 7 a , 7 b , 7 c , 7 d , 7 e , and 7 f illustrate the operation of the present invention by referring to a series of representative screenshot views of the platform of the present invention, in one representative implementation of the present invention.
  • access to features of the platform ( 10 ) are shown as enhancements to AE.
  • FIG. 7 a shows a screen displayed by the platform ( 10 ) that a UI designer user would use to create a UI layout using for example animations from AE.
  • the UI designer can define the position of UI items, how they animate, and how they interact with one another.
  • FIG. 7 b shows that any given UI design is made of multiple components, each component being for example a composition from AE.
  • each button, or list item, in a given UI design is its own composition.
  • the UI developer then adds animations to the components, and puts specific markers on the timelines to give a context to the timeline. For example, they can add ‘DownOn, ‘UpPn’, etc. markers to the timeline for a button.
  • FIG. 7 c illustrates a screen where the UI developer can access various other tools to create complex UI controls in order to generate looks and animations based on the EU content.
  • FIG. 7 d shows a representative screen once a project has been assigned to a software developer.
  • the platform ( 10 ) integrates tools and workflows that are used by designers and developers respectively, in a seamless manner.
  • the developer can provide each composition a class name by adding information in.
  • FIG. 7 e shows a representative screen displaying the functions of the plugin.
  • the plugin of the platform ( 10 ) allows the developer to write a timeline and a layout file for each composition.
  • the developer then loads the top-level root composition layout file.
  • This file in turn loads all of the compositions it uses, along with their timeline files.
  • a working UI tree is created that can be rendered with a 3D engine, and can be interacted with by the user.
  • the final step is for the software developer to add business logic—what happens when a button is pushed, what kind of data is used in the list view, etc.
  • the collaborative platform of the present invention enables designers and programmers to collaborate on complex UI development projects in an efficient manner.
  • the platform enables rapid prototyping and creation of demonstration UI assets that users can provide feedback on, thereby improving the user engagement levels of UI assets.
  • UI assets can be swapped out, thus enabling rapid development of UI assets based on different scenarios.
  • the platform permits the capture of the designer's intent in an efficient manner.
  • the development platform allows development of UI assets across several operating systems.
  • FIG. 1 an example of the implementation of the present invention is illustrated, as a computer network implemented system. It should be understood that FIG. 1 illustrates only one possible computer network architecture for implementing the present invention, but the invention is not limited to the particular implementation shown in FIG. 1 .
  • FIG. 1 illustrates a server environment A linked to a server computer ( 50 ).
  • Server computer ( 50 ) is linked to a server application ( 52 ) that includes a collaborative development system ( 54 ), which may be implemented as an application repository.
  • the collaborative development system ( 104 ) includes the display utility ( 11 ) and the project management utility, and in one implementation also integrates the resources of the 3D rendering system ( 12 ), which may be implemented as a middleware resource as shown in FIG. 1 .
  • the server computer ( 50 ) provides online access to the functions of the 3D rendering system ( 12 ) in order to enable access to platform users to UI assets developed using the collaborative development system ( 54 ).
  • FIG. 4 shows a generic computer device 100 that may include a central processing unit (“CPU”) 102 connected to a storage unit 104 and to a random access memory 106 .
  • the CPU 102 may process an operating system 101 , application program 103 , and data 123 .
  • the operating system 101 , application program 103 , and data 123 may be stored in storage unit 104 and loaded into memory 106 , as may be required.
  • Computer device 100 may further include a graphics processing unit (GPU) 122 which is operatively connected to CPU 102 and to memory 106 to offload intensive image processing calculations from CPU 102 and run these calculations in parallel with CPU 102 .
  • GPU graphics processing unit
  • An operator 107 may interact with the computer device 100 using a video display 108 connected by a video interface 105 , and various input/output devices such as a keyboard 110 , mouse 112 , and disk drive or solid state drive 114 connected by an I/O interface 109 .
  • the mouse 112 may be configured to control movement of a cursor in the video display 108 , and to operate various graphical user interface (GUI) controls appearing in the video display 108 with a mouse button.
  • GUI graphical user interface
  • the disk drive or solid state drive 114 may be configured to accept computer readable media 116 .
  • the computer device 100 may form part of a network via a network interface 111 , allowing the computer device 100 to communicate through wired or wireless communications with other suitably configured data processing systems (not shown).
  • the generic computer device 100 may be embodied in various form factors including desktop and laptop computers, and wireless mobile computer devices such as tablets, smart phones and super phones operating on various operating systems. It will be appreciated that the present description does not limit the size or form factor of the computing device on which the present system and method may be embodied.
  • the disclosure provides systems, devices, methods, and computer programming products, including non-transient machine-readable instruction sets, for use in implementing such methods and enabling the functionality described previously.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

A computer network implemented system is provided that provides a development platform for streamlining the creation of graphical user interface having natural user interface (NUI) attributes (the “platform”). The platform is based on a novel and innovative development framework and workflow that enables the creation of executable GUI elements. A display utility provides a series of screens that provide access to functions that capture a designer's user interface design intent. A user interface is provided that allows a designer to capture the various elements and information indicative of their intent relative to the UI design, to enable creation of a series of UI assets that are encoded with sufficient information to enable their rendering with significantly reduced development effort than in prior art solutions. In one aspect, the assets referred to herein incorporate coding elements corresponding to relevant effects such as position, animation, and “look”. The elements together represent the design intent that is important to convey the important elements for accurately rendering a NUI. More specifically, the platform is designed to encode the assets with state to state transitions such that the assets themselves can be manipulated (to change order and make modifications for example), and yet these assets are also executable.

Description

    FIELD OF THE INVENTION
  • The present invention relates generally to user interface development. The present invention further relates to solutions for aiding in the development of natural user interfaces (NUI).
  • BACKGROUND OF THE INVENTION
  • User interface (UI) is created for use in many software programs and to interact with functions of a variety of hardware device. For example, cameras, DVD players, music players, set-top boxes, mobile devices, and so on all have UI components.
  • More broadly speaking, user experience (UX) refers to a person's perceptions and responses that result from the use or anticipated use of a product, system or service. UI elements are designed in part around user workflows driven when a user interacts with system features that are part of the overall UX.
  • There is significant demand for UI that is developed based on “natural user interface” principles. Natural user interface is the common parlance used by designers and developers of human-machine interfaces to refer to a user interface that is (1) effectively invisible, or becomes invisible with successive learned interactions, to its users, and (2) is based on “natural” elements. “Natural” in this context means that the elements need not be learned, or require less learning, as they mimic known or already learned processes. The natural elements are contrasted for example with the artificial user elements used for example in control interfaces that are not created based on natural user interface principles.
  • Designing and implementing UI components, and broader UX elements, however, is generally quite time consuming and expensive.
  • Generally speaking, NUI elements are created using processes involving both designers and programmers. Designers use such tools as AFTER EFFECTS™ to compose the look and feel of a UI or UX.
  • Software developers then attempt to develop assets that convey the intention of the designers, but many times key elements intended by the designers are lost in translation. As a result, assets are generally sent back to the designers and they identify elements that require changes. This results in back and forth and contributes generally to an iterative and time consuming process. Furthermore, UI development requires iteration generally, and this is time consuming and expensive to do using prior art solutions.
  • Therefore, what is needed is a solution to address at least some of these limitations.
  • SUMMARY OF THE INVENTION
  • A computer network implemented system is provided that provides a development platform for streamlining the creation of graphical user interface having natural user interface (NUI) attributes (the “platform”). The platform is based on a novel and innovative development framework and workflow that enables the creation of executable GUI elements. A display utility provides a series of screens that provide access to functions that capture a designer's user interface design intent.
  • A composition panel allows a designer to capture the various elements and information indicative of their intent relative to the UI design, to enable creation of a series of UI assets that are encoded with sufficient information to enable their rendering with significantly reduced development effort than in prior art solutions. In one aspect, the assets referred to herein incorporate coding elements corresponding to relevant effects such as position, animation, and “look”. The elements together represent the design intent that is important to convey the important elements for accurately rendering a NUI. More specifically, the platform is designed to encode the assets with state to state transitions such that the assets themselves can be manipulated (to change order and make modifications for example), and yet these assets are also executable.
  • In one aspect of the invention, a computer implemented method is provided comprising: a first computing device hosting a collaborative user interface development environment; a second computing device associated with a designer, that connects to the collaborative user interface development environment for composing a user interface using a plurality of user interface assets that encode the designer's intent; and a third computing device rendering an executable user interface, by compiling the using interface assets.
  • In another aspect, a computer system is provided for streamlining the design of user interface elements comprising: one or more computers, executing one or more computer programs, that define a collaborative user interface development environment (the “platform”), wherein: the platform includes a composition utility that enables one or more users to create a series of user interface assets that are encoded with sufficient information to enable their rendering by a rendering utility; and the rendering utility renders user interface elements using the user interface assets in a what that related effects are rendered accurately.
  • In another aspect, the computer system comprises an extractor that decomposes a user interface design into a series of user interface components, and wherein the composition utility enables one or more users to (A) create one or more user interface controls for a user interface design of the video content, and (B) add behavior class names for each behavior control.
  • In a still other aspect, the platform enables the creation of the assets by (A) exporting UI controls to an asset file, (B) importing UI controls into source code, and accessing or generating one or more code components to execute business logic related to user interface functions (“business logic”) by importing the user interface controls into source code, and writing or importing specific code into the source code to execute the business logic.
  • In another aspect, the computer system enables one or more users to encode the user interface assets with state to state transitions, such that the assets can be manipulated in connection with a user interface design workflow and also executed by the rendering utility.
  • In this respect, before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or the examples provided therein, or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • In the drawings, embodiments of the invention are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of the invention.
  • FIG. 1 illustrates an example of a computer network implementation of the system of the present invention.
  • FIG. 2 is a high level workflow diagram illustrating a workflow representing an example of the computer implemented method of the present invention.
  • FIG. 3 is a system resource diagram illustrating at a high level a possible implementation of the system of the present invention.
  • FIG. 4 is a more detail system resource diagram illustrating resources of the rendering system.
  • FIG. 5 illustrates the process of importation of files to the platform of the present invention.
  • FIG. 6 illustrates the tree structure presented by the platform of the present invention.
  • FIGS. 7 a, 7 b, 7 c, 7 d, 7 e and 7 f illustrate aspects of the development workflow of the present invention, while illustrating representative screens presented by the platform.
  • FIG. 8 illustrate a generic computer system which may provide a suitable operating environment for various embodiments.
  • DETAILED DESCRIPTION
  • A computer network implemented system is provided that provides a development platform for streamlining the creation of graphical user interface having natural user interface (NUI) attributes (the “platform”). The platform is based on a novel and innovative development framework and workflow that enables the creation of executable GUI elements.
  • As shown in FIG. 1, in one aspect, the platform (10) includes a display utility (11) that presents a series of screens made accessible to platform users to access the functions and features described herein. In one implementation, the platform (10) includes or links to a project management utility (13). The project management utility (13) may incorporate standard features for creating and managing UI development related projects.
  • The display utility (11) may present a project panel (to access features of the project management utility (13)), a composition panel, and a timeline panel. The project panel acts as a bin to import stills, video, and audio footage items. Footage items from the project panel are used in the timeline panel, where layer order and timing can be adjusted. The items visible at the current time marker are displayed in the composition panel, which is used to add transitions and other effects.
  • The composition panel allows a designer to capture the various elements and information indicative of their intent relative to the UI design, to enable creation of a series of UI assets that are encoded with sufficient information to enable their rendering with significantly reduced development effort than in prior art solutions. In one aspect, the assets referred to herein incorporate coding elements corresponding to relevant effects such as position, animation, and “look”. The elements together represent the design intent that is important to convey the important elements for accurately rendering a NUI. More specifically, the platform is designed to encode the assets with state to state transitions such that the assets themselves can be manipulated (to change order and make modifications for example), and yet these assets are also executable.
  • In one aspect, the platform includes a 3D rendering system (12) that is programmed to display and animate UI elements. The rendering system (12) renders UI elements using the assets in a way that related effects are accurately rendered.
  • In one aspect, a designer uses an existing motion graphics/visual graphics platform (11) (“visual graphics platform”), such as AE, and the output from the visual graphic platform is imported to the platform (10) of the present invention. The output generally consists of video content that shows UI elements organized in an animated workflow that illustrates interaction with the UI elements.
  • In one aspect, the 3D rendering system (12) includes: (A) a scene manager (16), (B) a scene renderer (17), and (C) a timeline utility (18).
  • FIG. 2 illustrates one example of implementation of the workflow described herein. A representative implementation of the invention is shown in FIG. 2 as a possible workflow for practicing the invention. (A) A platform user who is a designer creates a UI layout and associated animations for example using AE or a similar platform. (B) A platform user who is a developer loads an AE project file into AE, wherein AE includes a plug-in in accordance with the present invention. (C) Using the plug-in the platform user who is a developer adds class names to compositions, based on the present invention. (D) The platform user who is a developer optionally selects “Export all compositions and timelines” or equivalent from the plug-in and chooses a directory. (D) The platform user who is a developer loads root composition layout files using source code. (E) The platform user who is a developer adds business logic and data to the application.
  • In another aspect, a computer implemented method is provided for designing and implementing a series of UI elements in a streamlined way. One implementation of the method consists of: (A) decomposing a UI design into a series of UI components (usually UI designer); (B) creating UI controls for the UI design, using a design tool or composition tool (usually UI designer); (C) add behavior class names for each behavior control (usually UI designer); (D) export UI controls to an asset file (usually UI designer); (E) import UI controls into source code (UI designer or UI developer); (F) access or generate specific code to handle business logic (for example what happens when a specific button is pushed) (usually UI developer); (G) Export UI controls to asset files (UI Developer); (H) import UI controls into source code (UI Developer); and (I) write or import specific code to handle business logic—e.g. what happens when a button is pushed (UI Developer).
  • The present invention provides a collaborative user interface development environment. As previously explained, user interface development usually involves both designers and developers. Possible steps of a development workflow are explained, including which steps may be executed (using the platform of the present invention) by a designer, and which steps by a developer. One aspect of the present invention, is that user interface is simplified and therefore it should be understood that steps previously requiring a developer may be executed by a designer, and possibly vice versa. Furthermore, the overall workflow is streamlined thereby allowing new collaborative methods and ways of working, including for example making modifications to a user interface development quickly or possibly in real time upon receipt of feedback from a designer, allowing tighter and more efficient working relationships between collaborators. The platform also allows larger teams to work together more effectively by providing a consistent way of completing work elements, allowing multiple individuals to pick up project tasks, with minimal effort required for a new person to a project to get up to speed.
  • If necessary, an additional step consists of iterating the UI controls using the design tool.
  • A “UI control” is an element of a GUI that displays an information arrangement changeable by the user, such as a window or a text box. UI controls can have both presentation and behavior.
  • The input to the computer system and method of the present invention may consist of output files from a number of systems such as: (A) graphics files such as PNG or JPEG; (B) scene trees such as those generated by such platforms as AFTER EFFECTS™ (AE) of Adobe; or (C) animation files generated by AE or FLASH™.
  • In one aspect, the present invention handles both time-based animation files and frame-based animation files.
  • In one aspect, the platform receives as input files from a motion graphics/visual effects platform such as AE. AE is used to develop sophisticated videos using a large number of effects and a 3D stage. Adobe publishes an API which allows developers to extract information within the content. There are many commercial graphical design tools which help designers create 3D content in this way such as Maya™, 3D Studio™, Blender™, etc., and the present invention may interoperate with such platforms as well.
  • In one aspect, the platform includes an extractor (15) that analyzes the output files. The computer system provides a number of tools that allows platform users to add class names to as to identify components that are UI controls.
  • Alternatively, the extractor (15) analyzes, and identifies, components that constitute UI controls. In addition, in one aspect, the platform categorizes UI controls into “presentation” related UI controls and “behavior related” UI controls. UI controls can have both presentation and behavior.
  • The separate treatment of presentation and behavior, allows the treatment of presentation as part of asset files. In one aspect, the extractor (15) is operable to differentiate between presentation related attributes and behavior related attributes, and generate UI assets based on the input in a way that attributes are encoded based on their type, i.e. whether they are presentation or behavior related. This aspect allows the UI assets to be manipulated as required by the designer, and also for UI assets to encode the designer's intent for aspects of the computer system operations managed by developer users.
  • The presentation may consist of: (A) location (position, scale, rotation), (B) opacity (how translucent it is), (C) bitmaps (the graphics to display the control), (D) visual effects (lighting, shadows, blur), and (D) animations (how parts of the control move in response to events). The behavior consists of: states of the control (pushed up, pushed down, scrolling, etc) information on how to transition from one state to another. The presentation of a UI control is developed in a commercial tool that UI designers know how to use, and then exported as asset files.
  • This workflow is for UI controls that are described in 3D space, using a 3D coordinate system and a camera which determines how they are rendered on the screen. These controls are described visually using a tree structure, where each node in the tree can be a graphical element to display, as shown in FIG. 6. This graphical element can be a single quad of triangles, or a large mesh of many thousands of triangles. The behavior may be defined in C++ classes within the platform (10), by instantiating appropriate classes for the UI control when the asset file is loaded.
  • Even though presentation and behavior are separated, they need to communicate with each other. For example, at some point a change in the behavior may trigger a particular visual effect, and so this effect has to be communicated to the behavior code. There are a number of ways that the presentation content can communicate to the behavior code, for example specific presentation related names can be used to name each element. These names can then be used in behavior code to identify a particular element. Markers can be added to timelines thereby indicating for example for an animation, a start, end and name for a particular portion of an animation. The behavior code can then use this to extract a specific animation based on a name.
  • Assigning a class name to a control—each UI control can have a string associated with it which indicates the C++ class that will be instantiated when the asset file is loaded. This is the mechanism used to bind the presentation to behavior.
  • A single UI control can export two files—a layout file and a timeline file. The layout files describe the attributes of each node in the control such as position, rotation, anchor point, scale, orientation opacity source—either a bitmap file, a solid color, or another UI control.
  • If the source is another UI Control, the export process becomes recursive and writes files for each embedded control. During this process, animation information is saved within a separate timeline file. The animation information contains data needed to change attributes of nodes within the control over time.
  • Importing Presentation Data
  • Importing layout files results in creating C++ classes that both render to the screen and provide UI behavior. Rendering is achieved by using a 3D scene graph data model, which allows the user interface to be hardware accelerated by a GPU. Behavior is achieved by classes which are derived from a base view class. This base view class is in turn derived from a base scene node, as shown in FIG. 6 for example.
  • Importing .timeline files results in creating instances of the timeline class. This class stores all information required to animate attributes of the scene tree that was created when importing the .layout file. The timeline markers are used to create named timelines of a given duration that will be started and stopped based on UI behavior and state.
  • In one aspect, the scene manager (16) allows a user to define a 3D scene for a UI project using a tree structure, wherein each component corresponds to a UI element, and its position in the tree structure is indicative of the UI elements position in 3D space.
  • FIG. 6 illustrates the operations of the scene manager (16), wherein a tree structure comprising a plurality of nodes that are called scene nodes. The scene manager (16) assists platform users in being able to organize UI scenes and their relationships with one another. The scene manager (16) also is configured to enable transformations of scenes through the UI that it presents. For example application of a transform to a specific node applies the same transformation to all of its child nodes, if any.
  • The scene renderer (17) includes a data modeller (20) to drive the rendering of UI elements.
  • Significantly, the scene nodes contain not only their position data, but other data related to rendering such as for example vertex, mesh and material data, thereby enabling the scene renderer (17) to render on a node by node basis if this is requested by a user.
  • The scene renderer (17), in one implementation, is programmed to render 3D graphics on a wide range of rendering APIs such as OpenGL ES 2.0, DirectX, PS3, etc. In one implementation, the scene renderer (17) requests a display list from the scene manager (16). The display list can consist of an optimized data stream that compiles information for each scene node.
  • In one implementation, the scene renderer (17) uses abstract constructs for rendering one or more UI graphical objects, including meshes, materials, and uniform buffers. Significantly, these graphical objects represent sufficient information for rendering as 3D objects. The UI graphical objects are not however a concrete implementation; these using the sub-classes explained below. This way, however, the platform users can interact with 3D content without the need to use underlying rendering APIs that would be required in order to execute to a graphics processing unit (“GPU”).
  • The design of the scene renderer (17) allows a user to generate generic 3D drawings and a range of effects. This allows a designer to compose UI elements, review and make changes to UI elements. In effect, this allows a designer to advance the development of UI and UX elements, without the need to wait for work product from programmers. This speeds up the development process and saves resources. It also allows designers to engage in a development process that is more natural and therefore allows designers to be more creative by following inspiration and testing particular design ideas. The disintermediation of the design process that is involved in prior art solutions, where the workflow is divided between designers and programmers is far less advantageous than the present invention. A key insight of the present invention is the particular level of rendering capability provided, and the UI design features that are associated with the rendering capability.
  • Timeline Utility
  • The timeline utility (18) is used to animate scene nodes (transform, opacity) and their contents (vertices, shader variables, etc). The timeline utility (18) provides a list of “tracks”, where each track corresponds to one attribute for animation. Each track also contains any number of key times, which are used to identify at what point a value changes to a new value. The key times can also be expressed as Bezier curves which indicate how the values change over time.
  • In one aspect the rendering system workflow consists of: (A) current time is evaluated and updated; (B) user input is received and handled; (C) timelines are updated with a new time; (D) this results in an update to affected scene nodes and their content; (E) the scene manager is processed; (F) any dirty scene nodes re-calculate their composite world matrices; (G) a draw list is rendered; and (H) the draw list is destroyed.
  • In one aspect of the present invention, the rendering system (22) of the present invention can be implemented over a middleware layer (24), as shown in FIG. 1. The computer system also includes one or more libraries (26). The libraries (26) may include for example templates for different UI development projects. The libraries (26) may also include routines for items such as image decoding, text glyph rendering, etc.
  • A template may include a look for a view, or layout. The layout may contain one or more scene nodes that together define the graphical elements for a view. A template can also define the animations for a view for example how items move and relate together when responding to specific states.
  • As shown in FIG. 3, the platform (10) may include: (A) an import utility (30); (B) scene templates (32) repository; (C) a views utility (34); (D) a framework component (36); (E) an animation utility (38); and (F) the rendering system (12).
  • In one aspect, the framework component (36) acts as connector or communication layer between the various components in order to enable the workflows described.
  • The import utility (30) extracts information from one or more tools and makes the information available to the other components. The views utility (34) may include common UI behaviors, for example ListView, PushButtonView, etc
  • A representative implementation of the rendering system (12) may be implemented with the components shown in FIG. 4. These may include: (A) a component referred to as a CYIDisplay, which is a display component, which abstracts access to the OpenGL hardware; (B) a component referred to as the CYIBuffer, which contains bitmap data and blitting routines for both hardware and software buffers; (C) a component referred to as a CYIRenderer, which contains a list of scene node data, renders using OpenGL; (D) a component referred to as the CYISceneNode, which contains a building block of graphics sub-components, and provides a scene tree that includes positional information (this is no the component where drawing occurs); (E) a component referred to as the CYISceneObject, which is what the user draws; and finally (F) a component referred to as the CYISceneManager, which returns a render list of scene nodes and provides scene node data to the renderer.
  • Example In Operation
  • In one possible implementation, the platform (10) uses After Effects™ from Adobe. What follows is an example of the operation of the platform (10), working with After Effects (“AE”). A UI design tool such as AE can communicate with the server application (52) shown in FIG. 1. Alternatively, a plug-in may be provided (not shown) which incorporates aspects of the features described into a UI design tool such as AE. Alternatively, UI design features such as those available through AE, as an example, may be implemented to the server application (52). What follows is an example of the implementation of the present invention, wherein features of a UI design tool such as AE are enhanced using a plug-in, including by optionally leveraging the resources of the server computer (50) that includes collaborative development features that enhance the UI development functions.
  • AE files consist of nested compositions. Each composition can have layers and/or other compositions. The platform (10) may include a plug-in that extracts files from AE, and specifically layout files for each composition from AE. The layout files extracted from AE may include: (A) a list of layers used in the composition, the layers containing 3D positional data such as position, rotation, anchor point, scale, orientation; and (B) a list of sources used in the composition, which may include bitmaps, NULL objects, solid colors, or compositions.
  • A platform user can name a composition. Class names are assigned for the purposes of the scene view. The various elements for the composition are defined. Width/height for the scene may be selected using 2D coordinates. The plug-in exports timeline related information from AE (although AE does not contain a timeline concept similar to that used by the platform.
  • The user selects values for multiple tracks for each element. Each track an element's property—such as position rotation.
  • For each track, key time information is obtained. This includes for example what a particular value was at a given time, i.e. that at time X track Y had Z value. Rate of change can be determined using for example Bezier control points.
  • Marker information is obtained from AE that represents logical or functional elements. For example, one marker may indicate ‘pushdown’, another ‘pushup’. These have a start time and duration and allow the platform (10) to automatically bind behavior state to the animation. For example, if the composition class name is ‘CYIPushButtonView’, and the timeline contained a marker for ‘ondown’, then that part of the timeline would be used when the user pushes down on the button
  • The platform aligns automatically the various elements with the timeline automatically.
  • In one implementation, the platform loads the layout file for the root composition and automatically loads other layout files for any embedded compositions. These other layout files are created as CYIAETemplate instances and stored in the asset manager. Timeline files are automatically loaded as well, one per composition. The timeline data is stored in the CYIETTemplate instance.
  • As the file loading continues, a scene tree of CYISceneView (or derived) instances are created, complete with timeline animation data, from the CYIAETemplates.
  • FIGS. 7 a, 7 b, 7 c, 7 d, 7 e, and 7 f illustrate the operation of the present invention by referring to a series of representative screenshot views of the platform of the present invention, in one representative implementation of the present invention. In the implementations of the invention shown in these FIGS., in order to illustrate the operation of the present invention, access to features of the platform (10) are shown as enhancements to AE.
  • FIG. 7 a shows a screen displayed by the platform (10) that a UI designer user would use to create a UI layout using for example animations from AE. The UI designer can define the position of UI items, how they animate, and how they interact with one another.
  • FIG. 7 b shows that any given UI design is made of multiple components, each component being for example a composition from AE. For example, each button, or list item, in a given UI design is its own composition. The UI developer then adds animations to the components, and puts specific markers on the timelines to give a context to the timeline. For example, they can add ‘DownOn, ‘UpPn’, etc. markers to the timeline for a button.
  • FIG. 7 c illustrates a screen where the UI developer can access various other tools to create complex UI controls in order to generate looks and animations based on the EU content.
  • FIG. 7 d shows a representative screen once a project has been assigned to a software developer. The platform (10) integrates tools and workflows that are used by designers and developers respectively, in a seamless manner. As shown in FIG. 6 d, the developer can provide each composition a class name by adding information in.
  • FIG. 7 e shows a representative screen displaying the functions of the plugin. The plugin of the platform (10) allows the developer to write a timeline and a layout file for each composition.
  • As shown in FIG. 7 f, the developer then loads the top-level root composition layout file. This file in turn loads all of the compositions it uses, along with their timeline files. Once loaded, a working UI tree is created that can be rendered with a 3D engine, and can be interacted with by the user. The final step is for the software developer to add business logic—what happens when a button is pushed, what kind of data is used in the list view, etc.
  • Advantages
  • Various advantages of the present invention have been mentioned above. Additional representative advantages are now described.
  • The collaborative platform of the present invention enables designers and programmers to collaborate on complex UI development projects in an efficient manner.
  • High value UI assets are created in an efficient manner that respects the need for flexibility and responsiveness to support the creative process of UI design.
  • Use of the platform provides learning opportunities to developers.
  • The platform enables rapid prototyping and creation of demonstration UI assets that users can provide feedback on, thereby improving the user engagement levels of UI assets.
  • UI assets can be swapped out, thus enabling rapid development of UI assets based on different scenarios.
  • The platform permits the capture of the designer's intent in an efficient manner.
  • The development platform allows development of UI assets across several operating systems.
  • System Implementation
  • Referring to FIG. 1 an example of the implementation of the present invention is illustrated, as a computer network implemented system. It should be understood that FIG. 1 illustrates only one possible computer network architecture for implementing the present invention, but the invention is not limited to the particular implementation shown in FIG. 1.
  • FIG. 1 illustrates a server environment A linked to a server computer (50). Server computer (50) is linked to a server application (52) that includes a collaborative development system (54), which may be implemented as an application repository. The collaborative development system (104) includes the display utility (11) and the project management utility, and in one implementation also integrates the resources of the 3D rendering system (12), which may be implemented as a middleware resource as shown in FIG. 1. In one implementation, the server computer (50) provides online access to the functions of the 3D rendering system (12) in order to enable access to platform users to UI assets developed using the collaborative development system (54).
  • Referring to FIG. 8, the present system and method may be practiced in various embodiments. A suitably configured generic computer device, and associated communications networks, devices, software and firmware may provide a platform for enabling one or more embodiments as described above. By way of example, FIG. 4 shows a generic computer device 100 that may include a central processing unit (“CPU”) 102 connected to a storage unit 104 and to a random access memory 106. The CPU 102 may process an operating system 101, application program 103, and data 123. The operating system 101, application program 103, and data 123 may be stored in storage unit 104 and loaded into memory 106, as may be required. Computer device 100 may further include a graphics processing unit (GPU) 122 which is operatively connected to CPU 102 and to memory 106 to offload intensive image processing calculations from CPU 102 and run these calculations in parallel with CPU 102. An operator 107 may interact with the computer device 100 using a video display 108 connected by a video interface 105, and various input/output devices such as a keyboard 110, mouse 112, and disk drive or solid state drive 114 connected by an I/O interface 109. In known manner, the mouse 112 may be configured to control movement of a cursor in the video display 108, and to operate various graphical user interface (GUI) controls appearing in the video display 108 with a mouse button. The disk drive or solid state drive 114 may be configured to accept computer readable media 116. The computer device 100 may form part of a network via a network interface 111, allowing the computer device 100 to communicate through wired or wireless communications with other suitably configured data processing systems (not shown). The generic computer device 100 may be embodied in various form factors including desktop and laptop computers, and wireless mobile computer devices such as tablets, smart phones and super phones operating on various operating systems. It will be appreciated that the present description does not limit the size or form factor of the computing device on which the present system and method may be embodied.
  • While illustrative embodiments of the invention have been described above, it will be appreciated that various changes and modifications may be made without departing from the scope of the present invention.
  • In further aspects, the disclosure provides systems, devices, methods, and computer programming products, including non-transient machine-readable instruction sets, for use in implementing such methods and enabling the functionality described previously.
  • Although the disclosure has been described and illustrated in exemplary forms with a certain degree of particularity, it is noted that the description and illustrations have been made by way of example only. Numerous changes in the details of construction and combination and arrangement of parts and steps may be made. Accordingly, such changes are intended to be included in the invention, the scope of which is defined by the claims.
  • Except to the extent explicitly stated or inherent within the processes described, including any optional steps or components thereof, no required order, sequence, or combination is intended or implied. As will be will be understood by those skilled in the relevant arts, with respect to both processes and any systems, devices, etc., described herein, a wide range of variations is possible, and even advantageous, in various circumstances, without departing from the scope of the invention, which is to be limited only by the claims.

Claims (14)

What is claimed is:
1. A method comprising:
(a) a first computing device hosting a collaborative user interface development environment;
(b) a second computing device associated with a designer, that connects to the collaborative user interface development environment for composing a user interface using a plurality of user interface assets that encode the designer's intent; and
(c) a third computing device rendering an executable user interface, by compiling the user interface assets.
2. The method of claim 1, comprising encoding the user interface assets with one or more data components indicative of a designer's intent relative to a user interface project, such that the user interface assets are can be manipulated by a user in connection with a user interface design workflow, and can be executed by a rendering system.
3. The method of claim 1, comprising encoding the user interface assets with state transitions, while maintaining the ability to manipulate the user interface assets as part of an interface design workflow.
4. The method of claim 1 further comprising:
(a) decomposing a user interface design into a plurality of user interface components, including one or more presentation components and one or more behaviour related components;
(b) using a design tool to create one or more user interface controls, including one or more behavior controls, for the user inter face design; and
(c) adding behavior class names for each behaviour control of the user interface design.
5. The method of claim 4, comprising:
(a) exporting the user interface controls to asset files;
(b) importing the user interface controls into source code; and
(c) accessing or generating specific code elements for handling one or more user interface related business logic rules.
6. The method of claim 4, further comprising:
(a) exporting the user interlace controls to asset files;
(b) importing the user interface controls into source code
7. A computer system for streamlining the design of user interface elements comprising:
(a) one or more computers, executing one or more computer programs, that define a collaborative user interface development environment (the “platform”), wherein:
(i) the platform includes a composition utility that enables one or more users to create a series of user interface assets that are encoded with sufficient information to enable their rendering by a rendering utility; and
(ii) the rendering utility renders user interface elements using the user interface assets in a way that related effects are rendered accurately.
8. The system of claim 7, wherein the platform includes an import utility for importing video content related to a user interface design from a motion graphics platform or a visual graphics platform.
9. The system of claim 7, comprising an extractor that decomposes a user interface design into a series of user interface components, and wherein the composition utility enables one or more users to (A) create one or more user interface controls for a user interface design of the video content, and (B) add behavior class names for each behavior control.
10. The system of claim 9 wherein the extractor analyzes user interface components and determines which user interface components are related to user interface controls, and categorizes user interface controls so as to identify presentation related user interface controls, and behavior related user interface controls.
11. The system of claim 7, wherein the platform enables the creation of the assets by (A) exporting UI controls to an asset file, (B) importing UI controls into source code, and accessing or generating one or more code components to execute business logic related to user interface functions (“business logic”) by importing the user interface controls into source code, and writing or importing specific code into the source code to execute the business logic.
12. The system of claim 7, wherein the composition utility enables the one or more users to encode the user interface assets with state to state transitions, such that the assets can be manipulated in connection with a user interface design workflow and also executed by the rendering utility.
13. The system of claim 7, wherein the system includes or links to a computer system executing a motion graphics platform or a visual graphics platform, and also a plug-in for incorporating user interface design features into the motion graphics platform or visual graphic platform.
14. The computer system of claim 7, wherein the computer system comprises a display, and the composition utility presents on the display (A) a composition panel and (B) a timeline panel, wherein the composition panel and the timeline panel are integrated and allow a user to compose user interface elements, including controls, and use the timeline panel allows to apply effects to the user interface elements.
US13/933,351 2013-07-02 2013-07-02 System and method for streamlining user interface development Active 2033-09-01 US9858050B2 (en)

Priority Applications (6)

Application Number Priority Date Filing Date Title
US13/933,351 US9858050B2 (en) 2013-07-02 2013-07-02 System and method for streamlining user interface development
CA2922860A CA2922860C (en) 2013-07-02 2014-06-30 System and method for streamlining user interface development
PCT/CA2014/000530 WO2015000054A1 (en) 2013-07-02 2014-06-30 System and method for streamlining user interface development
EP14819710.6A EP3017364B1 (en) 2013-07-02 2014-06-30 System and method for streamlining user interface development
EP22196528.8A EP4130978A1 (en) 2013-07-02 2014-06-30 System and method for streamlining user interface development
US15/856,465 US20180203675A1 (en) 2013-07-02 2017-12-28 System and method for streamlining user interface development

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US13/933,351 US9858050B2 (en) 2013-07-02 2013-07-02 System and method for streamlining user interface development

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/856,465 Continuation US20180203675A1 (en) 2013-07-02 2017-12-28 System and method for streamlining user interface development

Publications (2)

Publication Number Publication Date
US20150012905A1 true US20150012905A1 (en) 2015-01-08
US9858050B2 US9858050B2 (en) 2018-01-02

Family

ID=52133694

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/933,351 Active 2033-09-01 US9858050B2 (en) 2013-07-02 2013-07-02 System and method for streamlining user interface development
US15/856,465 Abandoned US20180203675A1 (en) 2013-07-02 2017-12-28 System and method for streamlining user interface development

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/856,465 Abandoned US20180203675A1 (en) 2013-07-02 2017-12-28 System and method for streamlining user interface development

Country Status (4)

Country Link
US (2) US9858050B2 (en)
EP (2) EP3017364B1 (en)
CA (1) CA2922860C (en)
WO (1) WO2015000054A1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140129935A1 (en) * 2012-11-05 2014-05-08 Dolly OVADIA NAHON Method and Apparatus for Developing and Playing Natural User Interface Applications
CN110853121A (en) * 2019-10-25 2020-02-28 稿定(厦门)科技有限公司 Cross-platform data processing method and device based on AE
EP3559802A4 (en) * 2016-12-21 2020-07-29 You i Labs Inc. System and method for cloud-based user interface application deployment
CN112068909A (en) * 2020-07-31 2020-12-11 中核核电运行管理有限公司 Nuclear power operation rule mobile execution card generation system and method
CN112486483A (en) * 2020-11-23 2021-03-12 广州经传多赢投资咨询有限公司 UI framework implementation method, system, device and storage medium based on XML
CN112862668A (en) * 2021-02-01 2021-05-28 北京恒泰实达科技股份有限公司 Method for implementing picture conversion from design effect picture to visualization
CN113268236A (en) * 2021-05-20 2021-08-17 湖南大学 UI element picking method and system based on function enhancement technology in RPA system
CN113296759A (en) * 2021-05-12 2021-08-24 广州博冠信息科技有限公司 User interface processing method, user interface processing system, user interface processing device, and storage medium
CN113590103A (en) * 2020-04-30 2021-11-02 深圳中砼物联网科技有限公司 Method, computer device and storage medium for realizing graphical code-free development user interface

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108021367B (en) * 2017-12-15 2020-11-17 广州赛意信息科技股份有限公司 UI development system and method based on metadata framework
CN109710258A (en) * 2018-12-28 2019-05-03 北京金山安全软件有限公司 WeChat applet interface generation method and device
CN110569096B (en) * 2019-08-20 2022-10-18 上海沣沅星科技有限公司 System, method, medium, and apparatus for decoding human-computer interaction interface
CN111258693B (en) * 2020-01-13 2024-04-09 奇安信科技集团股份有限公司 Remote display method and device
CN113867694B (en) * 2021-09-27 2022-06-24 上海汇付支付有限公司 Method and system for intelligently generating front-end code
US11861329B2 (en) * 2022-01-18 2024-01-02 Dell Products, L.P. Method and apparatus for codifying user experience designs and managing the codified user experience designs
CN114845173B (en) * 2022-05-30 2023-08-22 杭州国芯科技股份有限公司 Main menu generation method of set top box

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001095102A1 (en) * 2000-06-08 2001-12-13 Hypnotizer Method for structuring, transferring and interpreting a data set for designing graphic interfaces
US20040205515A1 (en) * 2003-04-10 2004-10-14 Simple Twists, Ltd. Multi-media story editing tool
US7441200B2 (en) * 2002-02-01 2008-10-21 Concepts Appsgo Inc. Method and apparatus for designing, rendering and programming a user interface
US20120174000A1 (en) * 2010-12-31 2012-07-05 Verizon Patent And Licensing, Inc. Automated graphical user interface design and development systems and methods
US8281238B2 (en) * 2009-11-10 2012-10-02 Primal Fusion Inc. System, method and computer program for creating and manipulating data structures using an interactive graphical interface
US8856667B2 (en) * 2005-04-19 2014-10-07 The Mathworks, Inc. Graphical state machine based programming for a graphical user interface
US8982132B2 (en) * 2011-02-28 2015-03-17 Adobe Systems Incorporated Value templates in animation timelines

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6747651B1 (en) * 1998-07-18 2004-06-08 National University Of Singapore System and method for creating bounding volume hierarchies utilizing model simplification
EP1331564A1 (en) * 2002-01-24 2003-07-30 Siemens Aktiengesellschaft Fuzzy logic based intelligent load control for distributed environment
US8963847B2 (en) 2010-12-06 2015-02-24 Netflix, Inc. User interface for a remote control device
US9098294B2 (en) 2010-12-31 2015-08-04 Verizon Patent And Licensing Inc. Building block based graphical user interface design and development systems and methods
US9152539B2 (en) 2011-08-03 2015-10-06 Verizon Patent And Licensing Inc. Tag-based graphical user interface production systems and methods

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001095102A1 (en) * 2000-06-08 2001-12-13 Hypnotizer Method for structuring, transferring and interpreting a data set for designing graphic interfaces
US7441200B2 (en) * 2002-02-01 2008-10-21 Concepts Appsgo Inc. Method and apparatus for designing, rendering and programming a user interface
US20040205515A1 (en) * 2003-04-10 2004-10-14 Simple Twists, Ltd. Multi-media story editing tool
US8856667B2 (en) * 2005-04-19 2014-10-07 The Mathworks, Inc. Graphical state machine based programming for a graphical user interface
US8281238B2 (en) * 2009-11-10 2012-10-02 Primal Fusion Inc. System, method and computer program for creating and manipulating data structures using an interactive graphical interface
US20120174000A1 (en) * 2010-12-31 2012-07-05 Verizon Patent And Licensing, Inc. Automated graphical user interface design and development systems and methods
US8982132B2 (en) * 2011-02-28 2015-03-17 Adobe Systems Incorporated Value templates in animation timelines

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140129935A1 (en) * 2012-11-05 2014-05-08 Dolly OVADIA NAHON Method and Apparatus for Developing and Playing Natural User Interface Applications
US9501140B2 (en) * 2012-11-05 2016-11-22 Onysus Software Ltd Method and apparatus for developing and playing natural user interface applications
EP3559802A4 (en) * 2016-12-21 2020-07-29 You i Labs Inc. System and method for cloud-based user interface application deployment
US10833956B2 (en) 2016-12-21 2020-11-10 You I Labs Inc. System and method for cloud-based user interface application deployment
US11770308B2 (en) 2016-12-21 2023-09-26 WarnerMedia Direct, LLC System and method for cloud-based user interface application deployment
CN110853121A (en) * 2019-10-25 2020-02-28 稿定(厦门)科技有限公司 Cross-platform data processing method and device based on AE
CN113590103A (en) * 2020-04-30 2021-11-02 深圳中砼物联网科技有限公司 Method, computer device and storage medium for realizing graphical code-free development user interface
CN112068909A (en) * 2020-07-31 2020-12-11 中核核电运行管理有限公司 Nuclear power operation rule mobile execution card generation system and method
CN112486483A (en) * 2020-11-23 2021-03-12 广州经传多赢投资咨询有限公司 UI framework implementation method, system, device and storage medium based on XML
CN112862668A (en) * 2021-02-01 2021-05-28 北京恒泰实达科技股份有限公司 Method for implementing picture conversion from design effect picture to visualization
CN113296759A (en) * 2021-05-12 2021-08-24 广州博冠信息科技有限公司 User interface processing method, user interface processing system, user interface processing device, and storage medium
CN113268236A (en) * 2021-05-20 2021-08-17 湖南大学 UI element picking method and system based on function enhancement technology in RPA system

Also Published As

Publication number Publication date
EP3017364A1 (en) 2016-05-11
EP3017364A4 (en) 2017-03-29
US20180203675A1 (en) 2018-07-19
CA2922860C (en) 2021-11-02
EP4130978A1 (en) 2023-02-08
US9858050B2 (en) 2018-01-02
CA2922860A1 (en) 2015-01-08
WO2015000054A1 (en) 2015-01-08
EP3017364B1 (en) 2022-09-21

Similar Documents

Publication Publication Date Title
US20180203675A1 (en) System and method for streamlining user interface development
US7836086B2 (en) Layering and referencing of scene description
KR101087427B1 (en) A computer implemented method and computer readable recording medium for integrating three dimensional scene hierarchy into two dimensional compositing system
Parisi Programming 3D Applications with HTML5 and WebGL: 3D Animation and Visualization for Web Pages
CA2795739C (en) File format for representing a scene
US10803647B1 (en) Generating animation rigs using scriptable reference modules
US9159168B2 (en) Methods and systems for generating a dynamic multimodal and multidimensional presentation
Pratama et al. Workflow in virtual reality tool development for AEC industry
US8902234B1 (en) Simulation primitives
US11625900B2 (en) Broker for instancing
Conlan The blender python API: Precision 3D modeling and add-on development
Gao et al. [Retracted] Realization of Music‐Assisted Interactive Teaching System Based on Virtual Reality Technology
US7917535B1 (en) Task membership and task masks
Stenning Direct3D Rendering Cookbook
Álvarez et al. EZ3. js: A robust and flexible WebGL-based 3D engine
Hering et al. 3DCIS: A real-time browser-rendered 3d campus information system based on webgl
US9569875B1 (en) Ordered list management
Conlan The Blender Python API
Capellman et al. MonoGame Architecture
Podila et al. WPF control development unleashed: Building advanced user experiences
Sunyer Caldú BeEngine: 2D Game Engine
Alizadeh Design and Implementation of a Web-Based Editor Optimized for Online Gambling Games
CN115205430A (en) 3D file importing and exporting method and device
Olsson et al. The Essential Guide to 3D in Flash
Jackson et al. Coding Gameplay: Set Up Gameplay Methods and Animated Camera View

Legal Events

Date Code Title Description
AS Assignment

Owner name: YOUI LABS INC., CANADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EMMONS, ANDREW T.;REEL/FRAME:030955/0749

Effective date: 20130807

AS Assignment

Owner name: COMERICA BANK, CANADA

Free format text: SECURITY INTEREST;ASSIGNOR:YOU I LABS INC.;REEL/FRAME:041986/0467

Effective date: 20170222

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: YOU I LABS INC., CANADA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED ON REEL 030955 FRAME 0749. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:EMMONS, ANDREW T.;REEL/FRAME:047376/0373

Effective date: 20130807

CC Certificate of correction
AS Assignment

Owner name: YOU I LABS INC., CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:COMERICA BANK;REEL/FRAME:052022/0108

Effective date: 20191029

CC Certificate of correction
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: YOU I LABS INC., CANADA

Free format text: MERGER AND CHANGE OF NAME;ASSIGNOR:YOU I LABS INC;REEL/FRAME:058301/0529

Effective date: 20210401

Owner name: YOU I LABS INC., CANADA

Free format text: MERGER AND CHANGE OF NAME;ASSIGNORS:YOU I LABS INC;YOU I LABS INC;REEL/FRAME:058301/0529

Effective date: 20210401

Owner name: WM INTERACTIVE MEDIA HOLDINGS INC., NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YOU I LABS INC.;REEL/FRAME:058281/0876

Effective date: 20211031

Owner name: YOU I LABS INC., CANADA

Free format text: CHANGE OF NAME;ASSIGNOR:YOU I LABS INC.;REEL/FRAME:058281/0719

Effective date: 20210329

AS Assignment

Owner name: WARNERMEDIA DIRECT, LLC, NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WM INTERACTIVE MEDIA HOLDINGS INC.;REEL/FRAME:058824/0195

Effective date: 20220101