AU2005203582A1 - User interface for authoring animated transitions - Google Patents

Description

S&FRef: 717710

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address of Applicant: Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3-chome, Ohta-ku, Tokyo, 146, Japan Laurence Crew Jonathon Anthony Duhig Zdzislaw Sliwinski Spruson Ferguson St Martins Tower Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) User interface for authoring animated transitions The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5845c N' USER INTERFACE FOR AUTHORING ANIMATED TRANSITIONS Field of the Invention The present invention relates generally to the authoring of multimedia and software user interfaces. Specifically, the invention relates to the authoring of user 00 In interface widgets based on animated state machines.

C€3 S 5 Background SGraphical user interfaces for computer software programs are a well known means for allowing users to interact with the underlying functions of a software program, without the need to memorise commands, for example.

Graphical user interfaces commonly include of collections of 'widgets' (also referred to as 'interactors', 'interaction objects' and 'user interface controls'). Each widget occupies an area upon a display screen and performs a particular function of user interaction. For example, a widget may be an iconic button for initiating an action, or a text entry box for entering data.

In complex graphical user interface, many widgets, alter their presentation in terms of shape and/or colour depending upon the particular selection or stage of selection that has been made. An example of this is shown in Fig. 1 where a "button" widget is initially presented as a square and after a first selection becomes a striped triangle. A further selection causes the widget to alter into a coloured circle. Such changes are typically instantaneous and derive from the user interface detecting the selection and updating the display with a new graphical object to replace that which was formerly displayed.

It is often desirable to animate transitions. One approach is to create a sequence of graphical objects that progressively alter their shape thus affording "cartoon" style of 717710 N animation from an initial shape to a final shape. The progression may be used in either ;direction. An example of this is shown in Fig. 2 for a transition from a square to a circle and vice versa. Colour transitions may be similarly applied to the graphical objects to further enhance the visual appearance, as depicted in Fig. 3.

00 t 5 The alteration of graphical objects as shown in Figs. 1 to 3 may be performed using any one of numerous graphical drawing software packages and editing tools well known in the art. The designer of the graphical user interface has still to incorporate those graphical objects into the user interface in a convenient manner such that the authoring of the user interface facilitates the myriad of options that may be desired in the user interface. Further the authoring in addition to affording the desired functionality, should promote ease of selection and manipulation of the graphical objects so as to enhance the aesthetic appeal, particularly through the application of animation and the management of animation parameters.

It follows that the authoring of graphical user interface can be an iterative process requiring many editing changes during the course of development which may be time consuming and frustration if the authoring tool is not flexible and easy to use.

Summary of the Invention In accordance with one aspect of the present invention there is disclosed a method for authoring an animated transition between states, the method comprising the steps of: displaying a representation of at least one transition between at least two states wherein the at least one transition connects two of the states; selecting one transition from the representation; displaying an animation timeline associated with the selected transition; and 717710 Nediting the animation timeline for the selected transition to thereby author the ;animated transition between the states.

Generally, the at least two states are displayed concurrently with the representation of the at least one transition.

00 In accordance with another aspect of the present invention there is disclosed a N method for authoring an animated transition between states, the method comprising the steps of: (Ni displaying a diagram having at least two states and at least one animation timeline where each animation timeline connects two of the states and relates to at least one animated transition connecting the two states; selecting an animation transition from the diagram; and editing an animation corresponding to the selected transition using an animation timeline associated with the transition to thereby author the transition.

The method may further comprise, intermediate steps and the further step of: (ba) rotating the diagram so that the associated animation timeline is at least substantially horizontally aligned. The diagram may be represented in three dimensions.

Desirably the states comprise states of operation of a graphical user interface corresponding to a display of a particular graphical form as part of the interface, and the transitions comprise alteration of the interface from one particular graphical form to another. Each state may have associated therewith a corresponding graphical state frame and the animation timeline defines a temporal display of an animated transition from one the graphical state frame of one state to another the graphical state frame of the other state of the selected transition.

717710 -4- N In accordance with another aspect of the present invention there is provided a ;method of authoring a graphical user interface for use in operating a device, the graphical user interface being displayable in operation upon a display associated with the device and including a plurality of widgets an interaction with which represents a state of 0 5 operation of the graphical user interface for control of the device, the method comprising the steps of: Sdisplaying a state transition diagram including a graphical representation of each state and a graphical representation of each transition interconnecting the states and defining operation of the graphical user interface being authored, each transition being formed between a corresponding two of the states; detecting a selection of one transition; displaying, in response to the selection, an animation timeline associated with the selected transition; editing an animation associated with the displayed animation timeline to thereby author the graphical user interface.

Advantageously, the animation may comprise a temporal alteration of the graphical user interface as represented upon the display of the device between a first state associated with one widget of the selected transition to a second state of the selected transition and the editing comprises altering the displayed animation timeline to thereby change the temporal alteration of the graphical user interface.

Other aspects of the invention, including apparatus for implementations, a computer program implementation and a device incorporating a GUI so formed, are also disclosed.

717710 SBrief Description of the Drawings ;At least one embodiment of the present invention and aspects of the prior art are described with reference to the drawings in which: Fig. 1 illustrates the successive animation of a widget;

(N

0 5 Fig. 2 illustrate animated change of a shape; SFig. 3 show the change of Fig. 2 with the addition of change in colour; SFig. 4A is a schematic diagram showing a state machine, Fig. 4B a schematic diagram showing a state-frame graph; Fig. 4B a schematic diagram showing a state-frameion graph; Fig. 4C a schematic diagram showing an animation graph; Fig. 4D a diagram showing the data objects and relationships used to implement state machines, state-frame graphs and animation graphs; Fig. 5 is a flowchart illustrating the process of determining the state-frame graph from a state machine and for initialising the animation graph; Fig. 6 illustrates a series of visual appearances of a widget corresponding to an animation sequence at various points in time; Fig. 7 is a user interface diagram illustrating a user interface for authoring an animation graph for a widget including a state-frame graph in list form and an animation timeline; Fig. 8 is a user interface diagram illustrating a user interface for authoring an animation graph for a widget including a state-frame graph in diagram form and an animation timeline; Fig. 9A is a user interface diagram illustrating a user interface for authoring an animation graph for a widget including a combined state-frame graph and multiple animation timelines in two dimensions; 717710 i Fig. 9B illustrates the rotation of an animation graph user interface in response to tb3 ;the selection of a new timeline; Fig. 10 is a drawing of a node graph in two dimensions having four nodes fully connected to each other by non-crossing bars;

(N

00 t 5 Figs. 11 A 11E show, in both orthogonal projection and perspective projection, 0 representations of a three dimensional node graph structure having six nodes fully Sconnected by non-intersecting bars; Figs. 12A-12D show, in both orthogonal projection and perspective projection, representations of a three dimensional node graph structure having seven nodes fully connected by non-crossing bars;.

Fig. 13 is a user interface diagram illustrating a user interface for authoring an animation graph for a widget including a combined state-frame graph and miltiple animation timelines in three dimensions in perspective view; Fig 14 illustrates a consumer device including a user interface authored according to the present disclosure; Fig. 15 shows an animation timeline according to the present disclosure; and Fig. 16 is a schematic block diagram representation of a general purpose computer system upon which the arrangements described herein may be practiced.

Detailed Description including Best mode Animating between the states of a user interface widget provides the ability for a smoother transition between states, which can lead to a more pleasurable user experience. Another benefit is that animations attract the eye of a user, alerting them to a change in state, through providing start and finish references for the change, this being 717710 N more effective than an immediate or non-animated state change. Also, animation allows ;for greater creative expression leading to a greater diversity of effects.

_Producing effective animated graphics however requires a skilled designer and usually involves a process of iterative testing and refinement. Therefore it is an 00 V 5 advantage to have an authoring tool that allows a designer with little or no programming skills to create working animated widgets and test them during the design process, and ultimately to provide finished animated widget designs for incorporation in a final product, without the risks associated with reinterpreting or re-implementing the design during the product implementation phase.

Presently disclosed is a graphical user interface (GUI) for an authoring tool for designing, specifying and editing animated state based widgets without the need for programming.

The GUI and the authoring tool are preferably practiced or implemented using a general-purpose computer system 1600, such as that shown in Fig. 16 wherein the processes of Figs. 4A to 15 to be described may be implemented as software, such as one or more application programs executing within the computer system 1600. In particular, the user interface and authoring steps are effected by instructions in the software that are carried out by the computer. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part performs the authoring methods and a second part manages the user interface between the first part and the user. The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer from the computer readable medium, and then executed by the computer. A computer readable 717710 1 medium having such software or computer program recorded on it is a computer ;program product. The use of the computer program product in the computer preferably effects an advantageous apparatus for designing and specifying animated state based widgets without the need for programming.

00 The computer system 1600 is formed by a computer module 1601, input devices O such as a keyboard 1602 and mouse 1603, output devices including a printer 1615, a display device 1614 and loudspeakers 1617. A Modulator-Demodulator (Modem) transceiver device 1616 is used by the computer module 1601 for communicating to and from a communications network 1620, for example connectable via a telephone line 1621 or other functional medium. The modem 1616 can be used to obtain access to the Internet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN), and may be incorporated into the computer module 1601 in some implementations.

The computer module 1601 typically includes at least one processor unit 1605, and a memory unit 1606, for example formed from semiconductor random access memory (RAM) and read only memory (ROM). The module 1601 also includes an number of input/output interfaces including an audio-video interface 1607 that couples to the video display 1614 and loudspeakers 1617, an I/O interface 1613 for the keyboard 1602 and mouse 1603 and optionally a joystick (not illustrated), and an interface 1608 for the modem 1616 and printer 1615. In some implementations, the modem 16116 may be incorporated within the computer module 1601, for example within the interface 1608. A storage device 1609 is provided and typically includes a hard disk drive 1610 and a floppy disk drive 1611. A magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 1612 is typically provided as a non- 717710 -9i volatile source of data. The components 1605 to 1613 of the computer module 1601, 01) ;typically communicate via an interconnected bus 1604 and in a manner which results in a conventional mode of operation of the computer system 1600 known to those in the relevant art. Examples of computers on which the described arrangements can be oO 5 practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer Vt) C€3 systems evolved therefrom. The audio capable devices 1607 and 1617 enable sounds to Sattached to and reproduced with animations and the like, if desired.

Typically, the application program is resident on the hard disk drive 1610 and read and controlled in its execution by the processor 1605. Intermediate storage of the program and any data fetched from the network 1620 may be accomplished using the semiconductor memory 1606, possibly in concert with the hard disk drive 1610. In some instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk and read via the corresponding drive 1612 or 1611, or alternatively may be read by the user from the network 1620 via the modem device 1616. Still further, the software can also be loaded into the computer system 1600 from other computer readable media. The term "computer readable medium" as used herein refers to any storage or transmission medium that participates in providing instructions and/or data to the computer system 1600 for execution and/or processing. Examples of storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 1601. Examples of transmission media include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and 717710 N the Internet or Intranets including e-mail transmissions and information recorded on ;Websites and the like.

_Prior to describing the user interface and authoring tool, an explanation may be desirable of the related concepts of a state machine, a state-frame graph and an animation

(N

00 graph for animated state based widgets. The state machine, state-frame graph and an animation graph are all types of node graphs, but with different features and functions within the context of the present disclosure.

The purpose of a state machine in the present disclosure is to define and control the functional aspects of a widget. A state machine comprises one or more states and zero or more transitions. At any time, exactly one state of a state machine is the 'current state'. Each state can be linked to any other state in one direction by a transition. In addition to defining a starting state and an ending state, each transition will typically define a 'trigger', which is an incoming condition or signal, based on which the state machine will initiate the transition. A transition may additionally define an 'event', which is an outgoing signal that is initiated when the transition occurs. An event may be used to control functions external to the widget such as other widgets or functions in the device. An example is the initiation of a copying process in a photocopier, based on an event initiated when the 'copy' button of a user interface on the photocopier transitions from the 'up' to the 'pressed' state. A state machine is typically implemented in software with flags for example representing various states and/or event. In some instances, a state machine may be implemented in hardware through the setting of one or more registers and logic interconnecting the registers to define the mechanics of state changes as desired by the user.

717710 -11- Fig. 4A is a schematic diagram representing a state machine 400. States 401, 402 ;and 403 are functional states of a user interface widget. Transitions 404 409 connect the states to each other, each transition connecting two of the states in one direction.

Triggers 410 415 are associated with each of the transitions 404 409 respectively.

00 tr 5 It can be noted that there are two transitions joining state 401 to state 402 in the C€3 same direction, being transitions 404 and 406. Although these transitions have the same Sstarting and ending states (and have the same direction), they may be differentiated from each other by having different triggers, being the triggers 410 and 411 respectively, and may have different events associated with them (not shown).

When a trigger condition occurs, and the trigger is associated with a transition of a state machine, and the starting state of the transition is the current state of the state machine, the transition will be initiated and the current state of the state machine will be set to the ending state of the transition. The change in state occurs immediately.

The purpose of a state-frame graph is to define the visual appearance of a widget at each of its functional states, and the high level animation structure between those states. A state-frame graph includes state-frames, which generally mirror the states of a corresponding state machine, and animations, which are weighted arcs joining the stateframes to one another. The animations generally form a subset of the transitions of the corresponding state machine. However, animations are functionally disparate from state transitions and may exist between state frames where there is no corresponding transition in the associated state machine.

Fig. 4B is a schematic diagram showing a state-frame graph 420 corresponding to state machine 400. State-frames 421, 422 and 423 correspond to states 401, 402 and 403 respectively of the state machine 400. However, instead of representing functional states 717710 -12i of a widget, the state-frames represent a visual appearance of the widget when it is in the ;corresponding state.

_The state-frames 421, 422 and 423 are connected by animations 424 429. In a specific implementation, there is only one animation between any two state-frames in

(N

00 V 5 any direction regardless of how many transitions connect the corresponding states of the e¢3 Sstate machine. For example, whilst states 401 and 402 are joined by both transitions 404 Sand 406 in the same direction, corresponding state frames 421 and 422 are joined by only one animation in that direction, being animation 425 in this example. The reason for this is that there are no triggers associated with the animations and triggers are the differentiating feature of transitions that have the same starting and ending states. In an alternate implementation each animation is associated with a transition (and hence a trigger) allowing more than one animation between two state-frames in the same direction.

State machine transitions, which change the state of the state machine immediately and have no temporal or other linear value associated with them, may be said to be unweighted arcs. In contrast, animations, of the state-frame graph have an animation time associated with them, and as such may be said to be weighted arcs.

The animation from one state-frame to another follows the transition of the corresponding states during normal operation. However in some situations it may be necessary to cut short the animation between states to immediately display to the user a new state. That is, the new state may need to be displayed without waiting for the time delay that is introduced during state transition animation. This may be important where displaying the new state would require transition through a number of other states as no direct animation exists. In this situation bypassing the animation may be appropriate.

717710 -13cI An animation graph corresponds to a state-frame graph. The purpose of an ;animation graph is to define the low level or detailed animation structure of the visual appearance of a widget. An animation graph includes the state-frames from the corresponding state-frame graph, but each animation is replaced by an animation 00 S 5 sequence having one or more animation intervals and zero or more keyframes in addition C€3 O to the state-frames associated with the animation.

(Ni SKeyframes represent the visual appearance of a widget at an instant in time when the animation sequence passes through the keyframe. Animation intervals each define the animation time between adjacent state-frames and/or keyframes as well as an animation function.

Fig. 4C is a schematic diagram showing an animation graph 430 corresponding to the state machine 400 and the state-frame graph 420. The animation graph 430 shares the state-frames with the corresponding state-frame graph 420, i.e. the state-frames 421, 422 and 423.

As seen in Fig. 4C, an animation interval 434 connects the state-frame 421 with a keyframe 435. Also an animation interval 436 connects the keyframe 435 with a keyframe 437 in turn, and an animation interval 438 connects a keyframe 437 to the state-frame 422. Collectively, the animation intervals 434, 436 and 438, the keyframes 435 and 437 and the state-frames 421 and 422 comprise an animation sequence corresponding to the animation 425 of the state-frame graph 420 of Fig. 4B. Similarly, animation intervals 433 and 431, keyframe 432 and state-frames 422 and 421 collectively comprise an animation sequence corresponding to the animation 424, whilst animation interval 442 and state-frames 423 and 421 comprise an animation sequence corresponding to the animation 429, with no keyframes used in this animation sequence.

717710 -14- C1 Each animation interval has associated with it an animation time, illustrated in ;Figs. 4B and 4C as a numerical value positioned in the animation intervals connecting state-frames and keyframes, and an animation function (not shown). The sum of the animation times of all of the animation intervals comprising a single animation sequence 0t 5 will be equal to the animation time of the corresponding animation in the state-frame ¢€3 0 graph. For example, the sum of the animation times of animation intervals 434, 436 and S438 (seen in Fig. 4C) is: (400 150 450)= 1000 which is equal to the animation time of the corresponding animation 425 (seen in Fig. 4B).

The animation function associated with an animation interval is a mapping function between elapsed time (as a proportion of the animation time of the animation interval) and the visible progress of the animation. Common animation functions are linear, step-wise and exponential.

A linear animation function is in the form: f(t) t= 1] A step-wise animation function is in the form: q, t q <=1 An exponential animation function is in the form: f(t) t n t 1] In these functions: t is elapsed time as a proportion of the animation time of the animation interval, q is the threshold value for a step-wise animation function, and n is the exponent of an exponential animation function.

717710 15 F-AUsing an exponential function with an exponent greater than 1, for example n 2, results in an effect commonly known as 'ease in', whilst using an exponent less Sthan 1 for example n 1/2 results in an effect commonly known as 'ease out'.

Animation intervals 439 and 441, shown with arrows at both ends, and keyframe

(N

00 t 5 440 collectively comprise an animation sequence corresponding to both animations 427 C€3 O and 428, being animations in both directions between state frames 422 and 423. This Sanimation sequence is referred to as a 'bi-directional animation sequence' and is played or reproduced in either direction depending on the direction of the transition of the corresponding states of the state machine.

Fig. 4D is a diagram showing the data objects and relationships used to implement state machines, state-frame graphs and animation graphs. Each State Machine object 450 is related to one or more State objects 451 and zero or more Transition objects 452. Each State Machine object 450 also has a property 'current state' which references one of the related State objects. Each Transition object 452 has properties 'start' and 'end' each of which references one of the related State objects (451); as well as trigger and event properties.

Each State-frame Graph object 453 is -related to one or more State-frame objects 454 and zero or more Animation objects 455. Each State-frame object 454 references one State object 451, and has an 'appearance' property which is a reference to a set of parameters defining a visual appearance for the widget (not shown). Each Animation object 455 has properties 'start' and 'end', each of which references one of the related State-frame objects (454), as well as an 'animation time' property.

Each Animation Graph object 456 is related to zero or more Animation Sequence objects 457; each of which in turn is related to zero or more Keyframe objects 458 and 717710 -16- 1 one or more Animation Interval objects 459. Each Keyframe object 458 has an ;'appearance' property which is a reference to a set of parameters defining a visual appearance for the widget (not shown). Each Animation Interval object 459 has properties 'start' and 'end' each of which references either one of the related Keyframe 00 objects (458) or one of the related State-frame objects (454) O During the authoring process for an animated state-based widget the state Smachine is pre-defined, the state-frame graph is determined from the state machine by the process described below, and the animation graph is authored using the user interface described herein.

Fig. 5 is a flowchart illustrating the process 500 of determining the state-frame graph from a state machine and for initialising the animation graph. The process 500 is preferably implemented as computer program stored upon and executable within the computer module 1601 of Fig. 16. The process 500 occurs the first time a new widget is created based on a pre-defined state machine. The process 500 begins with an entry point at step 501, for example when called by another program. At step 502, an empty state-frame graph is created. Step 503 then adds to the state-frame graph a state-frame corresponding to each state of the state machine. At step 504, an animation is added to the state-frame graph between every pair of state-frames, where at least one transition exists between the corresponding pair of states in the state machine. Step 504 is performed twice for each pair of state-frames once in each direction. At step 505, the animation time for each animation in the state-frame graph is set to a default animation time, being 1000ms in a preferred implementation.

At step 506, an empty animation graph is created. At step 507, a state-frame is added to the animation graph corresponding to each state-frame in the state-frame graph.

717710 -17- N Step 508 follows where an animation interval is added to the animation graph ;corresponding to each of the animations in the state-frame graph. The animation time of each animation interval is set equal to the animation time of the corresponding animation from the state-frame graph. At step 509, the animation function for each animation

(N

00 5 interval in the animation graph is set to a default animation function, being a linear O function in the preferred implementation. At step 510 the process 500 ends.

(Ni Fig. 6 illustrates a series of visual appearances of a widget corresponding to an (Ni animation sequence at various points in time. For example, visual appearance 601 might correspond to the state-frame 422 visual appearance 606 correspond to the state-frame 423 ("DOWN") and visual appearance 604 correspond to the keyframe 440 that lies between the state-frames 422 and 423 on the animation graph illustrated in Fig.

4C. Visual appearances corresponding to either state-frames or keyframes are defined by the user using the drawing tools associated with the user interface (not shown).

Visual appearances 602, 603 and 605 represent 'in-between' appearances and are generated by interpolating between two user-defined appearances. Interpolation may occur simultaneously on any number of parameters of the visual appearance of the widget, including but not limited to shape, colour, line thickness, scale and rotation.

For example, visual appearances 602 and 603 are two stages of interpolation of shape between visual appearances 601 and 604, each occurring at some point in time as the widget appearance animates smoothly (morphologically changes) between stateframes 422 and 423. Visual appearance 605 is an interpolation of colour between visual appearances 604 and 606.

The interpolation of a visual appearance based on one or more parameters is performed according to the following equation: 717710 -18- A, b, [ao t'(al ao), bo t'(bi bo), co t'(cl t' t 1] ;where: visual appearance A, is defined by parameter values a, b, at time t; time t is elapsed time as a proportion of the animation time of an animation t 5 interval where t=O at the start of the animation interval and t=l at the end of the animation interval; and S-f(t) is the animation function associated with the animation interval Each of the parameters a, b, c etc may for example be representative of an x or y coordinate of a particular, point with a plurality of points, for example in a twodimensional image space, such as that reproducible upon the display 1614 via the graphical user interface, which may now be described.

Fig. 7 illustrates a user interface 700 for authoring an animation graph for a widget including a state-frame graph in list form and an animation timeline. Editing the timeline, amongst other editing of the animation graph, results in an alteration to the presentation of the widgets corresponding to the various states.

In this implementation, the state-frame graph is represented as an indented list of frames and transitions, rather than in diagrammatic form. The advantage of the list form of state-frame graph is that display of the state-frame graph is then relatively simple to implement and use compared to diagrammatic forms, even for complex graphs.

The user interface 700 includes a state-frame graph window 701, an animation timeline window 702 and a visual appearance window 725. The contents of the three windows 701, 702 and 725 together comprise user manipulable information and are used in conjunction with each other, although at certain times whilst using the user interface 700 any of the windows 701, 702 and 725 may be rendered non-visible.

717710 -19- The state-frame graph window 701 contains a state-frame graph user interface ;703. The state-frame graph user interface 703 contains an indented list with primary items 704, 705 and 706 representing the state-frames of the state-frame graph 421, 422 and 423) and indented items 707, 708, 709, 710 and 711 representing animations of the state-frame graph 424, 425, 427, 428 and 429).

O The state-frames 704, 705 and 706 represented in the state-frame graph user interface 703 are selectable. When a state-frame is selected, the visual appearance of the widget 726 in the visual appearance window 725 is updated to show the appearance associated with the selected state-frame. Additionally the visual appearance of the selected state-frame can be edited using graphical editing tools (not shown).

Each animation is represented in the state frame graph user interface 703 by the name of the ending state-frame of the animation, indented underneath the starting stateframe of the animation. For example, the animation represented at 707 begins at stateframe 704 and ends at state-frame 705 ("OVER") whilst the animation represented at 709 begins at state-frame 705 ("OVER") and ends at state-frame 704 The animations represented in the state-frame graph user interface 703 are selectable. Animation 707 is shown selected as indicated by the border around it.

The animation timeline window 702 contains an animation timeline 712, animation function selector 721 and bi-directional animation button 722.

The animation timeline 712 contains state-frame icons 713 and 714 which correspond respectively with the starting and ending state-frames of the selected animation, being state-frames 704 and 705 respectively associated with selected animation 707 in this case. Between the state-frame icons 713 and 714 may be shown zero or more keyframe icons, in this example icons 715 and 716. Between each two 717710 N, adjacent state-frames and or keyframe icons is an animation interval, in this example ;the intervals 718 and 720.

The state-frames as represented by icons 713 and 714 on the animation timeline 712 are selectable. When a state-frame is selected, the visual appearance of the widget 00 726 in the visual appearance window 725 is updated to show the appearance associated ¢€3 with the selected state-frame. Additionally the visual appearance of the selected state- Sframe can be edited using graphical editing tools (not shown).

The keyframes as represented by keyframe icons e.g. 715 and 716 on the animation timeline are selectable. When a keyframe is selected, the visual appearance of the widget 726 in the visual appearance window 725 is updated to show the appearance associated with the selected keyframe. Additionally the visual appearance of the selected keyframe can be edited using graphical editing tools (not shown).

Label 717 indicates the animation time of the selected animation, in this particular example, 1000 milliseconds. In addition, on or near each of the animation intervals e.g. 718 is a label e.g. 719 showing the animation time for the animation interval.

By selecting the animation time label 717 using the mouse 1603 and entering a new value by typing upon the keyboard 1602 or by other means, the animation time of the selected animation may be edited or altered. When the animation time of the selected animation is altered, the animation times of all of the animation intervals (e.g.

718, 720 etc.) are adjusted in proportion to the change in the animation time, so as to maintain equality between the animation time of the selected animation and the sum of the animation times of all of the animation intervals.

717710 -21- Fi By selecting an animation interval label such as the label 719 on animation ;interval 718 and entering a new value by typing or by other means, the animation time Sfor the animation interval e.g. 718 may be altered. When the animation time of an animation interval is altered, the animation time of the selected animation as shown at

(N

00 717 is adjusted according to the change in the animation time of the animation interval, Sso as to maintain equality between the animation time of the selected animation and the Ssum of the animation times of all of the animation intervals.

New keyframes can be added to an animation interval e.g. 718 on the animation timeline. This is initiated by selecting an animation interval using the mouse 1603 then executing a command from a menu, by a key sequence or other means. When a new keyframe is added to an animation interval the animation interval is split into two (2) new animation intervals, with the total animation time of the new animation intervals being equal to the animation time of the original animation interval. In a preferred implementation this is calculated by dividing the original animation time infi half.

Alternatively, this may be calculated by measuring the location on the display screen 1614 of a cursor pointer, associated with the mouse 1603 for example, in relation to the on-screen location of the original animation interval at the time of selection of the animation interval or initiation of the command to add a new keyframe) and proportionally dividing the original animation time accordingly.

Any of the keyframes e.g. 715 and 716 can be moved within the animation timeline by dragging them with the mouse 1603 or by similar means, thereby altering the animation times of the two animation intervals that are adjacent to the keyframe. For example, dragging the keyframe icon 716 will alter the animation times of animation intervals 718 and 720. Any change to the animation time of the first animation interval 717710 -22iF is offset by an opposite change to the animation time of the second animation interval, ;thereby maintaining the overall animation time.

Any of the keyframes e.g. 715 and 716 can be deleted from the animation sequence by selecting the keyframe icon then executing a command from a menu, by a S 5' key sequence or other means. When a keyframe is deleted, the two animation intervals ¢€3 adjacent to the keyframe are merged into a single animation interval, with the animation Stime of the animation interval being set to the sum of the animation times of the original two animation intervals. For example, deleting keyframe 716 will result in animation intervals 718 and 720 being merged into a single animation interval with an animation time of 600ms.

The animation intervals e.g. 718, 720 in the timeline can be selected. Animation interval 720 is shown selected as indicated by the thick border around it. Animation function selector 721 applies to the currently selected animation interval. Animation function selector 721 allows the choice of a range of options corresponding to predetermined animation functions for the selected animation interval. For example the options may include "linear", "step-wise", "ease in" and "ease out". Each animation interval in an animation sequence may have its own animation function.

Bi-directional animation button 722 has two states, 'on' and 'off, and is shown in the 'off state in Fig. 7. If the bi-directional animation button 722 is set to the 'on' state, the same animation sequence will then be used for both animations between two state-frames in a state-frame graph, with the animation being reversed when played in the opposite direction. For example, the animation sequence including animation intervals 439 and 441 and keyframe 440 shown in the animation graph in Fig. 4C is bidirectional. When a corresponding animation 427 or 428) is selected in the state- 717710 -23iF graph user interface 703, the bi-directional animation button 722 will be shown in the state, and any changes made to the animation sequence within the timeline window 702 will apply to both corresponding animations e.g. 427 and 428.

If an animation e.g. 708 is selected in the animation graph user interface 703, t 5 again for example using the mouse pointer 1603 and corresponding cursor, then the O animation timeline 712 will be updated to correspond to the selected animation, including the state icons 713 and 714, keyframe icons e.g. 715 and 716, animation intervals e.g. 718 and 720 and associated animation time labels e.g. 719 and animation time label 717.

Fig. 8 illustrates an alternate user interface 800 for authoring an animation graph for a widget including a state-frame graph in diagram form and an animation timeline.

The user interface 800 includes a state-frame graph window 801, an animation timeline window 702 and a visual appearance window 725. The contents of the three windows 801, 702 and 725 together comprise the user interface 800 and are used in conjunction with each other, although at certain times any of the windows may be rendered nonvisible whilst using the user interface 800.

The state-frame graph window 801 contains a state-frame graph user interface 803. The state-frame graph user interface 803 contains a state-graph diagram having state-frame icons 804, 805 and 806 representing the state-frames of the state-frame graph 421, 422 and 423) and arrows 807, 808, 809, 810 and 811 representing animations of the state-frame graph 424, 425, 427, 428 and 429).

The state-frames as represented by state-frame icons e.g. 804, 805 and 806 in the state-frame graph user interface 803 are selectable. When a state-frame is selected, the visual appearance of the widget 726 in the visual appearance window 725 is updated to 717710 -24- F-A show the appearance associated with the selected state-frame. Additionally the visual ;appearance of the selected state-frame can be edited using graphical editing tools (not shown).

The animations represented in the state-frame graph panel are selectable.

(N

00 5 Animation 807 is shown selected as indicated by the darker arrow. If an animation e.g.

O 808 is selected in the animation graph user interface 803, for example using the mouse (Ni 1603 and its corresponding cursor represented within the GUI 800 upon the display (Ni 1614, then the animation timeline will be updated to correspond to the selected animation.

Fig. 9A illustrates another alternate user interface 900 for authoring an animation graph for a widget including a combined state-frame graph and multiple animation timelines, arranged in two dimensions. The user interface 900 includes an animation graph window 901 and a visual appearance window 725. The contents of the two windows 901 and 725 together comprise the user interface 900 and are used in conjunction with each other, although at certain times either of the windows may be rendered non-visible whilst using the user interface 900.

The animation graph window 901 contains an animation graph user interface 903, animation function selector 721 and a bi-directional animation button 722. The animation graph user interface .903 includes state-frame icons 804, 805 and 806 representing state-frames of an animation graph 421, 422 and 423) and timelines 904, 905 and 906 where each timeline joins two state-frames to each other and is used to define the animation sequences in both directions between the state-frames, where applicable. The timelines 904, 905 and 906 may vary in length in proportion to the animation time of the animation represented.

717710 The state-frames are represented by state-frame icons 804, 805 and 806 in the animation graph user interface 903 and are selectable for example via a cursor associated with the mouse pointer 1603. When a state-frame is selected, the visual appearance of the widget 726 in the visual appearance window 725 is updated to show the appearance associated with the selected state-frame. Additionally the visual appearance of the selected state-frame can be edited using graphical editing tools (not shown).

The timeline 904 that is shown in horizontal orientation and in full detail is the currently active timeline in the example of Fig. 9A. Timeline operations such as adding, deleting and moving keyframes and altering animation times as described with reference to Fig. 7 are performed using the currently active timeline. The bi-directional animation button 722 applies to the animation corresponding to the currently active timeline.

Directional buttons 908 and 909 are shown on a timeline 904 in the case that there are two separate animation sequences between the two state-frames 804 and 805 associated with the timeline 904, being one animation sequence in each direction.

Directional buttons are not shown on timelines associated with bi-directional animation sequences e.g. timeline 905, nor are they shown where there is only one animation between the two state-frames associated with a timeline, e.g. timeline 906.

The directional buttons 908, 909 have two states and are used to indicate which animation sequence is currently being edited, and to allow the user to select between the two animation sequences to allow either to be edited using the timeline 904. For example, directional button 908 is shown in its 'active' state (ie. a hollow centre) indicating that the timeline 904 is currently being used to edit the animation sequence starting at state-frame 804 and ending at state-frame 805. Directional button 909 is shown in its 'inactive' state (ie. a filled centre) and, by operating the directional button 717710 -26- F,1 909, the timeline 904 will be updated to represent the animation sequence starting at ;state-frame 805 and ending at state-frame 804. When this occurs, the states of both directional buttons 908 and 909 are swapped, so that the directional button 908 will be in its 'inactive' state and directional button 909 will be in its 'active' state.

Non-active timelines such as the timelines 905 and 906 can be made active by O selection. When a new timeline is selected, the animation graph user interface 903 is Srotated so that the newly selected (active) timeline becomes orientated such that it is convenient for the user to perform timeline operations. Preferably, and as illustrated, the active timeline is oriented or aligned horizontally. Also when a new timeline is selected the animation graph user interface 903 is updated to show the full details (including animation times and directional buttons) of the newly selected timeline. All subsequent timeline operations are performed using the newly selected timeline until such time as yet another timeline is selected as being the active timeline. Fig. 9B illustrates the rotation of the animation graph user interface 903 of Fig. 9A in response to the selection of a new timeline 905.

There is a limitation in the representation of an animation graph user interface as described with reference to Fig. 9A and 9B in that, for animation graphs with more than four state-frames, it becomes necessary for the timelines to cross over each other.

For example, Fig 10 shows a node graph in two dimensions having four nodes fully connected to each other by non-crossing bars. A node graph of this type is applicable for rendering a user interface comprised of state-frame icons and timelines such as described with reference to Fig. 9A and 9B. If an additional node was added to the diagram, it would not be possible to connect the new node to all of the existing nodes with bars without crossing over at least one of the bars currently shown.

717710 27 Having timelines cross over each other in a user interface may be visually ;confusing and could therefore impair the ability for the timelines to be operated. One possible solution is to render non-active timelines in a partially or fully transparent manner to avoid crossing. However, a disadvantage of this approach is that the whole (,i 00 5 structure of the animation graph cannot be visualised at once whilst one or more tt') O timelines are rendered transparent.

Another solution to this problem is to introduce a third spatial dimension in which to render the node graphs. In three-dimensional space it is possible to have many more nodes connected by bars without the bars intersecting each other. For example Figs. 1lA to 1lE show orthogonal and perspective projections of a three-dimensional node graph structure having six nodes fully connected by bars, with none of the bars intersecting each other in three-dimensional space. The intent here is that the bars do not intersect each other in 3D space, so that by manipulating the viewpoint it is possible to view any bar without interruption by other bars. The bars may be still seen to 'crossover' from a 2D viewpoint, but will be at a different z-depth. Orthogonal and perspective views are provided to fully specify the 3D structure in an unambiguous manner. This is illustrated in Fig. 11E. Figs. 12A 12D shows similar views of a threedimensional node graph structure having seven nodes fully connected by bars. with none of the bars intersecting each other in three-dimensional space.

When rendering a three-dimensional node graph for display on a normal twodimensional display screen the same limitations apply with respect to bars crossing each other as apply when rendering a two-dimensional graph. For example, in the 'top view' of Fig. 11 A, the bar connecting node 2 to node 4 appears to cross two other bars in its path. However using three-dimensional space allows the node graph structure to be 717710 -28- N rotated freely in order for a user to examine the entire structure from a range of different ;angles. This allows any pair of nodes connected by a bar to be presented front-on to the user such that the bar appears horizontal and unobscured, providing the bar passes through clear space within the structure. This is seen for example in Fig. 11E with respect to nodes 2 and 3.

O Fig. 13 illustrates another alternate user interface 1300 for authoring an (i/ Sanimation graph for a widget 726 including a combined state-frame graph and multiple animation timelines in three dimensions. Although the animation graph 1303 represented in Fig. 13 has only three state-frames 804, 805 and 806 and therefore could be represented using a two-dimensional user interface such as that described with reference to Fig. 9A and 9B, Fig. 13 is an example of how a user interface for authoring animation graphs can be based on a three-dimensional node graph structure including those shown in Figs. 11 A- 11E and Figs. 12A-12D.

User interface 1300 includes an animation graph window 1301 and a visual appearance window 725. The contents of the two windows 1301 and 725 together comprise the user interface and are used in conjunction with each other, although at certain times either of the windows 1301 and 725 may be rendered non-visible whilst using the user interface.

The animation graph window 1301 contains a three-dimensional viewport 1302 in which is rendered a three-dimensional animation graph user interface 1303, animation function selector 721 and bi-directional animation button 722. The three-dimensional animation graph user interface 1303 has state-frame icons 804, 805 and 806 representing state-frames of an animation graph 421, 422 and 423) and timelines 904, 905 and 717710 -29- 906 where each timeline joins two state-frames to each other and is used to define the ;animation sequences in both directions between the state-frames, where applicable.

The state-frames as represented by the state-frame icons 804, 805 and 806 in the three-dimensional animation graph user interface 1303 are selectable. When a state- 00 5 frame is selected, the visual appearance of the widget 726 in the visual appearance Swindow 725 is updated to show the appearance associated with the selected state-frame.

Additionally the visual appearance of the selected state-frame can be edited using graphical editing tools (not shown).

The timeline 904 that is shown orientated horizontally, towards the front of the 3D viewport and parallel to the view of the user in full detail is the currently active timeline. Timeline operations such as adding, deleting and moving keyframes and altering animation times as described with reference to Fig. 7 are performed using the currently active timeline. Bi-directional animation button 722 applies to the animation corresponding to the currently active timeline.

Directional buttons 908 and 909 are used to indicate which animation sequence is currently being edited, and to allow the user to select between the two animation sequences to allow either to be edited using the timeline as described with reference to Fig. 9A.

The non-active timelines 905 and 906 can be made active by selection. When a new timeline is selected the animation graph user interface 903 is rotated so that the newly selected timeline becomes orientated such that it is convenient for the user to perform timeline operations. Again preferably the active timeline is oriented orientated horizontally, towards the front of the 3D viewpoint and parallel to the view of the user.

Also when a new timeline is selected the animation graph user interface is updated to 717710 N show the full details (including animation times and directional buttons) of the newly ;selected timeline. All subsequent timeline operations are performed using the newly Sselected timeline until such time as yet another timeline is selected as being the active timeline.

00

O

In an alternate implementation, the animation intervals as represented on the Sanimation timeline are visually representative of the animation functions applied to the animation intervals.

Fig. 15 shows a user interface 1500 that includes an animation timeline window 702. The animation timeline window 702 includes an animation timeline 712 and an animation function selector 721. The animation function selector 721 applies to the currently selected animation interval, being animation interval 1501 in this example.

The animation function selector 721 allows the choice of a range of options corresponding to pre-determined animation functions for the selected animation interval.

Examples of such options include "linear", "step-wise", "ease in" and "ease out". Each animation interval in an animation sequence may have its own animation function.

Fig. 15 shows the animation function in the timeline, eg. ease in, ease out, linear. This may apply to any of the 2D or 3D timelines described. If used in the example of Fig. 13, such may be within the timeline, at 718.

Animation intervals 1501, 1502, 1503 and 1504 each represent a portion of the animation between two adjacent keyframes. Within each animation interval is shown an animation function curve. In this example, animation interval 1501 shows an animation function curve representative of an "ease in" animation function (ie. the curve is exponentially shaped); animation interval 1502 shows an animation function curve representative of a "linear" animation function (ie. the curve is linear); animation interval 717710 -31 N 1503 shows an animation function curve representative of an "ease out" animation ;function (the curve is logarithmically shaped) and animation interval 1504 shows an animation function curve representative of an "step-wise" animation function (ie. the curve includes a step). Other forms of visual indicators may be used.

(N

00 5 Fig. 14 shows a consumer device 1401 such as a digital camera which includes a O display screen 1402, for example a colour Liquid Crystal Display or similar. A user interface 1403 is rendered on the screen 1402 and provides visual feedback to the user of the consumer device 1401. The user interface 1403 includes animated widgets 1404, 1405. Control buttons 1406 are used to control the functions of the consumer device 1401 and, in response to user actuation of the control buttons 1406, the state of one or both of the animated widgets 1404, 1405 may change. The appearance, animations, states and behaviours of the animated widgets 1404, 1405 comprising the user interface 1403 are desirably authored using the arrangements described herein wherein a controlling program of the user interface 1403 in concert with the buttons 1406 is stored with a computerised memory and/or controlling processor (not illustrated) of the device 1401.

It follows therefore that the arrangements described herein provide for the convenient development of animated graphical user interfaces that, once developed may be operated in devices that have a need for such interfaces.

Industrial Applicability 717710 -32c1 The arrangements described are applicable to the computer and data processing Sindustries and particularly those for the development of user interfaces that require some amount of animation.

The foregoing describes only some embodiments of the present invention, and 00 I¢ 5 modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

(Australia Only) In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings.

717710

Claims (19)

1. A method for authoring an animated transition between states, said method comprising the steps of: 0 5 displaying a representation of at least one transition between at least two states wherein said at least one transition connects two of the states; Sselecting one said transition from said representation; displaying an animation timeline associated with the selected transition; and editing the animation timeline for the selected transition to thereby author the animated transition between said states.

2. A method according to claim 1, wherein the at least two states.. are displayed concurrently with the representation of said at least one transition.

3. A method for authoring an animated transition between states, said method comprising the steps of: displaying a diagram having at least two states and at least one animation timeline where each animation timeline connects two of said states and relates to at least one animated transition connecting the two said states; selecting an animation transition from said diagram; and editing an animation corresponding to said selected transition using an animation timeline associated with the transition to thereby author the transition. 717710 -34- F-A 4. A method according to claim 3, further comprising, intermediate steps (b) tb3 ;and the further step of: (ba) rotating said diagram so that said associated animation timeline is at least substantially horizontally aligned. 00 ,In A method according to claim 4, where said diagram is represented in three dimensions.

6. A method according to any one of the preceding claims wherein said states comprise states of operation of a graphical user interface corresponding to a display of a particular graphical form as part of said interface, and said transitions comprise alteration of said interface from one particular graphical form to another.

7. A method according to any one of the preceding claims wherein said each said state has associated therewith a corresponding graphical state frame and said animation timeline defines a temporal display of an animated transition from one said graphical state frame of one said state to another said graphical state frame of the other said state of said selected transition.

8. A method according to claim 7 wherein said timeline further comprises at least one key frame associated therewith and forming part of the transition between the graphical state frames. 717710 FA 9. A method according to claim 8 wherein said editing comprises altering a time ;period between adjacent ones of said frames in said timeline of said selected transition. A method according to claim 7, 8 or 9 wherein said editing comprises altering at 00 5 least one said frame associated with said selected transition. C,

11. A method of authoring a graphical user interface for use in operating a device, the graphical user interface being displayable in operation upon a display associated with the device and including a plurality of widgets an interaction with which represents a state of operation of the graphical user interface for control of the device, said method comprising the steps of: displaying a state transition diagram including a graphical representation of each said state and a graphical representation of each transition interconnecting said states and defining operation of the graphical user interface being authored, each said transition being formed between a corresponding two of the states; detecting a selection of one said transition; displaying, in response to the selection, an animation timeline associated with the selected transition; editing an animation associated with the displayed animation timeline to thereby author the graphical user interface.

12. A method according to claim 11 wherein said animation comprises a temporal alteration of the graphical user interface as represented upon the display of the device between a first state associated with one said widget of the selected transition to a second 717710 -36- c state of the selected transition and said editing comprises altering the displayed ;animation timeline to thereby change the temporal alteration of the graphical user interface. 00 5 13. A method according to claim 12 wherein said transition is unidirectional and is O operable to cause animation of the graphical user interface from the first state to the second state.

14. A method according to claim 12 wherein said transition is bidirectional and is selectably operable according to the state of operation to cause animation of the graphical user interface from the first state to the second state, and is selectably operable according to another state of operation to cause animation of the graphical user interface from the second state to the first state.

15. A method according to claim 12 wherein said animation timeline comprises a graphical representation of each said state of the selected transition connected by a timeline defining the temporal transition.

16. A method according to claim 15 wherein said editing comprises altering the timeline.

17. A method according to claim 15 wherein said timeline further comprises at least one representation corresponding to a key frame displayable as part of the temporal alteration of the graphical user interface. 717710 -37-

18. A method according to claim 17 wherein said representations corresponding to one or more said keyframes divide said timeline into a plurality of time segments, said editing comprising editing at least one of said time segments. (N 00 In O 19. A method according to claim 15 wherein said editing comprises selecting one Ssaid state graphical representation to display a state frame corresponding to that said state and altering the displayed state frame.

20. A method according to claim 17 wherein said editing comprises selecting one said representation to display the corresponding key frame and altering the displayed key frame.

21. A method according to claim 12 wherein said editing comprising assigning a pre- determined animation function to at least one animation interval of the animation timeline.

22. A method of authoring at least an animation timeline substantially as described herein with reference to any one of the embodiments as that embodiment is described with reference to Figs. 4A to 16 of the drawings.

23. A computer readable medium having a computer program recorded thereon and executable to make computer device author an animation timeline in accordance with the method as claimed in any one of the preceding claims. 717710 -38-

24. A computer program executable to form a graphical user interface upon a device, the program being formed using the method as claimed in any one of claims 1 to 22.

25. A device having associated therewith a graphical user interface authored according to the method of any one of claims 1 to 22. Dated this ELEVENTH Day of AUGUST 2005 CANON KABUSHIKI KAISHA Patent Attorneys of r the Applicant Spruson&Ferguson 717710