WO2001031584A1 - System and method for automated visual configuration - Google Patents

Abstract

The present invention relates to a method and apparatus for automated visual configuration (110) of objects (140) on screen without the need for user coding. Specifically, the system generates visual representation of objects (Fig. 12) by a drag and drop process and is built on an extensible open architecture to permit varying technology to work interchangeably.

Description

SYSTEM AND METHOD FOR AUTOMATED VISUAL CONFIGURATION

BACKGROUND OF THE INVENTION

The present invention pertains to visual configuration for use in dynamic visual representation of objects, surfaces and environments when assessing variations and changes thereon.

Prior art systems for visual configuration provide users with ways of manipulating visual objects on a screen for on-line purchases or for simple viewing and/or presentation of visually oriented scenes. Computer aided sales presentations wherein rules-based configurators are employed have long been a valuable tool for allowing a customer or client to configure his or her own order of a product or products. This technique is employed in a network environment whereby a client configures his or her order and sends it over the Internet, or another type of network, to a seller, which can fill the order. For example, a complex computer system has many different features, accessories, options, etc. With the appropriate software program, a client can configure his or her own custom order for a particular brand or type of system. By employing an interactive method a client can apply certain desired features in his product-order configuration that were chosen from options provided by a seller.

More recently, object-oriented, constraint-based configurators have begun to replace rules-based configurators by virtue of their more flexible organization. For example, rules-based configurators rely on set rules that are programmed in a procedural computer language. Partly for this reason, every change in the merchandise offered may require substantial changes in the software, which can be very strenuous, considering the fast pace of trade today. However, what is needed is a separation between the product configuration knowledge and the process that manipulates the knowledge base. With such separation, the knowledge base can be changed easily via an editing process. The knowledge base applies the changes via linking to other elements of the process such as the object model.

The electronic retail industry is rapidly evolving. The demand from customers to have a true, real-life shopping experience each time they go on-line instills constant pressure on all the Business-to-Customer (B2C) and Business-to-Business (B2B) companies. These new and continuous demands for more visually rich and highly interactive experiences of product configuration, requires enterprises to redefine their on-line business over and over again.

Today, it is just not enough to sell products the traditional way over the Web. E-commerce businesses are scrambling to add more personalization and make their website more dynamic with visually rich and intelligent configurations that are easy to use for even the first time customer. Driving more user traffic to e-commerce web sites is not enough in today^'s marketplace, the key issues are customer stickiness and revisits. Furthermore, in order to have an effective e-commerce solution, all such applications should interact with the enterprise backend systems.

Many enterprises are striving for a visual configuration that is simple to implement, yet sophisticated enough for complex products. However, time-to-market and cost-to-market prohibit such a complex venture until now. With the present invention these and other applications are possible at a fraction of what it would cost to build the same thing from the ground up. The present invention gives both

Business-to-Business (B2B) and Business-to-Consumer (B2C) enterprises the power that is needed to remain competitive in the e-commerce market.

Market research shows that there is a growing demand for more sophisticated on-line interactivity and configuration to help the conversion rate from browser to purchaser in today's self service e-commerce applications. The lack of sufficient sales capabilities in today's web commerce applications is evident by a 1999 Forrester Research survey of on-line merchants that indicates 2.7 out of every 100 on-line browsers make the leap from browser to buyer. The low conversion rate indicates that today's e-commerce applications are not performing well in the sales process. However, current solutions offer limited versatility for users in several regards, not the least of which pertain to the fact that users cannot take full advantage of all manner of visual (and attendant audio) media manipulation. Moreover, even where certain provisions are made to users, the users of prior art systems must necessarily utilize coding or other expertise in order to effectuate project goals.

Accordingly, the present invention provides an enterprise with an effective e- commerce solution that is fast time-to-market and amazingly low cost-to-market. The visual configuration engine of the present invention enables the next generation of online selling strategies enterprise to give their customers a more dynamic, visually rich, interactive and personalized shopping experience while integrating with existing back office business systems. The visual configuration engine integrates a wide range of enterprise e-commerce solutions with enterprise databases to deliver the solutions needed for a variety of innovative e-commerce solutions. Further, the visual configurator of the present invention, with its ability to visually configure complex objects, provides e-commerce retailers with the ability to attract and retain customers for reasonable periods of time, converting browsers to buyers.

OBJECT AND SUMMARY OF THE INVENTION

Therefore, it is an object of the present invention to provide a system and method which overcomes the short comings of the prior art system.

Another object of the present invention is to provide a system and method which generates visual representation real life objects by drag and drop process without the need for coding or specialized expertise by the end user.

A further object of the present invention is to provide a system and method as aforesaid, which provides a collaborative environment for enabling a team of people to work together, regardless of geographical or temporal constraints, while at the same time continuing to accommodate the access needs of the individual team members. Such environment is provided while establishing the relative positioning of the visual objects during distributed user manipulation of the visual objects. Yet a further object of the present invention is to provide a system and method as aforesaid, which utilizes its extensible architecture to permit resource types from varying technologies to work together interchangeably and also enable the user to add new formats through plug-ins. The present invention is designed to meet a wide range of interaction and configuration needs from pack-and-go applications, which can be bundled with e- mails to allow the customer to configure and buy offline, to complex visual configurations. Traditional visually interactive and configuration tools have been text-based with a design that needs constant maintenance by an experienced web developer. Implementation into existing enterprise solutions is rather difficult and maintenance even harder. With the present invention, the user can obtain visually rich, highly interactive, and complex configurations. Additionally, the present invention enables the users to quickly incorporate each model with minimal work from design to deployment. This makes time-to-market timelines a reality. Further, the present invention allows the user to sell in new and innovative ways, improve customer interaction and satisfaction, and enhance customer relationships. All these steps can lead to a higher bottom line.

The system and method of the present invention supports and integrates with a wide range of software platforms. This open architecture enables the visual configuration generated with the present invention to integrate easily with industry- standard enterprise databases, providing platform, infrastructure, and browser independence. Deployment of generated applications utilizing the present invention is quick and painless, giving the user (e-commerce consumer) more time to design applications and concentrate on the more important aspects of an on-line enterprise. Accordingly, the present invention provides visually rich and interactive applications that enable e-commerce consumers to easily configure products on-line. This can be achieved at a fast time-to-market and a low cost-to-market.

The system and method of the present invention is not only simple for the consumer, but for the enterprise as well. The design and development cycle is not in the hands of the application developers but in the hands of content creators or marketing analyst. The advantage reduces the time required to generate these media rich applications. That's because even the most intensive application does not require even a single line of coding. Sophisticated on-line applications generated using the system and method of the present invention help to increase the conversion rate from browser to purchaser in today's self-service e-commerce applications.

The present invention supports and provides the following: support for most popular media formats like JPG. midi, wav, avi, Flash, etc.; robust state-based configuration engine; Java and DHTML (dynamic hyper text overlap language) generation; support for Internet Explorer and Netscape 4.0 and up; output is platform independent; fast runtime environment; media streaming and caching technology; collaboration tools for development team; most known databases supported; straightforward XML-driven (extensible markup language-driven) integration with corporate ERP (enterprise resource planning), and with industry-standard e-commerce packages; localized for foreign markets; support of network configurations in a full object oriented Visio-like development environment; support for additional media formats, including 3-D media elements, and CAD (computer-aided design); support for additional output formats, including Flash generation; support for additional databases; multiple database support; enhanced scalability of generated applications; support for various user interactions including drag-and-drop; enhanced and more intuitive configuration builder; and extended XML support.

Various other objects, advantages, and features of this invention will become readily apparent from the ensuing detailed description and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS The following detailed description, given by way of example, and not intended to limit the present invention solely thereto, will best be understood in conjunction with the accompanying drawings in which:

Figure 1 is a block diagram of the functional components of the automated visual configuration system in accordance with an embodiment of the present invention; Fig. 2 is a schematic representation of a top level architecture of the automated visual configuration system in accordance with an embodiment of the present invention;

Fig. 3A is a block diagram of the components of the automated visual configuration system in accordance with an embodiment of the present invention;

Fig. 3B is a block diagram of the Runtime manager and the servers contained within it.

Fig. 4 is a block diagram of an exemplary information directory of the present invention; Fig. 5 is graphical representation of an exemplary connection map implemented using local scope in accordance with an embodiment of the present invention;

Fig. 6 is a visual representation of an exemplary connection map implemented using reference and origin points in accordance with an embodiment of the present invention;

Fig. 7 is a visual representation of the connection map of Fig. 6 after a visual change to an object;

Fig. 8 is a visual representation of an exemplary connection map including contained items in accordance with an embodiment of the present invention; Fig. 9 is a visual representation of an exemplary connection map implemented using links to connect nodes to a root in accordance with an embodiment of the present invention;

Fig. 10 is a visual representation of the connection map of Fig. 10 implemented using links connect nodes in accordance with an embodiment of the present invention;

Fig. 11 is a snapshot representation of exemplary containments and connection map of the present invention;

Fig. 12 is a visual representation of a delta adjustment when an item is moved in accordance with an embodiment of the present invention; and Fig. 13 is a visual representation of a link adjustment when a positioner is moved in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION In a fast evolving e-commerce marketplace, e-tailing web sites must adopt technologies that will allow them to turn browsers or visitors into shoppers, retain customers, and personalize the shopping experience, all in an effort to add to the bottom line. The visualization and configuration tools of the present invention allow the enterprise to generate and add visually rich, interactive, and configurable elements to their web site at both a fast speed-to-market and a low cost-to-market.

Working together with an enterprise e-commerce solution, the present invention enables the enterprises to empower their online customers with more lifelike shopping experiences, bring customers back to their web sites again and again through a truly dynamic personalized shopping experience (i.e., turn customers into repeat customers), help turn browsers into shoppers through try-before-you-buy, mix- and-match, and even more complex configurations.

In accordance with an embodiment of the present invention, the automated visual configuration system 100 includes a configurator 110, an event manager 120, a media asset manager or media manager 130, an object storage layer 140 and a database 150. The event (or event-interaction) manager 120 drives the applications generated by the present invention through user interactions. Preferably, the event manager 120 is designed for speed, robustness, and extensibility. The event manager 120 allows objects to respond to and generate events, trigger actions, and pass messages between one another or external objects performing "actions" based on the event-map.

In accordance with an embodiment of the present invention, the event manager 120 includes a 3-D positioning system (not shown), which supports both universal positioning and relative positioning. For example, if the universal positioning is set for objects, users 160 can freely crop, redraw, add, delete or move elements of the objects, and the event manager 120 will automatically reposition, compose and resize the elements accordingly That is, the event manager 120 automatically adapts the elements of the objects to a new set of environments based on the user's changes to the environment

The configurator 1 10 is a state-based configurator, which is preferable for event-driven applications The state-based configurator 1 10 advantageously permits the objects to be configured in a true object-oπented environment. Trying to use rule- based or constraint-based reasoning in these environments is like trying to fit a square peg in a round hole The state-based configurator 1 10 of the present invention allows complex configuration logic to be built into the objects as well as relationships between the objects

The configurator 1 10 is also a true visual configurator, where the visualization elements and configuration elements aie tied together into one bundle This provides a range of very powerful manifestations of various applications like designing complex machinery through assembly of relevant parts, designing a network visually, or buying a car on the web by trying all the options

The media manager 130 manages the media assets being used within the system for other components, such as the event manager 120 or the configurator 110, so as to provide immediate response through caching services, high availability, and robustness The media manager 130 is connected to the multimedia object storage system or layer 140 that can be leveraged through commercial enterprise-grade database management systems, thereby providing scalability for a wide range of development and runtime environments

The object-oriented storage layer 140 allows applications generated using the present invention to be updated on the fly in production environment stored m the database 150, thereby providing a powerful tool to users in designing and deploying dynamic enterprise-grade visually interactive and configurable solutions

The External Interface for Functionality

Turning now to Figs 2 and 3, there is illustrated an embodiment of the automated visual configuration system 100 of the present invention. The visual configuration system 100 includes an integrated development environment 200 comprising a run time client or browser module 210 for accessing Rimps. Rimp(s), as used herein, refers to resource(s) or Imp(s). An Imp is a smart object that brings "life" to a user^'s presentation by providing means to structure and group a collection of resource objects into a self-contained usable object, and as such, are basically smart containers for resource objects and other Imps. Resource objects may involve virtually any visual (or even attendant audio) files known in the art. The user 160

(Fig. 1 ) can access the database 140 via the browser module 210 to copy one or more

Rimps into his project and edit their characteristics. A project contains all Rimps necessary for a user presentation. The utilities 220 include the visual configurator 110 and the media manager 130. The visual configurator 1 10 comprises a visual graphic interface for enabling the user 160 to position the Rimps, save the Rimps as Imps and edit the Rimps' characteristics. Preferably, the visual configurator 1 10 includes a storyboard module (not shown) for enabling the user 160 to generate a sequence of scenes or Rimps by defining the relationship between the Rimps. Preferably, the utilities 220 includes the media manager 130 to provide a system administrator, or the user 160, with sufficient authority to establish other users and establish user groups, generate projects and define access rights to banks and projects. A bank contains libraries of Rimps that can be utilized in the projects. The event manager or run time module 120 enables the user 160 to preview and run the entire projects or individual

Imps.

The browser module 210 enables a user to access and view elements such as the Rimps, the bank, projects, a library, an Imp, a new resource, a WAVE file, a BMP file, a text file, an AVI file, a MIDI file, an MPEG file, and the like. A properly authorized user 160 can copy Rimps from the bank to projects as required and vice versa. The user 160 can access the visual configurator 1 10 via the browser module 210 to generate an Imp, edit Imp's characteristics and define the relative positions of the Rimps. Through the browser module 210, the user 160 can select a private or particular view of the Rimp generated by the media manager 130 while editing or configuring the Rimp. When the user 160 activates a certain private view, everything the user 160 generates or changes can be seen only in that pπvate or particular view until the user 160 "checks in" the Rimp The Rimp can be seen in all views only after it has been "checked in," thereby ensuπng the integrity of the Rimp Accordingly, in order to edit or change the Rimp. the user 160 must "check out" and activate the desired Rimp m the previously selected pπvate view When the user 160 changes an existing Rimp, the media manger 130 does not update the master version of the Rimp until the user 160 "checks in" the Rimp

The user 160 can use the visual configurator 1 10 to generate an Imp, edit the Imp's characteπstics and define the relative positions of the Rimps The visual configurator 160 saves each Rimp used in building an Imp as a containment The containment can be any resource, such as. AVI, BITMAP, MIDI, MPEG, TEXT, WAVE or another Imp It is appreciated that if the containment is another Imp, then the containment itself has a containment In accordance with an embodiment of the present invention, the user 160 can use the visual configurator 110 to edit the Imp's characteristics or the containment, such as, name, meta information, constraints, containment type, descriptive information within the containment, etc. The meta information enables the user 160 to modify the x, y and z coordinates of a containment For example, a user can define an alpha channel for a bitmap, or modify the repeat count for a MIDI file, etc The constraints are used to associate a particular containment with a certain state of the Rimp The user 160 can change the containment type by selecting a new Rimp for the containment from the bank or project The Imp's containment response to a specific trigger event (or hereinafter referred to as the Imp's behavior) depends on the particular state of the Imp For example, when an Imp is clicked, a sound file may start playing or running and when the Imp is clicked again, the sound file may stop playing

Preferably, the user 160 can use the storyboard module of the visual configurator 1 10 to build a Supeπmp A Supeπmp composes a plurality of Imps to form a sequence of scenes The user 160 can change the location of any scene in relation to another, thereby alteπng the sequence of the scenes The user 160 can also edit the Imp^'s characteristics and define the relative positions of the Rimps. It is appreciated that the Imp may contain other Imps that are the players that abide within the scene

In accordance with an embodiment of the present invention, the visual configurator 1 10 includes a relative positioning feature for enabling the user 160 to establish a connection between the Rimps Accordingly, if the user 160 moves a Rimp, the visual configurator 1 10 moves the connected Rimp proportionally. Hereinafter, the main Rimp is referred to as the "referee" and the connected Rimp as the "referent." The referee always moves the referent, but the referent can move independently of the referee Nevertheless, the connection between the referee and referent is always maintained

When the user 160 generates a Supeπmp comprising a plurality of Imps, the storyboard automatically assigns or provides each Imp with the property location, I e., the location values, of each scene, different states of each location and other appropriate constraints It is appreciated that each scene has an associated location value. For example, if the user 160 drags a Supeπmp from the browser module 210 to the visual configurator 1 10, particularly the storyboard module, the Imps associated with the Supeπmp are displayed according to their location values.

The media manager 130 enables the system administrator to establish new users, establish user groups and assign users to these user groups, generate projects, generate private views, and authorize to the users and user groups access rights to the banks, projects and private views The media manager 130 may grant to certain users and/or user groups limited or full access to various views, projects and the like.

After generating (or completing the configuration) the project or an Imp using the visual configurator 1 10, the user 160 can am the project or Imp using the event manager or run time manger 120 The configuration of the project may involve the user 160 editing the Imps' characteristics, such as their properties and values, states, constraints, behavior, containments, relative positioning via connection map, etc

As noted herein, a state is a unique set of values representing the properties of an Imp. Each Imp has a default state deπved from the default values assigned to each of the properties. Although, each Imp can have any number of states, it can only exist in one state at any given time Preferably, the states define the behavior for an Imp.

The Visual Configurator As noted herein, the Imps are basically smart containers for resource objects and other Imps In accordance with an embodiment of the present invention, the containment relation is not static over the Imp's lifetime, but can change dynamically based on the "current state" of the Imp, thereby advantageously providing a "state- based containment" or "smart" containment relation. The state-based containment of the Imps enables the visual configurator 1 10 of the present invention (state-based configurator) to alter the Imps over time. That is, duπng the lifetime of an Imp, the resource object of a given imp undergoes changes, e g., a person's face might change from happy to sad, a TV may change channels, or the like.

Additionally, the state-based containment of the Imps enables the visual configurator 1 10 of the present invention to reuse the Imps with a slight modification.

For example, the user 160 can have a dog Imp that is white and can re-use the dog Imp in another location as black rather than its oπginal white. Conceptually, this is similar to having construction parameters for creating vaπations of a class in C++. It is appreciated that this type of variation can take place along a development cycle within a single project, as new capabilities are added to an Imp.

Further, the state-based containment of the Imps enables the visual configurator 1 10 of the present invention to share interfaces between Imps, i.e., polymorphism Since, all Imps support the same interface for changing their state, the visual configurator 1 10 of the present invention can communicate with different Imps using (and reusing) the same calls For example, each Imp having white value for a color property will respond to a request for changing its color to white. Conceptually, this is similar to virtual function calls in C++ Accordingly, the visual configurator 110 can use the Imp's state changes to update an attribute of the Imp, as well as to tπgger events or behaviors within the Imp. To descπbe the state-based containment aspect of the present invention, the following conventions will be utilized to describe the characteπstics of the Imps, such as, properties, containment, constraints, overloading, and matching. Imp names are capitalized and prefixed with a small "1". e g , iTelevisionSet, resource names are prefixed with a small "1". e g , rMusicNoise, containment items are in lowercase and may be followed by their index and type, e g , noise, noiseO, noiseO [rMusicNoise]; unnamed containments can be shown with just their type, e g., [rMusicNoise], and referenced containments can be shown using " " notation, e g , iTelevisionSet noise or iTelevisionSet [rMusicNoise] A property is a variable having one of several possible values Both properties and their corresponding values are text strings that are compared using lexical matching techniques Properties are defined locally withm the scope of an Imp The set of properties defined within a given Imp is known as its Property Set For example, an Imp called "Television Set" can have the following properties Screen Size = {tiny, ordinary, big-screen}, Channel = (snow, MTV, sports. CNN}, Power =

{on, off} Accordingly, the Property Set for the "Television Set" Imp can be {Screen Size, Channel, Power} For each property, one of its corresponding values is preferably designated as the "default value" That is, the default value is the initial value that is assigned to the property if no other value is specified by the user 160 For example, the default values for the "Television Set" may be Screen Size = ordinary, Channel - snow , Power = off Properties can be defined duπng development time and stored m the database along with other information about the Imp

A state of an Imp is a particular set of values for all of the properties in its Property Set For example, a possible state of a "Television Set" Imp can be {Screen

Size = big-screen, Channel = mtv, Power=on} The current state refers the state of a running or active Imp. which essentially means that at any point m time there is always a current value for each property in the Property Set The "default state" of an Imp is the state derived from assigning default value to each property When an Imp is activated or running, this is the state the Imp will begin in, unless otherwise specified A "Partial State" for an Imp is a set of values for some subset of its Property Set. For example. { Channel = mtv, Power=on} is only a partial state for the Television Set. because the information regarding the screen size of the television is missing. An item contained by an Imp is referred to as a containment or contained item.

Conceptually, the containments are similar to class member data in C++, m that the containments represent an instance of a class whose name is local to the scope of the containing object Like member data, a containment has a name containing a simple text stπng and a type. The name can also be omitted from the containment to generate an unnamed containment, which cannot be directly referenced. The type can be any object (resource or Imp) in the database 150 For example, the Television Set Imp can contain an item called "Channel Chooser" of the type "Dial", wherein "Dial" is the name of an Imp which displays a generic dial The "Television Set" Imp may also contain an unnamed containment type "Snow Pattern", wherein "Snow Pattern" is a resource which displays a static filled background. The visual configurator 110 of the present can represent the named containment as iTelevisionSet.channel chooser [lDial] and unnamed containment as ιTelevιsιonSet.[rSnowPattern]. It is appreciated that there a numerous alternative representations of the containment using known object-oriented languages. Additionally, the containment affects the life span, visibility and access to the contained item. The lifetime (or life span) of the included item is bound by the lifetime of the containing item That is, the contained item is generated after the container and is destroyed no later then the containing Imp. Also, the Imp controls the visibility of its containments Items not visible to the visual configurator 110 are considered functionally non-existent and not to be contained withm the object. Items that are visible to the visual configurator 1 10 can express themselves in the audio, visual and behavioral domains Further, the contained item is affected in that it can be accessed only through its containing Imp

The visual configurator 1 10 utilizes the constraints to change the set of Imps and resources which are contained (or visible) in an Imp over time based on the current state of the Imp. A containment can have an empty set of property values resulting in a partial state for an Imp. The constraint serves as a selection mechanism for containment. An item is considered to be visible to the Imp when the constraint of such item matches the current state of the containing Imp. For example, an iTelevision Imp can have a containment item named "music noise" representing an audio resource which plays loud music. This item can be set to have the following constraint: {Power = on, Channel = MTV} . The significance of this is that the music noise will only be visible or audible (since this is a sound item) to the visual configurator 1 10 when the current state of the Television Set is powered on, with the channel set to MTV. However, since the constraint makes no reference to the Screen

Size property, the item will be visible regardless of the current Screen Size setting. The visual configurator 1 10 ignores all properties not explicitly included in the constraint.

The containment is considered to be "overloaded when two or more containment items are given the same name. For example, if iTelevision Imp contains two items named "noise", e.g., noise[rMusicNoise] when the channel is selected to MTV and noise[rSportAnnouncer] when channel is set to sports, the visual configurator 110 considers the item to be overloaded and automatically appends an index number to each item with the same name, such as, noiseO, noise 1, noise2, etc. Although the items are displayed to the user 160 as having the same name, the visual configurator 1 10 internally utilizes an index number to distinguish between the identically named items.

The containment overloading provides the visual configurator 1 10 with the following capabilities: polymorphic reference, shared positioning and automatic matching. The polymorphic reference permits the Imp, or even clients of the Imp, to refer to several items using a single name. For example, as noted herein, the user 160 can referred to "noise" without knowing what resource it really represents. The overloaded containments share runtime information, particularly their on-screen positions which can be beneficial in relative positioning of the objects. Also, when items are overloaded, the system 100 ensures that most of the items will be visible simultaneously, thereby providing automatic matching Further, the overloaded containment can be used to provide default behavior while the item is being developed

In accordance w ith an embodiment of the present invention, overloading is generallv used together with appropπate constraints so that a single item is chosen for each state of the Imp Foi example, if iTelevisionSet contains some items named

"picture" which display an image inside the screen of the television set, the items might be then setup as follows pιctureO[rMusιcVιdeo] constraint = {Channel = mtv, Power = on}, picture 1 [rTenmsMatch] constraint = {Channel=sports, Power = on}, pιcture2[ιSnowPattern] constraint ={Channel=snow, Power = on}, and pιcture3[rBlankScreen] constraint = {Power = off} , or as equivalentiv provided for under any of the available object-oπented programming languages Accordingly, this setup ensures that the picture will change as the channels are adjusted, and that the picture will go blank when the set is turned off

Nevertheless, when two or more items share the same name and their constraints both match the current state, the system guarantees that, at most, one item will be shown This is known as automatic matching, because the system chooses the item which is the "best match" for the current state The better match is preferably the item with the larger constraint set, 1 e , more properties In the case of two items with same number of constraints, the system 100 may choose arbitraπly but consistently

Automatic matching can be used to set up default behaviors Continuing with the above example, there is no "picture" item when the television is tuned to the "CNN" channel because the graphics for the CNN channel is not yet ready If an Imp is then set to such state, there would be an empty space for the picture item However, by providing for automatic matching, the present invention can rectify such a problem by allowing for a default look for the television set with the power on, wherein a rSnowPattern resource appears as the picture This can be accomplished by setting the constraints as follows PictureO[rMusic Video] constraint = {Channel = mtv, Power = on}, picture l[rTennisMatch]constraint = {Channel=sports, Power = on}, picture2[rSnowPattern] constraint = {Power = on} (default for power on), and picture3[rBlankScreen] constraint = { Power = off} (default for power off), or as might be expressed in any object-oriented programming languages known in the art. Accordingly, the television will always display snow when turned on, unless the channel is for example, set to the two channels for which the system has "more specific" resources. This functionality is especially useful during development, when new values are added to properties on an ongoing basis. For example, new channel values can now be added to the Imp (e.g., Channel = { MTV, sports, CNN, snow, weather} ) and the new values will automatically work with the default behavior. Additionally, at any point, refinements can be added to the Imp by providing a new item for a specific constraint set.

In addition to name and type, the containment can alternatively have initial setting information stored as an Information Directory, as shown in Fig. 4. The initial setting information is passed on to the generated instance (either a resource or Imp) at the "construction time". This is conceptually similar to the constructor parameters in C++. For resources, the content of initial settings is dependent on the resource format. For Imps, the settings will always consist of a "Partial State" which represents the initial state for the Imp. Although Initial state information may be partial or empty, the generated instance of the Imp will always begin life with a fully defined state (that is, each property has a value). This is because each property not specified by the initial state settings will be set to its default value.

The values and properties are semantic values and not enumerated types. Each Imp has its own private set of properties, and if two Imps have a property with the same name, it does not imply that the values for both properties are the same. For example, one Imp may have a property named "Color" with values: "Black," "White" and "Gray", and another Imp may also have a property named "Color," but with values "Bright" and "Dark". In accordance with an embodiment of the present invention, the properties of a running Imp can be changed, i.e., changing of its current state. The change is defined by a "Partial State" (i.e., a set of pairs of Property and Value) which is sent to the Imp as a "change state message^". Any properties not included in the Partial State remain unchanged. During this change of state, the following steps are performed:

1. The new state is checked against the containments of the Imp and a list of eligible containments is compiled.

2. The list is checked for overloaded names, and the best match for each overloaded name is chosen. 3. Any children contained by the Imp, which are not on the new list, are destroyed. When an Imp is destroyed, it is first set to the "null" state (i.e., an artificial state where no containments match) so that all of its containments are destroyed recursively.

4. Any containments on the list which are not currently contained in the Imp are generated. Initial setting information is passed to each newly generated instance. If the instance is an overload of a previous containment item, i.e., has the same name, then the shared information, i.e., its spatial position, from the previous item is passed to the new item. In accordance with an embodiment of the present invention, the containment model supports recursive structures by setting the initialization parameters of the contained Imps. For example, suppose the user 160 desires to implement a linked list of "Dogs" through an Imp named iDog. Accordingly, iDog has a property called Next

= {true, false}, where 'false' is the default value and iDog contains another iDog, e.g., through a containment item: nextdog[iDog], where the containment constraint is

{Next = true} . When an iDog is generated, its 'Next' property is by default 'false'.

When it is changed, e.g., by a behavior resource, to 'true', a new 'iDog' is generated and is now visible and contained in the first dog. This process can proceed, as long as needed, with each iteration, adding another Dog to the list. Continuing with the iTelevisionSet example as noted herein, if the user 160 desires to include a cartoon animation of sound blaring from the speaker, and also wanted to set up the constraints so that it played whenever the channel was set to MTV or sports, then the system should be capable of chaining constraint conditions with an "or" clause, which is not supported by any currently available system. The present invention resolves this problem by generating two separate, but identical containments, each with a single constraint. Given to optimizations at the storage level, there is hardly any overhead involved with this technique. Of course, each containment will have its own Initial Settings Information which must be maintained by the user to be identical. Preferably, the present invention also contemplates related solutions such as the ability to provide "or" constraints by having each containment be associated with a (possibly empty) set of constraints, rather that with just one.

Turning to the structure of the initial settings per state, in the iTelevisionSet Imp, discussed above, there is a contained Imp named channelChooser[iDial]. For example, if the iDial Imp provides a property Position = { 1,2,3,4,5,6} , which moves the dial through six different positions, then the dial should be in a different position for each different Channel that iTelevisionSet supports. To provide such dial, the user 160 should be able to specify a different "initial state" for the contained item channelChooser per state of the Channel property. That is, in general terms, the user 160 needs to specify the Initial Settings Information for a particular constraint.

However, currently available systems generally tend not to support such settings.

The system of the present invention resolves this problem by generating several containments (identical except for the respective Initial Settings) and setting each with an appropriate constraint. Here, the system provides four copies of channelChooser[iDial], with each constrained to one of the four possible Channels and set to the corresponding initial state.

Alternatively, the system can employ behavior resources to resolve the problem. The system generates a behavior resource that sends a changestate message to the channelChooser item. The system then generates four containments for the behavior resource, with each containment set to send the appropriate changestate message and constrained to the corresponding value of the Channel property.

Currently, if a contained item becomes non-visible due to a Change State operation (i.e., constraints of the item no longer match the current state), the instance which the item represents is destroyed. Later, if the item needs to reappear (i.e., due to another change of state which now matches the item's constraints), it will be reinstantiated. Between the two instantiations, the system preserves the spatial Positioning information so that the item can be redisplayed at its last screen position. However, the system does preserve the current state and the item is reset to its initial state, as specified in the Initial Settings Information of the containment. Preferably, the system may provide for direct support state persistence as an optional feature that can be set as part of the Initial Settings Information of the item.

The Event Manager The event manager or runtime manager 120 of the present invention provides the environment needed for running "runnable objects" 220 (referred to herein as 'Rimps' or runtime objects). The event manager or runtime manager 120 is built as a modular application which relies on a set of servers 122 (Fig. 3B), i.e., subsystems, which encapsulate some of the functionality needed by the runtime objects 220. The runtime manager 120 is responsible for loading and initializing these sub systems 122, and directing input and messages to them. Runtime objects 220 communicate with the servers 122 as needed in order to 'express' themselves in the multimedia domain. Communication between runtime objects 220 and servers are effectuated through a combination of internal method calls and special open interfaces, called Format Plug- ins 230 (Fig. 2), the specifics of which are detailed herein.

In accordance with an embodiment of the present invention, the runtime manager 120 limits interdependencies between different sets of functionality, e.g., the spatial position of an object is independent of the way it is displayed on-screen. This makes it easier to expand and change the functionality of each area independently. This is especially preferred for a multimedia product, given that the capabilities of multimedia systems are constantly expanding

Accordingly, the present invention is able to foster a certain level of modulaπty with the above described environment by providing for Runtime objects 220 in a certain fashion Specifically, Runtime objects have a standard way of communicating with and "attaching" themselves to Runtime servers 122, thereby making the system scalable A new server 122 can be added with little or no change to the runnable objects 220 themselves However, new types of servers 122 may be necessary to support future changes in the multimedia technology or requirements, e g , a new type of serv er 122 mav be necessary to process 3D inputs, multi-player gammg messages, artificial intelligence, etc The basic structure described herein for the runtime application 210 (runnable objects which connect with a seπes of servers) is mostly indepenαent ot the specific domain of multimedia authoπng so that it may be reused in other domains Bv simplv replacing the existing servers 122 with new servers 122 that represent a different domain of functionality (together with new open interfaces to communicate with them), a user can build applications for other domains As a hypothetical example, a user can build a system for collaborative newspaper editing by using a Layout server 122 instead of a Positioning Server 122, and having a Print Pre\ ιew server 122 instead of a Display 122 Server, etc All known servers can be used as the run time servers 122. as long as the servers aie capable of sharing a common interface used by all Rimps In accordance with an embodiment ot the present invention, this common interface includes provision for (1 ) attach and detach, (2) message dispatching, and (3) runtime mam object An "attach" message indicates that a particular Rimp is requesting to become a client of the server 122 It is the role of the server 122 to determine whether the

Rimp is of interest to it, e g , the audio server 122 determines, using the open interface protocol, whether the Rimp supports an audio output, and processes the attach request only on its affirmatπ e determination The "detach" request indicates that the Rimp no longer wants to be in contact ith the server 122 It is responsibility of each Rimp to call attach and detach as needed The default behavior of the built-in Rimps (resources and Imps) is to attach to all servers on construction, and to detach from all servers before destruction

In accordance with an embodiment of the present invention, the servers 122 use standard protocol in dispatching and passing messages to each other. Each server 122 can dispatch a message which will be sent to all other runtime servers 122.

Messages are then keyed by message identification "ID" (similar to the messages provided within the operating system sold under the trademark Windows, owned by Microsoft Corp of Redmond, WA), and each receiver can decide what (if any) relevance the message has to it For example, the position server 122 dispatches messages whenever any runtime object 220 has changed position These messages are handled by the display server 122 by updating the on-screen display of the relevant item Cooidination of message dispatching is handled by the Runtime Main Object 220

Turning now to the runtime mam object 220. this main object 220 is the top level object of the runtime application (or player) 210 and is implemented as a single object Since the mam object 220 contains a list of all loaded servers 122, the main object 220 encapsulates the ability to communicate or coordinate with the servers 122 without having to know which of these servers are actually present and active In addition, the main ob]ect 220 maintains the top-level runtime object 220 (which, by definition, is always an Imp) This adds to the overall modulaπty of the design In a preferred embodiment, this functionality might include attaching/detaching a Rimp to all loaded servers 122. dispatching a message to all servers 122, and communicating with the top-level Imp

In accordance w ith an embodiment of the present invention, as shown in Fig 3B, the event manager or runtime manager 120 is comprised of a behavior server 122, a display server 122, a time server 122, a position server 122, an audio server 122, an input server 122, and a control server 122 The behavior server 122 activates and gives objects 220 behavior, such as click, rollover, URLs, etc The purpose of the behavior server 122 is to generate an interactive system by letting objects 220 respond to and generate events, tngger actions, and pass messages Resources which support behavior will implement the open interface defined by the behavior server 122. Behavior resources can register for event notifications, respond to messages, tngger a change of state, set global variables, and the like.

The display server 122 manages visual display of runtime objects 220 onto an on-screen window. The display server 122 uses positioning information provided by the position server 122 The display server 122 defines an open interface to be implemented by the objects 220 to support visual display Preferably, an advanced interface is available for objects 220 with high refresh (or streamed) display.

The time server 122 tracks the passage of time during runtime, for the purpose of synchronizing and triggering events Objects 220 can set time markers, named points or regions of time The time server 122 dispatches messages about these markers and can be queried about them

The position server 122 manages abstract objects called "positions" Each position represents a set of reference points in 3D space. Positions can have complicated relationships with each other as further detailed in the relative position module described herein Objects 220 can generate and manipulate positions by communicating with the position server 122. The default behavior for Imps and resources is for the containing object to generate and maintain a position for each of its contained objects When the top-level Imp is generated, it gets a special root position (0,0,0)

The audio server 122 manages audio output of runtime objects 220 to the sound system of the computer The audio server 122 uses positioning information provided by the position system to generate the illusion of 3D sound. The audio server 122 defines an open interface to be implemented by objects 220 to support sound output The audio server 122 also uses the attach/detach messages to track which runtime objects 220 to play or run

The input server 122 controls and interprets mouse and keyboard input to the runtime application 210 The input server 122 dispatches notification messages to all servers 122 about mouse clicks and keyboard keys The Input server interprets low level input messages received from the operating system, translates the low level messages to higher level messages, and dispatches the higher level messages For example, when the input server receives a "mouse down" message from the operating system (OS), the input server first determines what the mouse click hit (by communicating with other servers 122, such as the display server 122) and then dispatches an "object clicked" message compπsing the object ID and the 3D position of the click

Lastly, the control server 122 provides a high level application program interface (API), which can be accessed by external clients of the runtime manager 120 to query and control what is happening withm the runtime manager 120 The control server 122 accepts messages through an IPC (inter-process communication) protocol and translates the messages into calls to other servers 122 or to runtime objects 220 For example, an external client can send a change state request via the control server 122 to a specific runtime object 220, query a runtime object 220 for a list of its containments, or install a callback hook for processing mouse clicks The control server 122 has a companion control client which can be generated outside the process boundary and can be used to communicate with the control server 122 The control client is used by applications such as the IDE (integrated development environment) 200 which must take control of the runtime application 210

In accordance w ith an embodiment of the present invention, a VisualObject Studio includes resources as one of the basic classes Resources provide one of the mechanism for extending the functionality of the system Resources are essentially wrappers to externally defined code, called Resource Format Plug-ins The mam advantage of the plug-in architecture is that it fosters the extensibility of the inventive system and allows outside developers to extend the system without integrating or compiling the core code This is important because it encourages specialization, better migration paths, leveraging of existing code, and the facilitation of third party development

First, developers can specialize the system for specific uses by adding handling of specialized formats Second, as multimedia standards evolve, new types of multimedia functionality may alter the way multimedia is handled Accordingly, it is important that the core code be free from assumptions about specific format or types of use, thereby enabling the code to evolve and avoid becoming outdated Third, there is a lot of existing code (usually owned by third parties) that handles formats or does other multimedia functions Accordingly, the plug-in architecture of the present invention makes it easier for a given user (or for a third party) to wrap their existing code as a plug-in to the inventive system of the present invention Fourth, the versatility of the present system increases with third party development of extensions to the core svstem Preferably, the code format of the system should remain independent and support open interfaces, so as to permit third parties to add their formats or technologies without having to compromise the propπetary nature of their own code

For example, Macromedia does not publish information about its Director file format so as not to compromise their proprietary technology However, in accordance with an embodiment of the present invention, such software owner can develop a plug-m format for its software product without compπsing its confidential information about the product In this regard, a loose distinction can be made between those areas of the system which represent core functionality or core architecture, and the additional functionality which can be added to the system by developing plug-ms accordingly, there is no "built-in" support for the most advanced features, instead, they are provided as Plug-ms Core functionality therefore includes ( 1 ) component structure (I e , Imps, states, containment), (2) multimedia pπmitn es (l e , basic functionality provided by the runtime servers 122 such as display or behauor), and (3) access to storage (I e , storage classes, database) It should be noted. howe\ er, that core functionality does not include any advanced multimedia functionality, specific data formats and uses, or interactivity

On the abstract level, resources represent a collection of data and functionality which is storable in the VisualObjectStudio database 150, editable from the VisualObjectStudio IDE 200, and which can be "played back" or expressed at runtime on some mix of the audio, \ isual and behavioral domains For example, if a bitmapped picture, l e , a BMP file, was implemented as a resource, the following might occur (a) the binary data of the BMP is stored in the database 150; (b) the IDE 200 launches a tool to allow a user to edit the data in the database 150, (c) at runtime, the runtime player 210 in conjunction with the runtime manager displays the bitmap on the display surface Accordingly, the system 100 supports a vaπety of different types of resources, e g , bitmaps, AVI movies, wave audio, 3D objects, behavior scπpts. timers, etc Each type of resource being called a Resource Format would be associated with exactly one recognized Resource Format Withm the system, the resources would preferably be treated as if they were exactly the same All functionality which must be specialized for each format would be hidden behind the plug-m mechanism

Accordingly, the key features of the plug-m mechanism are preferably structural such that ( 1 ) external developers are provided with a seπes of "open" interfaces wherein each interface has a name, is focused around a specific group of resource related functionality, and represents function calls that the system would like to call for a given resource, (2) plug-m providers can implement a Plug-m Format

Dynamic Link Library, w hich is a DLL or reentrant code that implements some subset of the provided open interfaces for a particular format, (e.g , a BMP Format DLL would implement the open interface for displaying pictures on the screen), (3) a mechanism is provided for Plug-ms to be registered with the system, and duπng this phase, information about the plug-m is stored in the database, (4) when a resource is generated, it is always associated w ith a particular format, such that the format of the resource determines how it will behave by determining which function implementations are used, and (5) when a subsystem wishes to make a call to one of the open interfaces of a resource, it must first check whether the interface is supported by the format that the interface supports The subsystem can proceed with the function call, such that by way of example, the Display server 122 would check each resource that attaches to it to see whether it supports the interface for visual display and if it does, it makes the appropπate calls to the interface m order to display the resource on screen In accordance with an embodiment of the present invention, the plug-in architecture is preferably directed to an "Infoπnation Directory", as shown in Fig. 4 The Information Directory is a system of hierarchically structured storage which is used at the database abstraction level (storage classes) 140 to provide an extensible and open means of storing "meta" information about objects Since the users of the system want to store additional information about many objects of vaπous types within the system 100, e g , Imps and resources might have a list of spatial reference points associated with them, a containment might have initial setting information, the information generally is in the binary form It is appreciated that the exact format of the data can be easily changed between development versions. Also, the information is often only optionally present for a given object Therefore, the preferred embodiment does not necessarily require the database to be structure dependent

Nevertneless. the full functionality of the system 100 requires that the database 150 be at least structured so as to notify a given user that a particular piece of information is a\ ailable for an object, and to retrieve the required piece of information exactly without incurring an overhead by retπeving an entire binary large object "blob," especially given that the total amount of information stored for an object can often be quite large Also, in the case of resources, much of the information stored for an object will need to be defined and managed by the external plug-in which handles that resource format For example, a sound resource may need to store additional information about the timing and synchronization points withm the sample, and a behavior resource may need to store a complicated set of initial settings which determine how it will act These requirements necessitate that the storage scheme be extensible on a dynamic basis Accordingly, the structure of the present invention provides for an information directory as a compromise, allowing the balancing of these needs, that accords with two attendant data schemes the data blob and the information directory installation InfoDir

In the context of the data model of the present invention, a blob is a contiguous binary block of data of arbitrary size which can be stored and retπeved as a single unit A special kind of blob is a "text block" which is simply a data block of ASCII characters which can be parsed line by line At the storage level, blobs are directly supported by the StorBlob and StorText classes) Throughout the system 100, classes are considered "seπahzable" if they implement a means to store and restore themseh es from a simple blob Many classes support seπahzation, providing the basic means for persistence within the system

Turning now to Fig 4, an InfoDir 400 is a database object with a name 410, a sections element 420 and a blob 430 The InfoDirs 400 are named with a simple text stπng and names 410 can be matched by lexical compaπson, although the name 410 can be blank for an unnamed or root directory An InfoDir 400 may optionally contain a single blob 430. considered the "content" of that directory Also, the Infodir

400 may contain sub-sections, where each section 420 is another InfoDir 400

The internal classes of the system 100 can preferably store and restore themseh es to an InfoDir 400 These "storables" work by creating their own subsection withm the InfoDir 400, and then seπahzing themselves to a blob 430 within that Section 420 For example, a Reference Points object would first generate a subsection (actually just another InfoDir 400) named "Reference Points" and then seπahze to the Blob 430 stored withm that section 420 When asked to retπeve itself from a given InfoDir 400, the class will first look for a subsection with the correct name 410. and if one is present, restore itself from the contained blob 430 Although this method has the limitation that (at most) one object of a given type can be stored at the same "section" lev el, the present invention does provide that this can be easily worked around by adding an additional level of sections 420 Preferably, the functionality of the present invention therefore has storage classes associated with InfoDir 400 and contains certain information as follows in the listing by the InfoDir 400 for a respective Imp which contains the following sections

ReferencePomts (optional) A set of spatial reference points for the Imp itself

ConnectionMap (optional) A tree of positioning relationships

ReferencePomtAhases and similarly, with respect to the resources, the InfoDir 400 for a resource contains the following sections:

ReferencePoints (optional) A set of spatial reference points for the resource

Metrics Size information about the resource

FormatlnfoRoot Section which will contain any third party information stores for the resource by the Format plug-in

While in a containment, the InfoDir 400 for a containment (within an Imp) contains the following sections:

Formats ettingsRoot Section which will contain any settings information used by the resource format being contained

ImpSettingsRoot/PartialState Contains a partial state indicating the initial state of an Imp

In accordance with an embodiment of the present invention, the event manager 120 includes a relative positioning software module. All runnable objects 220 within the system 100 can have a spatial position, which ultimately controls where the object will appear on-screen, or how the object will sound when played through the sound system or how the object behaves. Positioning information for all objects is maintained in a consistent, abstract form so that it is independent of particular formats and object types, and also not dependent on the display technology. The relative positioning module defines and uses the positioning information because objects do not have an absolute position, rather, each object is defined and positioned relative to other objects.

Preferably, since a team of people need to work simultaneously on a set of objects that will be displayed or "run" together as a coherent group, it is necessary for the relative positioning module to provide for versatility on a visual level. In order for this to work in a visual based group environment, positioning information must be maintained as shared information. For example, to produce a bar scene, a team of graphic artists and animators may need to split up the work so that one can draw the backgrounds, another the bottles, glasses and props, and another animates the characters. In addition, there may be other professionals, such as sound engineers and programmers, who are contributing additional resources to the scene that are position related (such as background sounds, or click points).

All these users 160 would need access to the position of objects in the scene. However, since the scene is a work in progress, each user's work can and will influence the correct position for other objects (for example, if a bar in a bar room scene is extended, then the position of the bartender may have to move in accordance therewith). What is needed is a way to specify these dependencies between objects, so that if one object is changed then other objects are automatically adjusted.

Accordingly, the relative positioning module provides for alteration according to the dynamic positioning of the Imps. The set of objects, contained by an Imp, changes dynamically as the current state of the Imp changes with the state-based containment structure of the Imp. Since Imps can contain other Imps, wherein each being in one of several states, the total number of combinations is very large. While previous approaches stressed the assigning of an absolute location to each object, these solutions are inadequate because positioning will often be different for different subsets of objects (e.g., a class called iBarStool class might supply three different looks for the barstools, requiring that the customer thereon be repositioned for each kind of stool). Therefore, a solution cannot contemplate that separate positions be specified for each permutation because there are simply too many of them. Accordingly, the system of the present invention strikes a technological compromise through the relative positioning system of the inventive relative positioning module.

Specifically, much of the same types of changes described herein exist when an object is reused in another context, but the positioning information may need to be adjusted to the new context. However, because the object is in effect shared in both contexts, the object must actually dynamically adjust itself to each context in which it is located. The relative positioning module of the present invention dynamically adjusts the object to each context.

Since several different subsystems may use positioning information, it is important to keep this information in an abstract form that is generic enough for multiple purposes so as to ensure maximum portability of the invention. Currently, the Display server uses positioning information (along with a definition of camera position) to project objects on the screen; the Audio server uses positioning information to set stereo volume levels to generate the illusion of 3D sound; the Behavior system uses positioning information as input for interactivity (collision, object in area, etc.). Once configured as such, the semantic referencing within the system utilizes the relative positioning. The relative positioning model uses a true 3D model, which places all objects in 3D space with three coordinates signifying the location (x,y,z) and three coordinates signifying the angle of rotation (a,b,c). Preferably, offsets and distances between objects are also stored as three coordinates signifying displacement and three coordinates signifying angle. The interpretation of this 3D model is left up to each subsystem or server that uses the information.

The relative positioning module can be classified according to two sub-topics: "design time" and "runtime". During the design phase, designers must specify positioning information for the sub-elements of an object. This information must be flexible enough to describe the position of the elements of the object in whatever state and whatever context the object is used. During runtime, the Position server uses the information stored with each object to "bind" all running objects to an absolute position and to update those positions as objects change and move.

In accordance with this structure, all objects in the system 100 are assumed to have an origin point. The origin point is an arbitrary point which represents the local zero {0,0,0} within the scope of the object. It provides the point of reference for talking about the location of the object. If the object is at location {20, 20,50}, then the origin point of the object is positioned on location {20, 20, 50} . It also provides the reference for objects that are positioned within a local scope. For example, the position of object A can be set at an offset of {20, 10, -50} from object B. That is, the origin point of object A is exactly {20, 10 -50} away from the origin point of object B. Preferably, each resource format is responsible for interpreting the meaning of the origin point for that type. For simple types, the preferred determination is to provide for the origin point to be set to the lower left corner of the resource. Turning now to Fig. 5, there is illustrated an example of how a position is stored as an offset from the origin point of a given object, i.e., the scope of the position is local to that object. For example, if the position of object B is defined as the offset {20, 10, -50} from object A, specifically, from object A's origin point, then object B position is defined to be in the local scope of object A. Since, the positions of objects A and B are now related, the absolute position of object B changes whenever the absolute position of object A changes. That is, even though object B's local position remains the same (i.e., it is at the same offset from object A), the absolute position of object B changes with A. Accordingly, object B is positioned by object A, or conversely object A is the positioner for object B. Fig. 6 illustrates an example of an implementation of the correct map with reference and origin points. Reference points are a way of specifying and naming significant locations within the local scope of an object. The points are relative to the origin point of the object, and the scope of these names will be local to that object.

With reference then to the attached figures, if rBarRoom is a picture of a bar, we may want to name some relevant points of reference within the picture, such as the bar counter and the exit door, as shown in Fig. 6. In actuality, these origin and reference points are stored as a name (a simple text string), and an offset from the origin point of the object.

Reference points provide the only legal way of referring to positions within an object (even the origin point of the object cannot be directly referred to). In short, this means that: (1) any item, which is positioned by an object, will be stored as an offset from a particular reference point within the object; (2) any dynamic reference to an on-screen position (e.g., a request from the behavior system to move an item to a specific location) can only be to some named reference point, or to an offset from a named reference point; and (3) accordingly, that all positioning and referencing must ultimately be resolved to reference points and offsets. Given this strong enforcement, users are assured that they can make as many visual changes to an object as needed as long as the reference points are adjusted accordingly, as shown in Figs. 6 and 7.

Both resources and Imps can have reference points. While the use of reference points on a visual resource such as a bitmap is more straightforward, Imps can also use reference points. One use of Imp reference points is to define placemarkers ("stand ins") for objects. A contained object can be set at a given offset from some named reference point in the space of an Imp. This provides the Imp designer with a way of specifying and adjusting locations independent of the contained elements. Another use of Imp reference points is to provide "attachment points" for clients of the Imp. For example, the iBarStool Imp may define a reference point called "seat". This would help users of the iBarStool so that they can position other objects relative to the "seat" of the stool.

Furthermore, Imps can also define a special kind of reference point known as an alias. Aliases are a powerful mechanism for defining dynamic reference points that change as the Imp changes state and is simply a "pointer" to another reference point. The alias has its own name and contains information about the reference point for which it is an alias. An alias defined at the Imp level must point to a reference point within the scope of the Imp. More specifically, the alias must point to a reference point of one of the contained items of that Imp.

For example, suppose the iBarRoom Imp contains two barstools of type iBarStool:leftstool[iBarStool], rightstool[iBarstool]. Users 160 of the iBarRoom Imp can position a character object sitting on one of the two sitting locations by adding two alias points to the iBarStool Imp, such as: "sitting position 1"= leftstooLseat, and

"Sitting position 2"= rightstool\seat.

The advantage of using aliases is that the alias points follow wherever the objects they point to go. For example, at different states of the Imp, the stools are positioned in different locations within the bar. Through the use of aliases, the "sitting position" reference point adjusts dynamically as the layout of the bar changes. Aliases use semantic referencing for contained items This allows them to participate in "containment overloading" that can occur when an Imp changes states. In the example above, "sitting position 1 " is an alias for "leftstool\seat". It is appreciated that the alias only makes reference to the name of the contained item. In different states, the "leftstool" item may actually refer to different objects. The alias is bound dynamically at runtime and will always be resolved to the actual containment This timing allows the generation of state-based reference points. Through the use of chaining, aliases can be transitive, that is, an alias may point to an alias of another Imp. Thus, aliases can be chained across the containment hierarchy. The present invention also provides for a way of specifying positioning relationships between a group of sub-elements of an object through the use of something called connection maps Connection maps are unique to Imps, and each Imp has exactly one connection map, which is specified duπng the design process. As detailed herein, Imps can contain a group of zero or more "contained items." Each contained item has a name (the system supports unnamed items by creating a default name for them), and a type (which is some runnable object, either Imp or resource). Sometimes several items will share the same name, this indicates that they are overloaded Containment items may be local to a specific set of states tied to the state of the Imp, which is specified through the use of constraints. Accordingly, an Imp is provided with a connection map for informing the system 100 of the relative position of all contained items (sub-items) for that Imp. The connection map only deals with relative position because it can only determine the positions of sub-items within the local scope of the parent Imp. The absolute position of the sub-items will only be determined at runtime, when the parent Imp is itself positioned.

A connection map is structured as a tree of nodes and links. With regard to the node element within the connection map, each node is simply a name, and each name represents some contained item. So, in the bar room example iBarRoom in Fig. 8, there would be nodes labeled background, leftstool, πghtstool, customer, bartender - one node for each named item in the Imp In addition, there would also be one extra node known as the root node The root node represents the containing Imp itself (in the above example, this ould be iBarRoom)

There is also provision within the inventive system for interconnectivity, such that a link is a directed connection between two nodes The link indicates the distance between those two nodes The direction of the link indicates the positioning relationship specifying which node is the positioner and which node is the positioned (1 e , its position will always be relative to the positioner) In a simpler case where a user only wants to arrange the items in iBarRoom as shown in the attached figures above picture, the connection map would be a simple tree with the iBarRoom node at the root, and all the items would be linked to the root, thereby indicating that they are positioned relative to the position of iBarRoom

In addition to a distance, a link can also specify the reference points to use when "connecting" the two nodes For example, in Fig 9, the link between the stool and the root may connect a reference point named "leftstool\seat" with one named "barroom\floor" By specifying reference points, the user indicates how to store the relative distance (given that all distances are always stored as an offset from some reference point) At runtime, the absolute distance will be computed using the specified reference points Hence, if one of those reference points is adjusted, the distance ill be adjusted accordingly When a link does not specify a reference point, or when the reference point specified does not exist, a default reference point is used Default reference points are defined at the object level Each object with reference points must specify its default reference points There are actually two default reference points for each object (which can be the same) The "positioner" point indicates the reference for other objects to use when using the current object for a positioning reference The

"positioned" point indicates the reference for "attaching" or positioning the current object relative to another In most cases, these w ill be the same reference point, but the option of using separate points is available

In above example, all items are directly positioned by (or "linked to") the root of the connection map, representing the containing Imp This is the default

J - configuration with which a connection map begins its life. It allows all items to be positioned relative to the containing object While this configuration will often be sufficient, there are cases where a more complicated positioning relationship is needed For example, in the case of the iBarRoom Imp, there is a natural positioning relationship between the bartender item and the background item. Given that the background contains the drawing of the counter, if the user would like the bartender item to be positioned relative to the position of the counter on the background then the system needs to provide for two things '( 1) to have the bartender node be linked to the background node, and (2) to set up the link to use the reference point name "background/counter", as shown in Fig. 10. Links and reference points work together to allow the user to specify the positioning relationship exactly. As such, the customer item may also be linked to the background, but would be linked to a reference point named "background\exιt " This will ensure that the customer will always be walking in the door, no matter where the door gets drawn, as shown m Fig. 10.

The prefeπed rules for creating links are simple, and their purpose is to ensure that the connection map remains a connected tree with the containing item at the root. In a preferred embodiment, the rules are such that. (1) each node other than the root must have exactly one positioner, but the root can have no positioner; and (2) all items must be transitively positioned by the root node (they must be at least indirectly connected to the root)

Due to the semantic nature of connection maps, the maps support the overloading abilities of the state-based containment model. For example, in the iBarRoom described above, there may be several different objects contained in the scene with the containment name of "bartender", and perhaps each bartender is constrained to a particular value of an "OnDuty" property so that, as you change the OnDuty property, a different bartender appears In this case, the node name "bartender" is actually representing a group of objects Having a single node means that the different bartenders will, in effect, share the same position. It is appreciated that although a group of overloaded items share the same logical node in the connection map, this does not mean that they will appear at exactly the same position on-screen This is because reference points are local to each item. So, in yet another example, if the "bartender" node is linked to the background via a reference point named "bartender\elbow" (so that it is always positioned at the correct height for the counter) then, considering that each different bartender may define the local "elbow" reference point in a different location, the position of the bartender will need to be adjusted accordingly each time the bartender item is switched. A similar situation would result even if no reference point is specified for the bartender link. This is because when no reference point is specified, the default "positioning" reference point w ill be used Once again, this point is local to each object and is likely to be defined in a different place for each bartender

Having a single node item for a group of overloaded contained items provides a powerful way of simplifying the positioning of complex scenes However, this feature may not always be desirable in the case where the objects do not have the same reference points, or w here the positioning must be hand adjusted for each object. When this is the case, special "override" links can be added to the connection map to overπde the default information in the link Specifically, information about this point relates to a link that connects two nodes withm the Connection Map, and specifies the distance between them and (optionally) the reference points to use when connecting the objects This information is known as the default link because the information specified is true for all items that match the node names However, in addition to the default link, the link between two nodes can further be configured to include any number of additional overπde links An override link adds information for a specific pair of items that overrides the corresponding information in the default link. By way of example, if there are. say three items in the iBarRoom Imp named bartender bartenderO[ιSammy], bartender 1 [lBilly], and bartender2[ιCathy], then if there were two different backgrounds such as backgroundO[rSleazyBar], and background l frElegantBar]. in theory then, there are six different combinations of background and bartender that can be displayed withm iBarRoom For each one of those combinations, the user may want to add an overπde link that specifies a different reference point or a different distance to use for that combination. Any combinations which aie not specified with an overπde link will use the default information. Accordingly, overπde links will preferably contain the following information. (1 ) an index of positioner items that identifies an item from a group with the same name; (2) and index of positioning items; (3) a positioned reference point (optional, if no present default is used), and (4) a distance (optional, if no present default is used) The configuration of the ov erride links within the system 100 is shown in Fig. 1 1 Two ovemdes are illustrated or specified in Fig. 1 1. The first overπdes the distance between the bartender and the background in the case that the iSammy item is used together with lSleazyBar The second overπdes the distance for the case where iCathy is used together with lElegantBar In addition, the second overπde specifies that the "background\table" reference point be used in place of "background\counter"

During runtime, Imps and resources are loaded into memory, and are "run" by having them attach to each of the runtime servers 122. As part of this process, each runnable item is bound to a particular absolute position As items move and change, these positions are constantly tracked and updated. This role is served by the position server 122. The position server 122 is one of the runtime servers 122 (or subsystems) withm the server module or runtime manager 120 The position server maintains a consistent model of all the referenced positions in 3D space, handles updates and changes, and implements the positioning relationships defined by the links and reference points, and the like Other systems, such as the display server 122 or behavior server 122, can query the position server 122 for information about the position of a particular item In this way, all runtime positioning information is available from a single source

It is appreciated that positions are abstract objects which represent a location in space that is of relevance to the system Positions are the main way of interacting with the Position server 122, as the server provides the means to generate, destroy and update positions There is a simple relationship between positions and other runtime objects 220 every "placeable" object (1 e , all Imps and resources) has its own position In accordance with an embodiment of the present invention, the position server maintains the following information for each position (1) all positions have an ID with which they are identified, which is used internally, (2) reference points for each position can have any number of named points withm its local scope, (3) a positioner for all positions can be positioned by something other than a single special root position, and must therefore be identified by its ID, (4) a link for each position is "linked" to its parent position, (5) the linking information must include (a) a distance to positioner that is the default distance between the object and its positioner (note that this value does not change when the position, or object associated with it, is moved, and instead, this value indicates only the initial position of the object m relation to its positioner), (b) reference points for linking a reference point on the positioner, and reference points on the positioned to be used when computing the distance between the items, and (c) motion delta absolute distance that indicate when the item has moved from its initial position (note that this number is updated as the item is moved)

As can be seen, this information is very similar to what is stored in the connection map The difference, however, is that the connection map represents static data for a particular Imp, while the position server 122 contains dynamically updated data for all running objects

With respect to the positioning map, the positioning map is the internal representation maintained by the position server 122 It is a dynamic model of all the positions in the system and their current absolute location The positioning map is actually a tree which starts from a root positioner and proceeds from positioner to positioned The tree includes all positions in the system (given that all positions other than the root must always have some positioner, the postilions will always form a fully connected tree) The root positioner is always at location 0,0,0 The location of all other positions can be directly computed given the link information between them and their positioner The positioning map is of course very similar to the connection maps stored withm each Imp However, consideπng that it is built and maintained only at runtime, it is actually much simpler For example, there is only one link between positions, rather than a group of links This is because the current state of each Imp is already known resulting in the elimination of overπdden containments Also, the position map uses IDs rather than semantic referencing, because all names have already been bound to IDs Most importantly, the positioning map can compute the absolute position of each item, since all positions have been reduced to a single tree with one root position Positions themselves are generated and deleted from the positioning map by runtime Imps Given that Imps are the only containers for objects, the Imp class can generate positions for all objects Each Imp generates positions for each of its contained items The root Imp in the system is always at the special root position When creating positions, the Imp takes into account the information stored in its local connection map such that the position server 122 has no direct knowledge of the connection map Instead, it is the responsibility of each Imp to bind the semantic information in the connection map to actual items and links, and pass this information along to the position server 122 when the new position is generated

Each runtime object 220 (Imps and resources) is given a position at generation The position represents their location for their entire life cycle, but in actuality, this is accomplished by having the Imp or resource hold a copy of the ID position Conversely, each position also holds a copy of the ID of the associated object Positions can be moved as result of behavior or user actions When a position moves, the location of the associated object is updated, as well as that for all the objects which it positions As shown in Figs 12 and 13, when a position is moved, what actually gets updated in the positioning map is the "motion delta" field stored in the link between an object and its positioner This is maintained separately from an initial position of the object, so that at any given moment the system knows both what the object's initial location is, and what its current location is as an offset from that initial location This is important because the initial location of the object can get updated independently (and dynamically) as its positioner or positioner link changes. When contained items are overloaded (i.e., where they share the same name), a shared position is generated for all items of the group. The State-based containment ensures that at most one item from this group will be "visible" at any given time. By sharing a common position, all items with the same name will appear in the same location, no matter which state the Imp is in. Although overloaded items share a position, there are still location updates that may occur when an Imp changes state, and one overloaded containment is replaced with another. This is because each item will generally have a different set of reference points. The reference points affect how the item is positioned in relation to its positioner object. The positioning relationships captured in the positioning map are generally static for the life span of the Imp. However, the Position server does allow limited manipulation of the positioning tree; this allows a behavior resource to dynamically change positioning relationships. One change that is allowed is a "change parent" function which can dynamically reassign the positioning item of an object. This can be used to accomplish advanced behavior. By way of example, suppose an object iTruck displays a pickup truck. In this case the object can be designed so that whenever another object is dragged onto the truck then it is automatically dropped into the back of the truck (something which might require a custom behavior resource to be running within the truck). But, if the user were to say, pick up a crate object (rCrate) and drag it onto the truck such that now it is displayed in the back of the truck, the user would accordingly expect that if one started moving the truck, the crate would also move. The way to effectuate this is for the iTruck object to ask the position server 122 to change the parent of the crate from whatever it was to iTruck. By making the truck the parent of the crate, we effectively move the crate into the local scope of the truck, so that any location changes to the truck will directly affect the crate.

The position server 122 dispatches notifications each time the location of an item changes. Because all items are positioned in relation to each other, a change to one item will generally propagate down the tree and set off changes in other items. In order to allow efficient processing of these changes, the position server groups all related changes into a single notification known as a change session. A session is basically a list of all items whose locations have been changed as the result of a single action (e.g., moving an object, changing a reference point, etc.). Each runtime server processes this notification as it sees fit. For example, the display server adjusts the on-screen position of each item in the session.

The invention further provides for indirection and semantic referencing. Indirection occurs whenever an entity stores or uses an indirect reference to some information, rather than the information itself. "Semantic Referencing" is a general term for any kind of indirection within the system where the binding is based on text string matching. The design of the system tries to take advantage of indirection and especially semantic referencing wherever possible. For example, in the spatial positioning system, one can set the position of an object relative to another position by specifying the "name" of the point one would like the object to be positioned by. The "name" of this point (called a reference point) is simply a text string that is matched with the list of named points at runtime.

Provision of such semantic referencing offers several different advantages. In the case of indirection, which is the ability to refer to something indirectly, means that you are not unnecessarily dependent on the exact nature of something to which one is referring. Hence when one has an indirect reference from one entity to another, one should be able to switch around or change the entity being referred to without having to update the refemng entity and without inadvertently "breaking" the reference. Whenever one has an indirect reference, there is always some kind of "lookup key" which acts as the way of binding the reference to what it refers. A user can do anything he desires to the referred object so long as this key is not changed. The more flexible this key is, the "less bound" the reference is and the more changes a user can make to the referred entity without the reference having to know about it (this is similar to coding schemes such as found in C++ or other coding schemes, where, for example, virtual functions are bound by an index in a table, so you can change the implementation as long as you keep the same index). Moreover, the contemplated structure is such that groupware can be provided for in this manner of loosely bound references. Obviously, loosely-bound referencing is of great advantage when one needs a group of people to work together on the same thing, given that the loosely bound referencing minimizes the dependencies between each groups member's work. This allows users to work in their own domains and have other people use the work by adding references to it, while also protecting one user's changes from "breaking" the system. Loosely-bound referencing is also important during the reuse phase when an object may be reused outside of its original context. In this case, the system ensures that the object is not unnecessarily dependent on anything in the original context.

Semantic references (i.e., indirect references where the "key" for binding references is a text string, usually a name) are a useful form of indirection. They are generally very loosely-bound, because a user will usually be able to change almost everything about a referenced entity, as long as the user maintains the same name. In addition, semantic references are a simple and intuitive means of referring to things because it is close to natural language and gives the user flexibility in describing and categorizing items.

In terms of binding of references, binding a reference to what it refers to can be done either at design time (within the database), or at run-time. Design time binding will generally be more efficient, while run-time binding will be more flexible.

The primary client of run-time binding will be the Behavior server (see Runtime servers) because it will need the ability to reference information and objects in a dynamic way that cannot be pre-defined. For example, a behavior resource representing a light switch may decide that when clicked on, it will send a change state message of "lighting=bright" to all Imp objects which are contained in the same

'room' with it. This kind of behavior could only be bound to specific Imps, properties, and states at runtime. This is another advantage of using semantic references, because they are easy to synthesize dynamically from within a behavior script. Concerning names and IDs used within the system, many references are intended to be bound by an ID (database) rather than by a name (for example, the type of a containment, i.e., the object to which it refers is stored as an (Rimp) ID within the containment data). This is because IDs have two advantages over names: (1) the database can resolve them much faster; and (2) it is much easier to validate the integrity of the database when the dependencies are referenced by ID. Regarding the rule base therefrom: ( 1 ) most database entities have both a name and an ID, the items of which are not stored directly in the database (for example, anything in an InfoDir) will have only a name; (2) the user interface will always present names (wherever possible this will be resolved to an ID before storing the reference); and (3) the behavior system will be able to refer to entities by name (this can be resolved by revision to IDs at runtime).

The following are some examples of indirection within the inventive system: (1) State-base containment States and partial states are ways of selecting or referring to groups of contained objects within an Imp. States can be specified by ID (using

IDs of values) or during runtime using semantic referencing (e.g., "color = black, size = large"). (2) Containment Names are used to refer to a particular contained object (or during design time, to a set of contained objects which share that name). They are currently used semantically during design-time to store positioning relationships (connection maps), and during runtime by behavior resources that need to refer to particular containment. Because of overloading, the containment name has no single matching ID and can only be referred to semantically. (3) Reference Points are spatial positions on-screen or within a runnable object and are referred to by their reference point name, such that this name has no ID equivalent. Reference points are used to determine positioning relationships, and can be used by behavior resources and are always resolved dynamically at runtime. (4) runnable object type runnable objects always have a type which is either an Imp or a resource. At design time, users choose the type for a runnable object (seen as a containment of some Imp) by name. This name is bound to the corresponding ID before storing. During runtime, the Behavior server can generate new runnable objects dynamically. These objects can be bound to types by name (TBD). (5) The preferred embodiment is to allow behavior objects to respond to event messages which are essentially semantic, and this provides an indirect way of referring to behavior to be executed.

Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

Claims

What is Claimed:

1. A system for providing interactive development environment for automated visual configuration of objects, comprising: a configurator module for configuring and manipulating said objects and generating a sequence of scenes by a user, each scene having at least one of said objects; a media manager for managing said objects in said sequence of scenes; and an event manger for running said sequence of scenes to provide visual presentation of said objects.

2. The system of claim 1, wherein said configurator is a state-based configurator and wherein said sequence of scenes are event-driven.

3. The system of claim 1, further comprising a multi-object storage system; and wherein said media manger controls said storage system to deliver said objects to said event manager in real-time.

4. The system of claim 1 , wherein said configurator is operable to link a first object to a second object, thereby defining a relationship between said first and second objects to form a scene.

5. The system of claim 4, wherein said second object's position represents an offset from said first object's position.

6. The system of claim 5, wherein said configurator is operable to maintain the offset by adjusting the absolute position of the second object when said first object is repositioned by said user.

7. The system of claim 6, wherein each object includes meta information which can be edited by said user to adjust the position of said each object.

8. The system of claim 1 , wherein said objects are smart objects, each smart object comprising a collection of resource objects.

9. The system of claim 2, wherein said event manager is operable to run said sequence of scenes based on user interactions.

10. The system of claim 1 , wherein said system comprising an extensible open architecture to support plug-ins.

1 1. The system of claim 1 , wherein said configurator is operable to generate software codes to run said sequence of scenes in accordance with said user's configuration and manipulation of said objects.

12. A computer based method for providing interactive development environment for automated visual configuration of objects, comprising the steps of: configuring and manipulating said objects and generating a sequence of scenes by a user, each scene having at least one of said objects; managing said objects in said sequence of scenes; and running said sequence of scenes to provide visual presentation of said objects

13. The method of claim 12, further comprising the step of configuring said objects using state-based containment, thereby generating sequence of scenes that are event-driven.

14. The method of claim 12, further comprising the step of retrieving said objects from a multi-object storage system in real-time to run said sequence of scenes.

15. The method of claim 12, wherein the step of configuring and manipulating includes the step of linking a first object to a second object, thereby defining a relationship between said first and second objects to form a scene.

16. The method of 15, wherein said second object's position represents an offset from said first object's position.

17. The method of claim 16, further comprising the step of maintaining the offset by adjusting the absolute position of the second object when said first object is repositioned by said user.

18. The method of claim 17. wherein each object includes meta information which can be edited by said user to adjust the position of said each object.

19. The method of claim 12, wherein said objects are smart objects, each smart object comprising a collection of resource objects.

20. The method of claim 13, wherein the step of running runs said sequence of scenes based on user interactions.

21. The method of claim 12, wherein said interactive development environment comprising an extensible open architecture to support plug-ins.

22. The method of claim 12, further comprising the step of generating software codes to run said sequence of scenes in accordance with said user's configuration and manipulation of said objects.