WO2003098847A1

WO2003098847A1 - Multi-media interaction system

Info

Publication number: WO2003098847A1
Application number: PCT/EP2002/005604
Authority: WO
Inventors: Josef J. J. Helmes; Marigot G. J. G. Goossens
Original assignee: Helmes Josef J J; Goossens Marigot G J G
Priority date: 2002-05-22
Filing date: 2002-05-22
Publication date: 2003-11-27
Also published as: AU2002367948A1; WO2003098847A9; EP1506632A1

Abstract

The invention pertains to a multi-media system having at least three users for handling objects such as text, sound, images, video clips, light, motion etc., the system being suitable for interaction between the users, wherein the first user communicates in a first format, the second user communicates in a second format different from the first, and the third user communicates in a third format being the same or different than the first or second format, wherein interaction between two of the users takes place via a message send from one user to the other over a datapath connecting these users, wherein the datapath comprises a single backbone path suitable for carrying messages in a standard format, and each of the said users are connected to this backbone datapath via a corresponding module which is capable of translating the message in the format of the user and/or vice versa, and also is capable of determining whether a message that is carried on the backbone is meant for the corresponding user.

Description

Multi-media interaction system

The invention pertains to a multi-media system comprising at least three users for handling objects such as text, sound, images, video clips, light, motion etc., the system being suitable for interaction between the users, wherein the first user communicates in a first format, the second user communicates in a second format different from the first, and the third user communicates in a third format being the same or different than the first or second format, wherein interaction between two of the said users takes place via a message send from one user to the other over a datapath connecting these users, wherein the datapath comprises an interface.

Such a system is known in particular from the entertainment industry, embodied for example in television shows, amusement parks, live rock concerts and theatre shows. The known system is developed in particular, but not necessarily, for real-time interaction between users that handle different objects. It is for example common in an amusement park or fairground that light, sound and motion are created immediately following the hitting of a switch such as when a little cart moves along a certain position in a dark ride and a howling ghost appears, suddenly being enlighted by a spotlight and weaving with a sword. Dedicated users for which often a specific format for communicating is developed handle these objects. Sound related users often use the MIDI format or MP3, video is often handled via the WAV or MPEG format, motion is often handled via the CAN format, and light is commonly handled via the DMX format. In the known multimedia system, if interaction between two users takes place, this is initiated by one user sending a message to the other user via a direct datapath connecting the two users. If the two users communicate via the same format, for example a light panel for regulating the light (messages are send e.g. as DMX-out) and a spotlight for enlightening a specific object (message is received e.g. as DMX-in), the interface is normally a standardised DMX-in/DMX-out interface comprising a male and female DMX plug. In this case by shifting a button on the light panel, a message, e.g. "spot on", is send over the datapath, via the DMX-plugs, to the light spot, which receives the message. The spotlight is programmed to understand the message and thus reacts by turning itself into the state "on". When the two users however communicate via a different format, a non-standardised interface has to be used which is able to translate the message which is in the format of the sender into a message that is in the format of the receiver. This way a multimedia interaction system can be build, wherein for each type of interaction between two users that is foreseen, a dedicated interface is build in the datapath connecting the two users. The users itself are programmed to understand the messages and upon receiving a message, the corresponding action is e.g. taken from a memory, in particular located in a controller connected to the user, and the action is carried out.

The known system however has some disadvantages. First of all, for each interaction between two users a separate datapath has to be installed, which datapath connects these users. Often the datapath comprises a dedicated, specially developed interface. In particular when a system becomes larger it becomes very complicated to build and control such a system. In the entertainment industry it is for example common to have more then ten different users in such a system, which means that when interaction between each pair of users is foreseen, 45 separate datapaths have to be installed, many of them comprising non-standardised interfaces. It is only by collecting many of these lines in a so-called bus, i.e. a cable comprising a collection of datapaths, that the system can be somewhat clearly arranged. Such a system however remains very complex. Also, disturbances or bugs are very hard to find because each datapath has to be tapped in order to find where the disturbance or bug originates from. Installation of such is a system is a very complex task, requiring highly skilled persons. This makes the use of the known system expensive, especially when it has to be installed for a one- day happening such as a rock-concert.

It is an object of the present invention to mitigate these problems. Therefore a multi media system according to the preamble of claim 1 has been developed, wherein the datapath comprises a backbone path suitable for carrying messages in a standard format, the first user being connected to the backbone via a first module which is capable of translating a message which is in the first format, into a message in the standard format and optionally, is capable of translating a message which is in the standard format into a message in the first format, the second user being connected to the backbone via a second module which is capable of translating a message which is in the standard format into a message in the second format and optionally, is capable of translating a message in the second format into a message in the standard format, the third user being connected to the backbone via a third module which is capable of translating a message which is in the standard format into a message in the third format and/or translating a message which is in the third format into a message in the standard format, wherein the system is programmed to determine whether a message that is carried on the backbone datapath is to be transmitted to one or more of the users.

In the multi media system according to the present invention there is a backbone datapath, e.g. a twisted shielded pair as commonly used in this art, to which each of the users is connected via a module that is capable of translating the format of the corresponding user into the standard format and, if necessary, vice versa. When a message is send from a user A to another user B, the module of the former user "translates" the message, which is obviously in the format of the said user, e.g. MIDI, into the standard format. This can be any format that is suitable for the applied datapath type and might even be the format of one of the users. After translation, the module broadcasts the message over the backbone datapath. The message is then temporarily carried on the backbone and within reach for each of the other users connected to this backbone because there are no specific direct connections anymore. The system is programmed, e.g. in a controller which is part of the system or by means of a hard- and/or software message acceptance filter in one or more of the modules, to actually determine whether or not messages are meant for user B. In this case, since the message was meant for user B, the system will indeed determine that the message has to be transmitted to user B via the module corresponding to user B. Before transmission however, the message which is still in the standard format will be translated in the format of user B, e.g. the MPEG format, by the said module. Upon transmission, B will act according to the message as is commonly known from the prior art system.

The advantages of this system are that by connecting the users to one backbone datapath, there is no longer a need for separate connections between two specific users when an interaction is foreseen between these users. In the new system, from the beginning on each user can interact with all other user since there is a common backbone interconnecting them all. Next to this, there is no longer a need for the development of specific interfaces for each and every interaction that is wished. Simply by installing a module between each user and the backbone, communication between all users is possible because this takes place via a common standard format. This means that in case of 10 users, there need only to be 10 modules instead of 45 dedicated interfaces as was the case in the prior art system. This makes the new system easy to install and clear to organise. Also, disturbances or bugs are very easy to find because all data is transported over the backbone datapath. By simply tapping this datapath, the whole system can be centrally checked.

An important additional advantage of the system is that it is an open system. Once the basic system is installed, interaction between each pair of users is in principle possible (that is after the system is programmed). In the known system however, only the interactions as foreseen at the time of installation are possible. For each additional interaction a new datapath has to be made connecting the corresponding users and often a dedicated interface has to be developed for this datapath which takes a lot of time. Next to this, in the new system it is easy to add an additional user or delete any connected user without disturbing the system. Also, the system can easy adapt to existing users or subsystems (often clients all ready have custom made users) because it does not have to interfere in the events after a message has been transmitted by a module to a user.

As made clear above, the standard format for carrying messages on the backbone datapath can in principle be any format, as long as it is suitable for unambiguous bilateral translation of messages in a plurality of formats, and is suitable for being carried on such a backbone datapath. The CAN format appears to be suitable for a system according to the invention because of the ease of implementation and the fact that CAN is known to be particularly suited for connecting intelligent devices. Also, the CAN format makes it possible to transmit data over lengths even as long as 10 kilometres without loss of information. Regard to the maximum length for a datapath when sending messages in the MIDI format, viz. 5 meters, this is a superb property. Also, the CAN format makes it possible to make the datapath partly or substantially wireless which makes the system according to the invention even more versatile. Examples of the most important standards within the CAN format are CANopen and DeviceNet.

In an embodiment each of the said modules that is capable of translating a message which is in the standard format into the format of the user corresponding to that module, is programmed for determining whether a message, that is carried on the backbone datapath, is to be transmitted to that corresponding user. In this embodiment, the modules which are meant for passing messages from the backbone datapath to the user, are locally programmed for determining whether messages are indeed to be transmitted to the user. This way, modules can be developed that are very easy to implement in a system according to the present invention. By programming the modules, the system itself can be more or less generic.

In another embodiment, a computer is connected to the backbone datapath. This embodiment is advantageous in that the computer can be used for checking the system when in use, e.g. to find bugs, as well as for programming the modules after installation of the system, as well as for controlling the system when in use. This makes the system very easy to handle and control.

In further embodiment, at least one of the said modules is programmed partly in the module itself and partly on the computer. This embodiment is advantageous in that it is easier to make changes in the overall system after it has been installed. As explained, the system is programmed, for example by programming one or more of the modules, to determine whether or not a message carried on the backbone is meant for a certain user, e.g. via a known message acceptance filter. This programming is in this embodiment partly in the said module itself, via hardware, e.g. ASICS, and/or (optionally embedded) software. Part of the programming is done in the computer, possibly via hardware and/or software. This way it is possible to make somewhat more standardised modules, being capable for example in general to determine whether or not a message has to be transmitted to the user e.g. by solely implementing a hardware message acceptance filter. Specific programming, e.g. programming that is dependent on the specific application of the system or specific wishes of the owner of the system, can then be done via the computer. Such programming can be done by loading a program with a specific computer program code into the computer and using this program to instruct the module. It is not necessary to actually program the module locally. It is for example also possible, when a message is broadcast on the backbone datapath, that a code is send to a module telling that the said message is meant for the user corresponding to that module. It can also be done by programming the module itself with a specific program, possibly by using the computer. Such a program code is commonly carried on a datacarrier such as a CD or DVD but can of course be carried by any computer program element, e.g. a datapackage downloadable from the internet. This way an installed system can be programmed by making specific program codes at any location in the world, and then sending the program code via the internet to the computer connected to the backbone of the system. The present invention is also directed to a module, which is programmed for use in a multi media system according to the present invention. Such a module can be directly build in a system according to the invention so that a client can easily extend the capabilities of an already installed system. The programming itself may be directed only to generic hardware and/or software programming, solely suitable for basic communication within the system. All specific programming, e.g. what a connected user should do in response to a specific message send by a specific other user can be done later, possibly via a computer connected to the backbone.

The present invention is also directed to a computer program element comprising a computer program code for programming a module. The system according to the invention requires that the modules are programmed for example to be able and transmit the appropriate messages to the corresponding user and/or vice versa. As explained here above, such programming can be implemented for example partly via software programming. Such programming involves the loading of specific computer program code into the system, e.g. a module itself or a computer connected to the backbone. Computer program code is carried on a computer program element, e.g. a datapackage carried on the internet, or a more tangible element such as a DVD, CD or magnetic type data carrier.

The present invention is also directed to the use of the present multi-media system. By using the system, multimedia interaction can be performed conveniently, relatively cheap, and easy controllable. These are important advantages over the system as known from the prior art. Use of the system is particularly advantageous in the entertainment industry such as in attraction in an amusement park or a fairground, live shows, television quiz shows, theatre shows, rock concerts, applications in modem museums etc. However, the present invention is not limited to such application and can for example also be used in the domestic scenery, automotive applications, teaching applications, or any other form of industry where multiple media which can be perceived by the human, or even animal, organs of sense, have to interact. Applicant notes that the system according to the present invention can also be used to extend a system as known from the prior art, by connecting the new system to the existing system.

This summary of the invention has been provided so that the nature of the invention may be understood easily. A more complete understanding of the invention can be obtained by reference to the following description of an embodiment of the prior art system and an embodiment of the invention, which is illustrated in connection with examples 2 to 4. This embodiment is a mere example of the best mode known to the inventor to perform the present invention at the time the application is filed. It is in no way a limitation of the invention as will be understood by the person skilled in the art.

Fig. 1 gives a schematic overview of an exemplary embodiment of the prior art multi media system.

Fig. 2 gives a schematic overview of an embodiment of the system according to the invention.

Fig. 3 is a schematic represenation of a module for use in the system acoording to the present invention. Fig. 4 is a flow sheet of a determination process in a module according to the present invention.

Figure 1 Figure 1 gives a schematic overview of an exemplary embodiment of the prior art multi media system, which could for example be used in an Alice Cooper rock concert. The keyboard player of the band uses keyboard to make music. Hits of the keyboard are transformed into MIDI-format messages in the keyboard and transmitted via datapath 2 to audio player 3 which is capable of understanding the MIDI-format messages. Each of the messages will usually lead to a specific and corresponding sound coming out of the audio player. The datapath 2 comprises a standard MIDI-interface 4. The keyboard player can also handle video-effects, which are created in video-player 5. Some of the keys of the keyboard 1 are programmed to give messages that are meant for the video-player. These messages are send over datapath 6 to the video-player 5. Because the messages are originally made up in the MIDI-format, they have to be translated into the MPEG format, which is in this case the format for the video-player. For this translation a special interface 7 is developed in placed in the datapath 7. In the same way, the keyboard player can handle light-effects through programmed keys, which transmit messages to the spotlight 8 via datapath 9. Here, special interface 10 is placed in the line to translate messages from the keyboard, which are in the MIDI- format, into messages in the DMX-format. Note that the same spotlight can also be controlled from control panel 11 , via datapath 12, which comprises standard interface 13. Because the audio effects often have to be controlled in concurrence with the sound effects, the control panel 11 can in this case also be used to sent messages to the audio player 3. These messages, which are sent in the DMX-format, are transmitted to the audio-player 3, via datapath 14. Again, a special interface 15 is placed in the datapath to translate messages from the DMX-format into the MIDI-format. In this particular prior art embodiment, the audio-player 3 can also directly communicate with video-player 5. This is done since often audio and video signals relate to each other and should run at the same time. The communication runs via datapath 16 and special MIDI/MPEG interface 17. In a similar way the video-player 5 can directly communicate with spotlight 8 via datapath 18 through special DMX/MPEG interface 19.

The system is controlled by a computer, which is placed centrally, behind the stage, and connected to each user via datapaths 21 to 25. The computer comprises a means 26 for loading software into the computer. The computer is used for monitoring the system, but also for controlling the users if necessary, for trouble shooting but also for programming the system to adapt it to the wishes of the client, in this case the keyboard player.

The system is suited for its task, making communication between users of various kinds possible (7 interactions in total) but requires 7 different interfaces of which 5 are non- standard, and lot of cable connecting the users. Note that direct communication between the audio-player 3 and the spotlight 8, as well as between control panel 11 and the video-player 5 or keyboard 1 is not even possible, despite the extensiveness of the system.

Figure 2

In figure 2 a schematic overview is given for an embodiment of the system according to the present invention. In this system the same users as in the prior art system as illustrated in figure 1 are available. Now each of the users is connected via a corresponding module to a backbone datapath 30, which ends at one end with computer 20, comprising software loading means 26 and an internet connection 31. At the other end the datapath 30 ends with terminator resistance 32. Keyboard 1 is coupled to the datapath via module 33, which is capable of translating messages in the MIDI-format in CANopen and vice versa. Spotlight 8 is coupled to the backbone datapath 30 via module 34, which is capable of translating messages from the DMX format into messages in the CANopen format and vice versa. Control panel 11 is coupled to the backbone datapath 30 via module 35, which is capable of translating messages from the DMX format into the CANopen format. Thus, modules 34 and 35 are the same in basic configuration, for example hardware and embedded software. Video- player 5 is connected to the backbone datapath 30 via module 36, which is capable of translating messages from the MPEG format into the CANopen format. The audio- player 3 is connected to the backbone datapath 30 via module 37 which is, just as module 33, able to translate messages from the MIDI-format into the CANopen format and vice versa. The computer has an internal USB to CANopen interface.

In use, when the keyboard player now hits a certain key, the message, which is in the MIDI-format, is translated by module 33 into a CANopen message. This message is broadcast on datapath 30 for a certain time Δt and is picked up by all modules as well as the computer 20. Each of the modules is programmed to be able and determine whether or not the message is meant for the corresponding user. If not, the message is ignored. If so, the message will be translated from CANopen into the format of the user and transmitted to the user. Then, the user or users that receive the message will act accordingly in a way as known from the prior art. All other interactions are also possible in an analogous way.

This way, each user can communicate with each other user (a total of 10 different interactions) for which only 3 basic interfaces are needed and one single backbone datapath, in this case a twisted shielded pair. Such a system can be orderly arranged, for example by housing some modules together in a case, and the connections between the users are simple and easily overseen and monitored.

Applicant notes that the present invention is not restricted to the users shown but can in principle connect all existing users such as analysers, l/O-switches, actuators, net- managers, WAP-players, MP3-players, WAV-players etc. or even future users that can be used in a multi-media system simply by implementing a corresponding module for each user to translate the specific format in a standard format and vice versa. Figure 3

Figure 3 is a schematic representation of a module for use in the system according to the present invention. In this case module 33 as shown in figure 2, viz. the module via which keyboard 1 is connected to backbone 30, is depicted. In this configuration the hart of the module is formed by micro-controller 200, i.e. an integrated collection of a micro-processor, a RAM- and ROM-memory and an l/O- module. This controller 200 is the mastercontroller of the module and all software routines are performed by this controller. The microcontroller also holds the variables which are used for the operation of the module. CAN-interface 201 is integrated in module 33. This interface, which is connected to the micro-controller by datapath 202, is a hardware subsystem that is designed to make the communication between the backbone, i.e messages which are carried on the backbone, and the microcontroller possible. The microcontroller 200 supervises the CAN interface. The module also comprises a CANopen library 207 which is connected to the microcontroller via datapath 208. Communication in the system is established by use of the CANopen format. The library 207 holds all information which is necessary for using the CANopen protocol. For connection with the user 1 , a device interface 203 is integrated, which interface is connected to the micro-controller 200 via datapath 204. This device interface is the hardware subsystem that matches the electrical signals from the user with those of the micro-controller and vice versa, as is commonly known from the prior art. In this embodiment, the device interface itself also holds a dedicated microcontroller (not shown) for preprocessing of messages going from the user to the microcontroller and vice versa. Micro-controller 200 supervises the device interface. The module in this embodiment also comprises user specific software element 205, which is connected to the micro-controller 200 via datapath 206. The software in this element is used for making the right data-format match, i.e. the right translation of the message. A message received from the backbone, which is in the CAN format, is analysed and mapped into the structure of the MIDI-format, which in this particular case is the format of the user. For this a mapping model is made, as is commonly known from the prior art, and stored in the software element 205.

The module behaves with regard to the backbone as if it was a digital 8-bit Input/Output module according to the well known CANopen Cia401 norm. Cia 401 describes the general procedure for such a digital interface connected to a CANopen system. Cia 401 defines the networkcontrol, downloading of software, managament parameters and an area that can be defined by any person who uses the system. This so-called "user definable area" is in the range from index 2000 upto 5000. For the system according to this embodiment certain user areas are predefined, e.g. Index 2200, subindex 1 is predefined as "Write Command"; subindex 2 is predefined as "Node ID"; subindex 3 is predefined as "CAN-baudrate"; subindex 4 is predefined as "Serial Baud-rate". The module does not contain dipswitches for baudrate setting and node-id setting. These settings are entered by writing the desired values to the corresponding indexes.

If a message is carried on the backbone datapath 30 it is received by the CANopen interface 201 and collected in a buffer (not shown) in the library 207 via micro-controller 200. This buffer is in the CAN format. Then the micro-controller checks whether this message is meant for user 1. If so, the micro-controller transfers the data into another buffer in element 205 which is in the MIDI format. Then this data is ready for transfer to the user via device interface 203. A message from the user to the backbone is firstly stored in the buffer according to the MIDI-format. Then the system transfers the message to the buffer in the CANopen format. After the system has checked that the buffer is complete, the data is transfered to the backbone via interface 201.

Figure 4

In figure 4 a flow sheet is given of a determination process in a module according to the present invention. Firstly, that is when the system is already activated, the module samples the backbone every time a period of time equal to Δt has passed. Thus in step 100, the module waits for Δt. Then in step 101 the module checks whether there is a message carried on the backbone. If not, the module returns to step 100. If so, the module receives the message in stepl 02. Then in step 103 the module determines whether or not the message is meant for the user. The module is able to do this on the basis of its programming. If the message is not meant for the user corresponding to the module then the message is ignored (step 104) and the module returns to step 100. If the message is indeed meant for the user corresponding to this module, then the module translates the message, which is in the standard format, e.g. CANopen, in step 105 into the format of the user, e.g. the MPEG format if the user is video-player. Then, in step 106, the translated message is transmitted to the user. Normally, if the system is running without any problems, the user will act in correspondence to the message in a way as known from the prior art. The system according to this embodiment checks in step 107 if the user indeed does so. If not, then in step 108 a fault message is sent to the computer that is connected to the backbone. The controller who controls the system via the computer can then handle the fault. If the user however acts the way he should, the module ends the sequence in step 109 and returns to step 100.

Claims

1. Multi-media system comprising at least three users for handling objects such as text, sound, images, video clips, light, motion etc., the system being suitable for interaction between the users, wherein

- the first user communicates in a first format,

- the second user communicates in a second format different from the first,

- the third user communicates in a third format being the same or different than the first or second format,

wherein interaction between two of the said users takes place via a message send from one user to the other over a datapath connecting these users, wherein the datapath comprises an interface,

characterised in that

- the datapath comprises a backbone datapath suitable for carrying messages in a standard format, - the first user being connected to the backbone datapath via a first module which is capable of translating a message which is in the first format, into a message in the standard format and optionally, is capable of translating a message which is in the standard format into a message in the first format,

- the second user being connected to the backbone datapath via a second module which is capable of translating a message which is in the standard format into a message in the second format and optionally, is capable of translating a message in the second format into a message in the standard format,

- the third user being connected to the backbone datapath via a third module which is capable of translating a message which is in the standard format into a message in the third format and/or translating a message which is in the third format into a message in the standard format,

wherein the system is programmed to determine whether a message that is carried on the backbone datapath is to be transmitted to one or more of the users.

2. A system according to claim 1 , wherein each of the said modules that is capable of translating a message which is in the standard format into the format of the user corresponding to that module, is programmed to determine whether a message that is carried on the backbone datapath, is to be transmitted to the said user.

3. System according to any of the preceding claims, wherein a computer is connected to the backbone datapath.

4. A system according to claim 3, wherein at least one of the said modules is programmed partly in the module itself and partly on the computer.

5. Module programmed for use in a system according to any of the preceding claims.

6. Computer program element comprising a computer program code for programming a module for use in the system according to any of the claims 1 to 4.

7. Use of the system according to any of the claims 1 to 4.