GB2386299A

GB2386299A - A method to classify and structure a multimedia message wherein each portion of the message may be independently edited

Info

Publication number: GB2386299A
Application number: GB0205564A
Authority: GB
Inventors: Paola Marcella Hobson
Original assignee: Motorola Inc
Current assignee: Motorola Solutions Inc
Priority date: 2002-03-09
Filing date: 2002-03-09
Publication date: 2003-09-10
Anticipated expiration: 2022-03-09
Also published as: GB0205564D0; WO2003077520A1; GB2386299B; AU2003206953A1

Abstract

A method (200) describes a construction of a structured multimedia message (100). The method includes the steps of: generating two or more of the following: an introduction portion (210) of the multimedia message; at least one message portion (235) of the multimedia message; an ending portion (265) of the multimedia message; and combining any two or more of said portions to form a structured multimedia message wherein, each portion may be independently edited. A structured multimedia message, multimedia communication unit multimedia communication system and database record are also described. This provides network operators and consumers the benefits of easily creating and editing multimedia messages.

Description

A METHOD TO CLASSIFY AND STRUCTURE A MULTIMEDIA MESSAGE Field of the Invention This invention relates to the provision of multimedia messages. The invention is applicable to, but not limited to, a method for classifying and structuring elements in a multimedia message to be used in a mobile communication environment.

Background of the Invention In the field of this invention, it is known that future multimedia messaging systems (often referred to in public domain literature as MMS) assume that users will send and receive multimedia messages from a handheld terminal such as a portable cellular phone. However, construction of a multimedia message on such a small device is problematic for a number of reasons, such as: (i) Time constraints: typical users are not anticipated to have long periods of time available to edit and re-edit a message before it is sent; (ii) Limited device capability: for the foreseeable future, portable cellular phones will continue to be limited by battery power, memory, screen resolution, processing power, and display size ; and

(iii) Control ergonomics: small devices will have limited space for keypads and user interfaces, leading to increasingly small keys and where users will experience difficulties attempting complex editing functions with pen/tablet input devices.

An example would be a user wishing to send a multimedia message. Their mobile terminal allows them to acquire some audio-visual material, for example the user directs the terminal's camera at himself/herself as they are talking, then the user pans the camera round to show the recipient what he/she can see. On reviewing the message, the user discovers that the camera pan is very blurred, that they have said"hello Fred"instead of"hello Cyril", and that the inappropriate phrase they used when they dropped the terminal on their foot has been saved as part of the message, along with 5 seconds of video of the floor. Re-recording the message could lead to a set of similar problems.

Alternatively, editing the message is likely to be awkward because of the problems described above. In addition, multimedia editing is complex even with current personal computer (PC) based tools, and many prospective users will be unwilling to learn complex editing procedures.

It is envisaged that such problems could severely limit the take-up of multimedia messaging services. As an example, most video editing means require multiple views

of a video sequence to be presented to a user, referenced by time-code. Most editing requires users to cut, splice, re-join, etc. frames of a video sequence, with an implicit assumption that the user can visualise what the finished result will be when they make an editing decision. Editing is also time consuming and it is very difficult to seamlessly replace small segments of video and associated audio.

An M-Services concept [GSM Association PRD AA. 35 MServices Guidelines ver 3.0. 0. 31st May 2001] has been developed that allows for'templates'to be created for composing multimedia messages. However, this does not specify what these templates should be. The M-Services guidelines also mandate that a"save"option should be provided when the multimedia message has been composed.

Problematically, the M-Services guidelines do not help the user who wishes to rapidly compose, review, edit and send a general multimedia message containing video, audio, text, still images, graphics, animations and annotations. In addition, the M-Services guidelines allow for predictive text editing, but do not extend this concept to video or audio. M-Services recommendation of a'save'option does not allow for a user's need to readily edit the message, which is difficult to do if it has been recorded as one long segment.

Automatic video classification techniques are known for broadcast and entertainment material, but so far they have not been considered for automatically sorting and classifying multimedia message content. One example is described in"Video Keyframe Production by Efficient Clustering of Compressed Chromaticity Signatures"by Mark S. Drew and James Au, Proceedings of the ACM, 2000.

Known automatic video classification techniques focus almost exclusively on professionally generated and edited multimedia material for entertainment and business applications. They do not take into account the fact that material for multimedia messaging is generally acquired by amateurs and can be of specific and limited format, for example head and shoulders of a speaker, pan of a scenic view, etc. Material acquired by amateur users for multimedia messaging will tend to be noisy (video and audio noise), of low contrast, and will not use the artistic video material generation techniques that provide the characteristic points needed for automated classification of broadcast footage.

A Synchronised Multimedia Integration Language (SMIL) has been defined for multimedia authoring. SMIL provides a mark-up language that allows multimedia objects to be composed in a presentation, if rendering software, which is compatible with SMIL, is available.

Although SMIL is a detailed and flexible markup language for composition of multimedia content such as message

generation, it is at a level of complexity beyond most mobile communication users. SMIL would most likely be automatically generated by software on the handheld device after the multimedia message has been edited by the average user, rather than the user creating SMIL documents directly.

Thus, there exists a need in the field of the present invention to provide an improved means and method to classify and structure a multimedia message, wherein the abovementioned disadvantages associated with prior art arrangements may be alleviated.

Statement of Invention In accordance with a first aspect of the present invention there is provided a method to construct a structured multimedia message, as claimed in claim 1.

In accordance with a second aspect of the present invention there is provided a structured multimedia message, as claimed in claim 10.

In accordance with a third aspect of the present invention there is provided a database, as claimed in claim 12.

In accordance with a fourth aspect of the present invention, there is provided a database, as claimed in claim 13.

In accordance with a fifth aspect of the present invention, there is provided a multimedia communication unit, as claimed in claim 14.

In accordance with a sixth aspect of the present invention, there is provided a multimedia communication unit, as claimed in claim 15.

In accordance with a seventh aspect of the present invention, there is provided a multimedia communication system, as claimed in claim 16.

In accordance with an eighth aspect of the present invention, there is provided a multimedia communication system, as claimed in claim 17.

In accordance with a ninth aspect of the present invention, there is provided a database, as claimed in claim 18.

Further aspects of the present invention are as claimed in the dependent claims.

In summary, the inventive concepts of the present invention provide for automated classification by the mobile/portable handset of the types of multimedia message segment, and editing a multimedia message using a

menu-driven interface, in a mobile communication environment.

The first element of the preferred embodiment of the present invention provides the structure of the multimedia message, where the structure description provides the framework for the remaining parts of the preferred embodiment.

The second element of this invention provides for a means of creating, reviewing and editing multimedia messages based on the previously defined structure. This sets the context for definition of the segments of a multimedia message.

Optionally, the third element of this invention provides means for keeping track of the structure of the multimedia message during its generation. Such a tracking process, using say a file or database record, may show each of the elements of the message, any important details relating to the segments (e. g. time of creation, duration, format, medium, etc), and their order of presentation with presentation times.

Advantageously, the fourth element provides means for automating the multimedia message structuring and classification steps, and specifically applying this in a wireless communication environment.

Brief Description of the Drawings Exemplary embodiments of the present invention will now be described, with reference to the accompanying drawings, in which: FIG. 1 shows a multimedia message structure, according to a preferred embodiment of the present invention.

FIG. 2 shows a message construction flowchart, according to a preferred embodiment of the present invention.

FIG. 3 shows a multimedia message structure record, according to a preferred embodiment of the present invention.

Description of Preferred Embodiments The inventive concepts of the present invention provide means for automatically (and/or manually) classifying portions or segments of a multimedia message such that the portions or segments can be stored as a defined structure for ease of review and editing. The invention has additional benefits in reducing complexity when composing a multimedia message, and when reviewing received multimedia messages.

In particular, the present invention provides at least the following features over known means and methods:

(i) Application of the structuring concept to a multimedia message; (ii) Construction of a multimedia message using the defined structure; (iii) Management of the multimedia message via a record of its structure; and (iv) Automatic analysis of multimedia messages to define and classify segments according to the structure.

In the context of the preferred embodiments of the present invention, a multimedia message may be composed from any combination of a number of elements such as video, audio, text, graphics, animations, annotations, still images etc. Furthermore, such a multimedia message may be formed in one of the following ways: (i) Formed at a time that the message is composed, using manual means, or (ii) Formed at a time that the message is composed, using automated means, or (iii) Formed by retrieving one or more portions from a storage element on a user's device, or (iv) Formed by retrieving one or more portions from a store accessible by a user's device, or (v) Formed by automated or manual analysis of a received (incoming) message, or (vi) Formed by automated or manual analysis of a previously stored multimedia message.

The inventor of the present invention envisages a number of activities that a user may engage in when generating a

multimedia message. Such activities include at least one or more of the following: (i) Video/audio acquisition of himself/herself; (ii) Video/audio acquisition of views of objects near to the user and distant from the user; (iii) Video/audio acquisition of other people; (iv) Providing a video/audio reply to a received video/audio/textual message from another user; and (v) Providing video/audio/textual details of how a response may be sent back to them.

Multimedia message structure: Referring first to FIG. 1, a structure 100 of a multimedia message is shown, in accordance with a preferred embodiment of the present invention. A multimedia message consists of a greeting or introduction 110, one or more main body elements 120,130, 140,150, and an ending 160. It is within the contemplation of the invention that certain message parts may be omitted, but the message must contain at least two of the elements (an introduction, at least one message body and an ending) shown in FIG. 1. For example, a valid message may consist of a greeting and ending only. Alternatively, it is envisaged that a valid message may consist of two message body elements only, without any greeting or ending. A skilled artisan, utilising the inventive

concepts described herein would recognise that many more message configurations are possible.

The benefits of such a multimedia message structure include at least the following: (i) Message elements can be readily edited without needing to edit the complete message ; (ii) Message body sections can be added and deleted very easily by even a novice user; and (iii) A familiar structure is provided for users who are used to other forms of messaging, for example email and letter writing.

Composition and editing of a multimedia message: The second aspect of the preferred embodiment of this invention provides for a means of creating, reviewing and editing multimedia messages, based on the structure shown in FIG. 1. A flowchart illustrating these stages relating to multimedia message composition is shown in FIG. 2. It is within the contemplation of the invention that such a flowchart applies equally to reviewing and editing operations.

Referring now to FIG. 2, a proposed structure 200 for a user to follow in composing, reviewing, and editing a

multimedia message, is shown. Note that in the preferred embodiment of the present invention, at least one of the steps of greeting, message body, or ending should be performed in order for a valid multimedia message to exist.

The process starts in step 205, with the greeting section, in step 210. In the first instance, the type of greeting may be selected, or generated, as shown in step 215. This may be a new greeting recorded at the time the message is composed or edited, as in step 220, or may be a pre-stored greeting (any combination of audio, video, text, etc), or may be a synthesised greeting created by audio, video, textual, or other predictive software, as shown in step 230. It is within the contemplation of the invention that newly recorded greetings may also be labelled and stored for subsequent retrieval, as shown in step 225. Optionally, the greeting may be omitted (215).

If present, once a greeting has been generated, the greeting may be retrieved or synthesised. The user is then given the optional to provide some message content, in step 235. First, a type of message body content is selected or generated in step 240. Such a message body may be new content recorded at the time the message is composed or edited, as in step 245. Alternatively, such a message body may be pre-stored content (any combination of audio, video, text, still images, animations etc), or may be synthesised content created by audio, video, textual or other predictive software, as shown in step 255. Furthermore, it is envisaged that the message body

content may be acquired from a received message, or the content acquired from another source (e. g. remote database or another terminal) at the time of message composition (not shown).

It is again within the contemplation of the invention that newly recorded message segments may also be labelled and stored for subsequent retrieval, as shown in step 250.

Preferably, the user may repeat the message body content generation, retrieval or synthesis step as many times as they desire, accumulating more message segments as shown in step 260. These segments may be as large or small as the user wishes, depending on their confidence in any subsequent editing process. Small segments can also be readily discarded or substituted at the editing stage.

Small segments also have the advantage that they can be rearranged easily, in temporal or spatial order, when editing the message. In contrast, larger numbers of segments implies that the user will be able to apply and manage meaningful labels such that the segments can be recalled and processed during review and editing.

Once the user has completed the generation of all message body segments, they may optionally provide an ending to the multimedia message, as shown in step 265. A first step 270 decides on the type of ending to the multimedia message to be created. This may be a new ending recorded at the time the message is composed or edited, as in step 275, or may be a pre-stored ending (any combination of

audio, video, text, still image, animation etc), or may be a synthesised ending created by audio, video, textual, or other predictive software, as in step 285. It is envisaged that messaging services operators may provide standard greetings and endings for ease of use of multimedia messaging editors.

It is again within the contemplation of the invention that newly recorded endings may also be labelled and stored for subsequent retrieval, as shown in step 280.

The multimedia message construction process is then complete, as shown in step 290.

Managing the structure of a multimedia message: The third element of the preferred embodiment of the present invention provides a means for keeping track of the structure of the multimedia message during its generation.

The multimedia message structure is preferably contained within a file or database record 300, which preferably shows each of the elements of the message 320,330, 340, 350, any important details relating to the segments, for example their time of creation, duration, format, medium, etc. and their order of presentation with presentation times. The database (300) record may also store references to structured multimedia message portions.

An example of such a file or record is shown in FIG. 3.

The multimedia message structure record may also contain the greeting segment 310 and the ending segment 360, if these are present.

When the user reviews the multimedia message, the record 300 acts as a"play list", such that stored elements can be retrieved and composed in the desired order. Editing the message is now simplified as none of the actual multimedia segments need be manipulated-just the play list. This has enormous benefits in saving memory space when creating composite effects in accordance with the preferred embodiment, for example joining two video clips in a conventional editor has the effect of doubling the memory usage.

The file or record also beneficially allows text-based editing. An experienced user can then manipulate elements and segments of the multimedia message within the record without having to review any of the actual message content. For example, a segment may be deleted by removing its reference from the record. The actual segment remains stored on the device and may be re-used in another message until it is actually deleted from the device disk by the user.

In a fourth aspect of the preferred embodiment of the present invention, means are provided for automating the steps of FIG. 2, such that the user does not have to manually move from one section or step to the next, if

they prefer to record a multimedia message in one single recording step.

The inventor of the present invention envisages that experienced users would prefer to create their own segments of the multimedia message. In this case, editing software would provide means for a user to identify when they wanted to move from one segment of the multimedia message to the next. Such means could include, but are not limited to, button presses, audio silence, touch screen or tablet interaction, audio keywords, null or blank video input, etc.

However, many novice users may prefer to record a message in an unstructured way, and/or may wish to record the message as one single segment, preferring the editing software to automatically assign the multimedia message structure. This type of automation has additional benefits in managing received messages that may not have a predefined structure.

The automated classification method analyses the multimedia message for structuring elements. As an example, the first audio parts of a message may be analysed for detection of key greetings words (in the user's chosen language), or the first image segments may be analysed for detection of a facial image (including the device owner's facial image) or company logo.

In the preferred embodiment of the present invention, message body segment boundaries are applied depending on

audio and video cues that include, but are not limited to, the following: (i) Change of medium e. g. from video to animation showing that the user inserted something, or change from music to speech, which can be detected using statistical methods; (ii) Change of video camera view which can readily be detected using frame differencing techniques ; (iii) Change of speaker, as detected by speaker recognition methods; (iv) Silence detection, indicating that the user is changing to a new topic ; (v) Change of subject (e. g. facial recognition detects a different person); and (vi) Addition of new material e. g. extra voice, extra face, some annotation, text overlay etc Finally, the latter parts of a message may be analysed for detection of key ending words (in the user's chosen language), or reappearance of a certain facial image (including the device owner's facial image) or company logo.

Preferably, a multimedia message record is generated during the automated analysis. The user may then review the message segment by segment. This has benefits for received messages where perhaps the user only has time to replay the greeting and ending. However, the main benefit is when the message creator wishes to edit their message. An easy to use structure is now available to them such that they can add to the message, delete or substitute segments, change the order of message elements etc.

It will be understood that the means and method to automatically classify and structure a multimedia message described above provides the following advantages: (i) Provides ease of use in multimedia messaging systems, which will have significant competitive advantage in consumer handset markets.

(ii) Network operators as well as consumers benefit as messages are easy to create and edit, and therefore users will likely send more of them, thereby paying for more airtime. Text messaging has become very popular with mobile phone users, and operators would like to repeat this success with multimedia messaging.

(iii) Handset manufacturers benefit by having handsets that are well adapted to the new multimedia messaging services, which will be offered soon by many mobile operators.

Thus, a method to construct a structured multimedia message has been described. The method includes the steps of: generating two or more of the following: an introduction portion of the multimedia message; at least one message portion (235) of the multimedia message; an ending portion of the multimedia message; and combining any two or more of said portions to form the structured multimedia message.

Furthermore, a structured multimedia message has been described having any two or more of the following portions: an introduction portion (110) of a multimedia message; said introduction portion (110) followed by: at least one message portion (120,130, 140,150) of the structured multimedia message (100), followed by: an ending portion of the structured multimedia message.

Whilst specific, and preferred, implementations of the present invention are described above, it is clear that one skilled in the art could readily apply variations and modifications of such inventive concepts.

Thus, an improved means and method to automatically classify and structure a multimedia message has been described, wherein the aforementioned disadvantages associated with prior art arrangements have been substantially alleviated.

Claims

Claims 1. A method (200) to construct a structured multimedia message (100), the method characterised by the steps of: generating two or more of the following: an introduction portion (210) of the multimedia message; at least one message portion (235) of the multimedia message; an ending portion (265) of the multimedia message ; and combining any two or more of said portions to form a structured multimedia message.
2. The method (200) to construct a structured multimedia message (100) according to Claim 1, wherein the step of combining any two or more of said portions only combines two or more message body portions to form said multimedia message.
3. The method to construct a structured multimedia message according to Claim 1 or Claim 2, wherein the step of combining includes combining the portions in such a manner as to allow, where present, said introduction portion, said message portion and said ending portion to be independently edited.
4. The method to construct a structured multimedia message according to any preceding Claim, wherein the

<Desc/Clms Page number 21>

structured multimedia message is formed in one of the following ways: (i) formed at a time that the message is composed, using manual means, or (ii) formed at a time that the message is composed, using automated means, or (iii) formed by retrieving one or more portions from a storage element on a user's device, or (iv) formed by retrieving one or more portions from a store accessible by a user's device, or (v) formed by automated or manual analysis of a received (incoming) message, or (vi) formed by automated or manual analysis of a previously stored multimedia message.
5. The method to construct a structured multimedia message according to any preceding Claim, the method further characterised by the step of: storing (225,250, 280) one or more portions from any of said introduction portion, said message portion (s) and said ending portion in a storage element on a user's device, or in a store accessible by a user's device.
6. The method to construct a structured multimedia message according to Claim 5, the method further characterised by the step of: classifying (300) said stored one or more portions by type of multimedia message segment.

<Desc/Clms Page number 22>
7. The method to construct a structured multimedia message according to Claim 5 or Claim 6, the method further characterised by the step of: storing information relating to respective one or more portions wherein said information includes at least one of the following: time of creation, duration of portion, format of portion, medium of multimedia message, order of presentation, presentation time, file name, medium of portion, owner of portion, source of portion.
8. The method to construct a structured multimedia message according to any preceding Claim, the method further characterised by the step of: editing said structured multimedia message, for example, using a menu-driven interface, to form a structured multimedia message for transmittal in a mobile communication environment.
9. The method to construct a structured multimedia message according to any preceding Claim, the method further characterised by the steps of: partitioning a received multimedia message into a number of independent portions; and storing said partitioned independent portions for subsequent re-use in generating a structured multimedia message.
10. A structured multimedia message (100) characterised by any two or more of the following portions:

<Desc/Clms Page number 23>

an introduction portion (110) of a multimedia message; said introduction portion (110), followed by: at least one message portion (120,130, 140,150) of the structured multimedia message (100), followed by: an ending portion of the structured multimedia message.
11. The structured multimedia message (100) according to claim 10, wherein the structured multimedia message is formed of only message body portions.
12. A database (300) adapted to store structured multimedia message portions according to Claim 10 or Claim 11.
13. A database (300) adapted to store references to structured multimedia message portions according to Claim 10 or Claim 11.
14. A multimedia communication unit adapted to perform any of the steps to construct a multimedia message, according to any of Claims 1 to 9.
15. A multimedia communication unit adapted to store a structured multimedia message portion according to Claim 10 or Claim 11.
16. A multimedia communication system adapted to facilitate the performance of any of the steps to construct a multimedia message, according to any of Claims 1 to 9.

<Desc/Clms Page number 24>
17. A multimedia communication system adapted to facilitate storing of a structured multimedia message portion according to Claim 10 or Claim 11.
18. A database (300) adapted to structured multimedia messages constructed according to any of method Claims 1 to 9.
19. A structured multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 1 of the accompanying drawings.
20. A method to classify and/or structure a multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 2 of the accompanying drawings.
21. A database record of a structured multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 3 of the accompanying drawings.

21. A database record of a structured multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 3 of the accompanying drawings.

<Desc/Clms Page number 25>

Amendments to the claims have been filed as follows Claims 1. A method (200) to construct a structured multimedia message (100,300) comprising any or all of video, audio, text, still images, graphics, animations and annotations, the method characterised by the steps of: generating two or more of the following: an introduction portion (110, 210,310) of the multimedia message ; at least one message portion (130,235, 320,330, 340,350) of the multimedia message ; an ending portion (160,265, 360) of the multimedia message ; and combining any two or more of said portions to form a structured multimedia message.

2. The method (200) to construct a structured multimedia message (100) according to Claim 1, wherein the step of combining any two or more of said portions only combines two or more message body portions to form said multimedia message.

3. The method to construct a structured multimedia

message according to Claim 1 or Claim 2, wherein the step of combining includes combining the portions in such a manner as to allow, where present, said introduction portion, said message portion and said ending portion to be independently edited.

4. The method to construct a structured multimedia message according to any preceding Claim, wherein the

<Desc/Clms Page number 26>

storing information relating to respective one or more portions wherein said information includes at least one of the following : time of creation, duration of portion, format of portion, medium of multimedia message, order of presentation, presentation time, file name, medium of portion, owner of portion, source of portion.

8. The method to construct a structured multimedia message according to any preceding Claim, the method further characterised by the step of: editing said structured multimedia message, for example, using a menu-driven interface, to form a structured multimedia message for transmittal in a mobile communication environment.

9. The method to construct a structured multimedia message according to any preceding Claim, the method further characterised by the steps of : partitioning a received multimedia message into a number of independent portions; and storing said partitioned independent portions for subsequent re-use in generating a structured multimedia message.

10. A structured multimedia message (100, 300) comprising any or all of video, audio, text, still images, graphics, animations and annotations characterised by any two or more of the following portions: an introduction portion (110, 310) of a multimedia message; said introduction portion (110, 310), followed by:

<Desc/Clms Page number 27>

at least one message portion (130,120, 320,130, 330,140, 340,150, 350) of the structured multimedia

message (100, 300), followed by : , an ending portion (150, 360) of the structured multimedia message.

11. The structured multimedia message (100) according to claim 10, wherein the structured multimedia message is formed of only message body portions.

12. A database (300) adapted to store structured multimedia message portions according to Claim 10 or Claim 11.

13. A database (300) adapted to store references to structured multimedia message portions according to Claim 10 or Claim 11.

14. A multimedia communication unit adapted to perform any of the steps to construct a multimedia message, according to any of Claims 1 to 9.

15. A multimedia communication unit adapted to store a structured multimedia message portion according to Claim 10 or Claim 11.

v 16. A multimedia communication system adapted to facilitate the performance of any of the steps to construct a multimedia message, according to any of Claims 1 to 9.

<Desc/Clms Page number 28>

17. A multimedia communication system adapted to facilitate storing of a structured multimedia message portion according to Claim 10 or Claim 11.

18. A database (300) adapted to structured multimedia messages constructed according to any of method Claims 1 to 9.

19. A structured multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 1 of the accompanying drawings.

20. A method to classify and/or structure a multimedia message substantially as hereinbefore described with reference to, and/or as illustrated by, FIG. 2 of the accompanying drawings.