WO2000068759A2 - Systeme et procede servant a effectuer l'indexation, l'acces et l'extraction de donnees audio/video presentant une activite simultanee d'execution de croquis - Google Patents

Systeme et procede servant a effectuer l'indexation, l'acces et l'extraction de donnees audio/video presentant une activite simultanee d'execution de croquis Download PDF

Info

Publication number
WO2000068759A2
WO2000068759A2 PCT/US2000/012833 US0012833W WO0068759A2 WO 2000068759 A2 WO2000068759 A2 WO 2000068759A2 US 0012833 W US0012833 W US 0012833W WO 0068759 A2 WO0068759 A2 WO 0068759A2
Authority
WO
WIPO (PCT)
Prior art keywords
audio
information
user units
sketch
initiation
Prior art date
Application number
PCT/US2000/012833
Other languages
English (en)
Other versions
WO2000068759A3 (fr
Inventor
Samuel Yen
Renate Fruchter
Larry Leifer
Original Assignee
The Board Of Trustees Of The Leland Stanford Junior University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Board Of Trustees Of The Leland Stanford Junior University filed Critical The Board Of Trustees Of The Leland Stanford Junior University
Priority to AU48367/00A priority Critical patent/AU4836700A/en
Publication of WO2000068759A2 publication Critical patent/WO2000068759A2/fr
Publication of WO2000068759A3 publication Critical patent/WO2000068759A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality

Definitions

  • the invention relates to the field of communication methods.
  • the invention relates to software for identifying sketch entities from sketch activity and for correlating media information to these sketch entities.
  • Short-term communication between two or more distant people is typically performed on the audio level.
  • a variety of telephone systems provide the proper tools for that type of communication.
  • communication solely on the audio level becomes often unsatisfactory.
  • Visual information in the form of graphics, pictures, sketches and the like are used to aid the information exchange.
  • RTMMCD Real time multi media communication devices
  • the buffering of the information is typically accomplished by independently saving audio information and/or video information. This buffering is accomplished temporally and/or permanently, at the location where the information is created and/or at a remote location. In a following step, the correlated information are transmitted chronologically with certain user definable parameter.
  • U.S. Pat. No. 4,656,654 to Dumas discloses a computer-assisted graphic teleconferencing method and apparatus that is designed for use with the PSTN.
  • the method and apparatus described in the patent work according to the principles described in the paragraph above.
  • the main disadvantage of this invention is that graphics and voice can be communicated only alternatingly.
  • a simultaneous distribution of a sketching activity with the contemporaneous explanatory verbal information is not possible with this invention.
  • the invention is not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
  • U.S. Pat. No. 5,801,757 to Saulsbury discloses an interactive communication device that allows simultaneous sending and receiving of audio and graphic information via a PSTN.
  • the device uses techniques for compression, merging and coding of signals to accomplish the transmission.
  • the patented device further uses techniques for decompressing, separating and decoding of signals to recreate the audio and graphic signals in their original form at the location of a receiver.
  • the patented device is placed between the telephone line and the PC.
  • the device provides a possibility for simultaneous exchange of audio and graphical information.
  • the main shortcoming of the device is that it needs to be physically installed in combination with a software program, which may result in problems of compatibility with existing hardware. Furthermore, it is not possible to communicate audio-graphically with a person that is not in possession of the device.
  • the invention is also not usable in combination with the Internet since no distribution system is described that may be implemented in a web page .
  • U.S. Pat. No. 5,832,065 to Bannister et al. discloses a synchronous voice/data message system that allows the exchange of audio-graphic messages between specific portable communication devices also via a PSTN.
  • the message system provides a replay function to display the creation process of the graphical information.
  • the message system simultaneously replays the correlated verbal information.
  • the chronological audio graphic information can be replayed at varying speeds.
  • the message system is one directional and chronological. It does not afford a recipient the option to selectively access segments of the chronologically retrieved message. It is not possible to communicate audio- graphically with a person that is not in possession of the portable communication device. Further, the invention is not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
  • US. Pat. No. 5,915,003 to Bremer et al. discloses a sketching unit for transmission of sketches and notes over normal telephone lines.
  • the teaching of the patent is similar to that of Saulsbury. It utilizes in addition a specific sketching unit that allows creating and/or displaying graphic information.
  • the patent further discloses a technique for a multiplexed transmission via a device that is switched between the telephone line and a computer. It is not possible to communicate audio- graphically with a person that is not in possession of the device.
  • the invention is also not usable in combination with the Internet since no distribution system is described that may be implemented in a web page.
  • a communication medium that is gaining more and more significance is the Internet.
  • a number of software products and web pages provide users possibilities to exchange audio and/or graphical information for the purpose of real time collaboration.
  • the RealityWave Inc. discloses on their web page ww . realitywave . com a software product called VizStream that allows to create 3D graphics that can be embedded within a web page and accessed by the client. Even though the software provides an enhanced display technique, it limits the client to view a prepared information. Bi-directional information exchange on the basis of a common document is not possible with that technique. Further, Vizstream provides only the display of 3D models without any additional medial information like for instance audio, video or graphics.
  • eDrawing a software program called "eDrawing" is presented, which allows to generate self extracting files that can be attached to emails.
  • the self extracting files unfold into an interactive screen where 2D mechanical drawings can be viewed together with remarks and any other text or graphical information necessary to make the drawing understandable.
  • eDrawing is also one-directional, which means that the client cannot add on his side to the contents of the information. Further, eDrawing provides no possibility to add verbal information to the drawing.
  • the present invention introduces a software program that allows clients to exchange graphical information together with correlated multi medial information.
  • Correlated multi medial information is primary verbal information and secondary video information.
  • the software program provides the exchange in a quasi simultaneous mode. Since real time information exchange is influenced by the transmission capacity of the communication infrastructure the software program provides a script log for each client and project. In the script log all events during the creation of a graphical and multi medial document are temporally correlabel. Further, the software program recognizes free created graphical entities by capturing the activities of input devices.
  • An input device is, for instance, a mouse, a digitizer tablet or a pointer of a touch screen.
  • the creation of a graphical entity begins typically with an initiation event performed by the user. This initiation event is performed with the down click of a mouse button or by bringing a pointer into contact with a touch screen.
  • the creation of a graphical entity ends typically with an termination event performed by the user. This termination event is performed, for instance, with the release of the down held mouse button.
  • the period between the initiation event and the termination event define the temporal boundary condition to combine a number of drawn line segments into a sketch entity.
  • This definition system is applied in a basic and an advanced form with the result of sketch entities with varying complexities .
  • a video input device as for instance a video camera may capture in addition visual information correlated to the graphical information.
  • the visual information is primarily provided by the user and may, for instance, be the facial expressions and gestures of the user or any other visual information correlated to the creation of the graphical information.
  • An audio input device as, for instance, a microphone captures audio information correlated to the graphical information.
  • the audio information is primarily provided by the user in the form of verbal information.
  • Graphical, visual and audio information are time stamped, captured and stored.
  • the storing is performed in the form of a dynamic script log on a direct-access storing medium like, for instance, a disk drive or the read active memory (RAM) of the users computer.
  • verbal information is not necessarily synchronous with the period of each correlated initiation action, the invention recognizes bulks of audio information and correlates them to the corresponding sketch entities.
  • the Internet allows each individual user to retrieve and transmit information independent of the capacity of the communication infrastructure.
  • the transmission capacity of the communication infrastructure solely influences the waiting time to send and/or retrieve the information.
  • the present invention provides a buffered transmission mode, during which the created script log is transmitted to a central server and eventually broadcasted in a quasi real time mode that corresponds to the transmission capacity of the communication infrastructure.
  • the Internet also allows streaming information transmission during which the information is presented as it is received and/or created. Streaming transmission is utilized for instance for so-called chat rooms or streaming video. With increasing transmission capacity of the communication infrastructure, on which the Internet is based, streaming data transmission via the Internet becomes increasingly relevant.
  • the present invention provides a streaming transmission mode, during which data is distributed between the number of participants as it is created.
  • the preferred system architecture of the present invention consists of one or more main server stations that can be accessed by the clients via a web page. Such a web page operates as a broadcasting site that receives and redistributes all information from the individual clients and/or participants.
  • the web page provides an interactive graphical interface, in which the clients can replay, view, edit and/or create sketch information.
  • the creation process of a document can be replayed on the interactive graphical interface in a real time mode and/or in a temporally altered mode. Correlated audio and/or video information is replayed simultaneously.
  • individual sketch entities can be selected and the correlated audio and/or video information is replayed. Since sketch entities do not necessarily have media information associated with them, the invention provides an optional highlight mode.
  • the highlight mode allows the reviewing client to visually recognize additional media information correlated to individual sketch entities.
  • the client can add sketch information to a retrieved document.
  • the client can record audio and/or video information to contribute to collaborative creation of a document.
  • the invention provides a selectable graphical vocabulary like, for instance, line fonts or colors that can be assigned to individual clients. As a result, each contribution can be correlated to its creator.
  • the invention provides the possibility to either broadcast the collaborative editing in a quasi real time mode respectively a streamed real time mode and/or an off-time mode. During the off-time mode, individual participants may contribute at any time to the creation of the document.
  • the invention provides thereby an information system that informs other participants about an update of a document under collaboration.
  • the interactive graphical interface a background display mode, during which graphical and/or pictographic images may be displayed.
  • clients are able to incorporate previously created documents like, for instance, blueprints, photographs, maps, snapshots and/or video frames.
  • a client may be provided with a software program of the present invention in the form of a self- extracting email message, and/or an installable program downloaded from a web page.
  • the installable program may also be retrieved from a storage medium like, for instance, a Floppy Disk or a Compact Disk.
  • the client is able to perform all operations of the present invention on his/her own computer without being connected to the Internet.
  • each client occasionally exchanges information either with a server station or directly with other clients to exchange all updates.
  • the present invention may further be part of an operating system that operates a computer and/or a communication device like, for instance, a cellular phone.
  • the operating system may include the operation of a communication network.
  • the system architecture may be centralistic and/or equalized.
  • a central server stores centrally the creations and activities of each individual client in a central log.
  • each client stores the creations and activities of his/her own and other clients in a personal log.
  • the clients personal log is updated during an update call to a central server performed during an update ring call to other clients. Update calls and update ring calls may be triggered by the client or automatically dependent on an available transmission capacity, or other definable parameters.
  • the invention and in particular the alternate embodiment may be applied to any communication system and particularly to a wireless communication system with inconsistent transmission capacities and arbitrary interruptions of connections.
  • Fig. 1 shows an example of a basic sketch entity with a single initiation event and a single termination event.
  • Fig. 2 shows an example of an advanced sketch entity with multiple initiation events and multiple termination events.
  • Fig. 3 shows an exemplary graph of a basic procedure to capture sketching activities and correlated media information.
  • Fig. 4 shows an exemplary graph of an advanced procedure to capture sketching activities and correlated media information.
  • Fig. 5 shows a simplified example of a interactive graphical interface with sketch entities that are marked and correlated to client identities.
  • Fig. 6 shows a simplified example of a interactive graphical interface with sketch entities that are marked to visualize the availability of correlated multi-media information.
  • Fig. 7 shows a simplified example of a interactive graphical interface with sketch entities that are marked to visualize the chronological creation process of the sketch entities.
  • Fig. 8 shows the simplified system architecture for a centralistic distribution system.
  • Fig. 9 shows the simplified system architecture for an equalized distribution system.
  • a interactive graphical interface 52 (see Figs. 5-7) is provided to a number of clients.
  • the interactive graphical interface 52 allows clients Cl-N, C2-N (see Figs. 8, 9) to create freehand drawn sketch entities.
  • the drawing process is captured in a real time manner such that simultaneously captured multi-media information can be precisely correlated.
  • the sketch entity is a curve 2 (see Figs. 1, 2) represented by a number of connected line segments 3 (see Figs. 1, 2) .
  • the sketch entity consists of one curve 2.
  • Fig. 1 shows an example of such a basic sketch entity.
  • Time stamps Tstll-IN, Tst21-2N have a clock frequency Clf (see Fig. 3) that may be defined: either by the clients operating system, or it may be a parameter that is uniformly defined for all clients.
  • the clock frequency Clf is processed as a function of a computers internal clock and is preferably constant.
  • the creation process of the sketch entity commences with the initiation event IE10-N, IE20-N (see Figs. 3, 4).
  • the initiation event IE10-N, IE20-N is, for instance, the down click of a mouse button at the time, when the cursor is within the drawing area 51 (see Figs. 5-7) of the interactive graphical interface 50.
  • the initiation event IE10-N, IE20-N may also be the contacting of a drawing pin with the surface of a touch screen or an activation click of a specified button of a digitizer board.
  • the initiation event IE10-N, IE20-N may be any interaction of the client with any kind of input device that is feasible to recognize a predetermined initiation command.
  • the voice recognition system may be incorporated in the system of the present invention or may be an independent system incorporated in the client's computer.
  • the drawing of the curve 2 is initiated at the initiation point 4.
  • the client' s drawing movement is captured in sequences that correspond to the clock frequency Clf of the time stamps Tstll- IN, Tst21-2N.
  • a progressive number of points 6 are created within the drawing area 51.
  • the points 6 are connected by line segments 3.
  • the creation of the sketch entity is finished, when the client initiates the termination event TE10-N, TE20-N (see Figs. 3, 4).
  • the termination event TE10-N, TE20-N is, for instance, the release of a pressed mouse button.
  • the termination event TE10-N, TE20-N may also be the removal of a contacting drawing pin from the surface of a touch screen or a termination click of a specified button of a digitizer board.
  • the termination event TE10-N, TE20-N may be any interaction of the client with any kind of input device that is feasible to recognize a predetermined termination command.
  • the voice recognition system may be incorporated in the system of the present invention or may be an independent system incorporated in the client's computer.
  • the system analyzes the numeric values of the coordinates of points 6. During this analysis, the extreme values of the x and y coordinates are recognized. These extreme values are utilized by the system to create a boundary rectangle 1.
  • the boundary rectangle 1 is defined to serve as a dummy object, which is utilized during the editing, viewing and replaying mode of the invention.
  • the clock frequency Clf defines in combination with the drawing speed the resolution of the curve 2. In other words, the faster the drawing speed for a given clock frequency Clf the longer the distance between individual points 6.
  • the clock frequency Clf is adjusted to a feasible level that balances the average drawing speed at which clients create the sketch entities with a minimal required curve resolution.
  • a basic sketch entity is created as an independent element of a more complex free hand drawing and/or to encircle or underline a feature of a background image that is displayed by the system in the viewable area 51.
  • Fig. 2 shows an example of an advanced sketch entity.
  • the system provides the possibility to create advanced sketch entities that consist of a number of combined curves 22a-d. Freehand drawings are typically created with a certain inaccuracy.
  • the system of the present invention assigns proximity areas 26a-d to the points 6.
  • the proximity areas 26a- d are predetermined areas surrounding the points 6.
  • the areal extension of the proximity areas 26a-d may be defined in a vector format or a coordinate format .
  • Proximity areas 26a-d are recognized in correlation to the curves 22a-d. As a result, proximity areas 26a-d that overlap with each other and do not belong to the same of the curves 22a- d trigger an automated combining of the correlated curves 22a-d.
  • the size of the proximity areas 26a-d is defined in correlation to the maximal space between the points 6 such that a closed area in the vicinity of the curves 22a-d is covered by the proximity areas 26a-d.
  • the combining function may be activated as part of the system setup and/or individually by assigning the initiation event IE10-N, IE20-N to two separate initiation commands. In case of a mouse this may be, for instance, the down click of the right mouse button for the initiation event IE10-N, IE20-N with combining function and the down click of the left mouse button for the initiation event IE10-N, IE20-N without combining function.
  • initiation commands for the initiation event IE10-N, IE20-N may be applied to any other input device, including a voice recognition system.
  • the boundary rectangles 21a-d may be combined to the combined boundary rectangle 21e and/or remain as independent dummy objects .
  • the system may further provide automated geometric feature recognition to correlate standardized geometric elements to the freehand drawn curves.
  • automated geometric feature recognition to correlate standardized geometric elements to the freehand drawn curves.
  • automated geometric feature recognition may be extended to recognize any free hand drawn geometric form and replace it with computer generated accurate geometric elements.
  • the automated feature recognition may be activated during the setup of the system or it may be independently activated with a feature recognition command.
  • the feature recognition command can be incorporated, for instance as the handling variation of the input device.
  • the handling variation may be a single down click for an initiation command without feature recognition and a double click for an initiation command including feature recognition.
  • additional multi-media information may be captured.
  • Fig. 3 is shown to explain the basic procedure of capturing sketching activities and correlated media information.
  • the combined graph shows in its top section a video signal Vi , in its middle section the audio signals AIO-N and in the bottom section the sketch activity curves SklO-N.
  • the top vertical axis V corresponds to the signal density of the video signal Vi
  • the middle vertical axis A corresponds to the acoustic level of the audio signals AIO-N
  • the bottom vertical axis SK corresponds to the drawing path during the creation of the curves 2, 22a-d.
  • the incline angle of the sketch activity curves SklO-N corresponds to the drawing speed at which curves 2, 22a-d are created.
  • the horizontal axis of the top, middle and bottom section represent the elapsed time.
  • the vertical raster lines that cover the top, middle and bottom section represent the time stamps Tstll-IN.
  • the spacing between the vertical raster lines represent the clock frequency Clf.
  • a conventional computer has hardware components like, for instance, a microphone and a sound card to capture and process audio information respectively a camera and a video card to capture and process video information.
  • a computer is typically equipped with an operating system that is able to process and embed this audio and video information in application systems like the one of the present invention.
  • An access procedure may be, for instance:
  • the system assigns the time stamps Tstll- IN during the creation and/or editing mode simultaneously to the sketching activities and to the captured audio and video.
  • Audio and video are continuously captured during the creation and/or editing mode.
  • the audio signals AIO-N are typically interrupted by silence periods AS.
  • the audio signals AIO-N represent preferably verbal information provided by the clients.
  • Silence periods AS typically separate blocks of coherent verbal information.
  • the video signal Vi is typically a consistent stream of video data that corresponds in size and structure to the image resolution, the color mode, the compression ratio and the frames per time unit.
  • the video signal may be a sequence of still images at a rate that the still images are recognized as still images or that they combine in a viewers mind to a continuous flow.
  • a selected document is replayed such that the individual sketch entities are automatically recreated in the drawing area 51.
  • the automatic recreation is performed in a chronological manner.
  • the audio signals AIO-N and video signal Vi are replayed synchronously together with the recreation of the individual sketch entities.
  • a selected document is displayed with all sketch entities.
  • the client selects one or more individual sketch entities.
  • a replay initiation routine analyzes all time stamps Tstll-IN correlated to the selected sketch entities and determines the earliest one. The earliest detected of the time stamps Tstll-IN is taken by the system to define a common starting moment for the video signal Vi and for the audio signals AIO-N respectively the silence periods AS. Audio and Video continue until the next selection of one or more sketch entities is performed by the client. At that moment, the replay initiation routine is initiated again.
  • the selection process is defined by the system in the preferred form of a selection rectangle.
  • the selection rectangle has to be created by the client by indicating two diagonal selection points within the drawing area 51.
  • the selection rectangle selects the sketch entities by surrounding and/or intersecting with their correlated dummy objects.
  • the selection process is performed by initiating a selection command when the cursor is placed by the client within one of the proximity areas 26a-d. By doing so, the client is able to distinctively select singular sketch entities.
  • the alternate embodiment is applied in cases of high densities of individual sketch entities within the drawing area 51.
  • the system provides an advanced procedure to capture sketching activities and correlated media information.
  • Fig. 4 is shown to explain the advanced procedure.
  • Fig. 4 corresponds with its elements mainly to those of Fig. 3.
  • the audio signals A20-N are comparable to the signals AIO-N, the sketch activity curves Sk20-N are comparable to the sketch activity curves SklO-N.
  • Fig. 4 introduces a audio switching level shown in the middle section with the horizontal line SI.
  • Block elements of media information are provided during the advanced procedure by recognizing only audio signals A20-N that are at a level above the switching level.
  • the system captures audio signals A20-N between the audio initiation moments AI1-N and the audio termination moments AT1-N.
  • the audio initiation moments AI1-N and the audio termination moments AT1-N share preferably the same switching level. It is noted that the invention applies also to the case, when the audio initiation moments AI1-N and the audio termination moments AT1-N are triggered at different switching levels.
  • the system assigns the audio initiation moments AI1-N and the audio termination moments AT1-N to the closest of the time stamps Tst21-2N. These times stamps Tst21-2N are utilized to cut the corresponding video sequences V20-N out of the video signal Vi and to assign them to the correlated audio signals A20-N.
  • Time relations are, for instance:
  • the audio assigning procedure and the block assigning procedure may be performed with an approximation algorithm provided by the system either simultaneously at the time the creation mode respectively the editing mode is activated, or after the creation mode respectively the editing mode is terminated.
  • the advanced procedure allows the client to selectively witness the multi-media blocks that is correlated to the selected sketch entity.
  • the system provides the client with an optional predetermined audio and/or video signature to inform him/her at the end of the correlated multimedia block.
  • the advanced procedure prevents the client from accidentally witnessing multi-media information that does not relate to the selected sketch entity.
  • the system optionally displays the individual s ket ch element s in varying styles .
  • the admini strative information is , for instance : 1) client identification correlated to individual sketch entities of a collaboratively created document; 2 ) information about available multi-media blocks for individual sketch entities contained in a document; 3) chronological creation of the sketch entities contained in a document.
  • Figs. 5, 6 and 7 show in that respect a simplified example of the interactive graphical interface 52 provided by the system together with examples of graphical coding of sketch entities according to the above listing.
  • Fig. 5 the sketch entities 53, 54, 55 are shown with first graphical codes to mark them according to their creators client identification.
  • the graphical codes are varying line fonts. Graphical codes may be of any color, shape, symbolic contents and/or dynamic respectively static luminescence variations.
  • a collaborating client list 57 is displayed together with the assigned graphical codes.
  • Fig. 6 the sketch entities 63 and 64 are shown with second graphical codes to mark them in case multi-media blocks are available.
  • the graphical codes are varying line fonts. Graphical codes may be of any color, shape, symbolic contents and/or dynamic respectively static luminescence variations.
  • a nomenclature 67 is displayed together with the assigned graphical codes. The second graphical codes may also be applied during the viewing mode to dynamically high light the sketch entity, whose multi-media block is replayed.
  • Fig. 7 the sketch entities 73-76 are shown with third graphical codes to mark them according to their creation chronology.
  • the graphical codes are varying line fonts.
  • Graphical codes may be of any color, shape, symbolic contents and/or dynamic respectively static luminescence variations.
  • a nomenclature 77 of the sketch entities is displayed together with the chronologically applied third graphical codes.
  • the third graphical codes may be preferably designed with a fluent transition such that the chronology of the creation process can be easily recognized. Fluent transitions are, for instance:
  • the system provides a variety of background images that may be displayed in the display area 51.
  • Background images are preferably pictographic images like, for instance: 1) photographs; 2) scans of graphics and/or blueprints;
  • system may also include background images in vector format as they are known to those skilled in the art for CAD drawings.
  • Background images may be imported at the beginning and/or at any time during the creation of a new document or under laid behind an existing creation of sketch entities.
  • the system utilizes the computers video capturing capability to retrieve snapshots of the displayed video and to provide the snapshots as background images.
  • the snapshot retrieval function is preferably activated during the creation mode.
  • the snapshot is taken by the client Cl-N, C2-N by performing a snapshot capturing command, which is simultaneously performed during the real time display of the displayed video.
  • a snapshot capturing command may for instance be a mouse click at the moment the cursor is placed within the video display screen 59A.
  • the snapshot retrieval function allows the client Cl-N, C2-N to comment in a quasi simultaneous way a captured video. Hence, the snapshot retrieval function is particular feasible to combine a live visual experience with a documentation procedure. Applications for the snapshot retrieval function are, for instance, inspection of construction sites.
  • FIG. 5-7 further show the optional video display screen 59A and the optional audio control screen 59B.
  • Video display screen 59A and the audio control screen 59B are conventionally provided by the operating system and may be controlled by the system of the present invention. It is noted that the video display screen 59A and/or the audio control screen 59B may be provided by the system of the present invention.
  • the video display screen 59A displays, for instance:
  • the audio control screen 59B performs functions, as they are commonly known to control the recording and replay of audio data on a computer.
  • the audio control screen 59B is typically provided by the operating system and may be controlled by the system of the present invention.
  • the system provides a number of standardized commandos to perform tasks like, for instance, opening, printing, viewing and scrolling a document.
  • the standardized commandos are commonly known for computer programs .
  • Fig. 8 and 9 show two different system architectures for the present invention.
  • Fig. 8 shows the preferred embodiment of a centralistic system architecture incorporated in a web page distribution system.
  • a server SI operates a web page, which is accessible by a number of clients C11-1N.
  • the client Clll After the client Clll has performed an identification routine, the client Clll is able to access the interactive graphical interface 52. ' A processing program that provides the creating, editing, replay and viewing modes becomes available.
  • the processing program enables the computer Coll to create and store the script logs Scll-N.
  • the script logs Scll-N contain all data gathered during the creation mode respectively during the editing mode.
  • the computer Coll is in bi-directional communication with the server SI, which stores the script log Sell in a permanent log PI.
  • the permanent log PI is the computer readable representation of the creation process of a document. It is continuously updated with all scrip logs Scll-SclN that are created on the computers Coll-ColN.
  • a database DblO maintained by the server SI stores the permanent logs PI of a number of documents created and edited by the clients C11-C11N.
  • the server SI is the central storing and redistribution site for all documents.
  • a client Cl wants to retrieve a document for the purpose of viewing or editing, he/she initiates a retrieval request command.
  • the retrieval request command prompts the interactive graphical interface 52 to provide the client Cll access the database DblO.
  • the requested document is transmitted in the form of the permanent log PI to the computer Coll and becomes accessible for replay, editing and viewing. All changes are documented in an additional script log Sclll-SclN that is sent back to the server SI, where the newly created script log Sclll-SclN is added to the already existing permanent log.
  • Erasing activity may be captured as a regular part of the creation process and/or removed from the script log and the permanent log during the editing mode.
  • the creation mode further provides a rewind function to allow the user to rewind and erase the captured creation process up to a chosen moment and to start over again.
  • the script logs Sclll-SclN may be transmitted to the server SI continuously during the creation mode respectively during the editing mode and/or after these modes are ended.
  • the centralistic system architecture may be applied to any form of network wherein the clients Cll-CllN can logon at any time to the server SI. Further, the centralistic system architecture may consist out of a number of servers SI that compare and update the context of their database DblO independently of the operation of the computers C11-C1N.
  • the system operates with an equalized system architecture as shown in Fig. 9.
  • each of a number of clients C21-C2N operates independently a computer Co21-Co2N, which maintains independently a database Db21-Db2N.
  • the databases Db21-Db2N are stored on a first direct access storage device (FDASD) .
  • the databases Db21-Db2N contain a number of permanent logs P121-P12N, which are created, accessed, edited and maintained as described under Fig. 8.
  • the processing program that provides the interactive graphical interface 52 and the functional operation of the system, as described above, is permanently stored on a second direct access storing device (SDASD) of the computers Co21-Co2N.
  • SDASD second direct access storing device
  • the storage medium of the SDASD and/or the FDASD may be a removable storage medium like, for instance, a CD or it may be incorporated in the computers Co21-Co2N as it is the case, for instance, in a hard disk drive.
  • the equalized system architecture allows the clients C21-C2N to operate the system independently of an available communication connection. Hence, the equalized system architecture is particularly feasible in combination with wireless communication systems.
  • the centralistic and the equalized system architecture may be combined temporarily or in any other feasible scheme to combine the specifics of each system architecture.
  • the centralistic system architecture and the equalized system architecture provide two communication modes:
  • a time independent communication mode is favorably utilized in combination with the equalized system architecture, whereas the quasi real time communication mode is favorably utilized in combination with the centralistic system architecture.
  • each of the clients C11-C1N, C21-C2N works at a document at any time.
  • the script logs Sclll-ScllN, Scl21-Scl2N are correspondingly created at any time.
  • the system performs a low level script log distribution management during the time independent communication mode.
  • the system has to perform a high level script log distribution management to reduce time delays in the distribution process between the clients C11-C1N, C21-C2N.
  • the system performs an automated ranking of data priorities. Data with low priority respectively less significance for a quasi real time collaboration ' is transmitted after high priority data has been transmitted.
  • Operating parameters include, for instance, user identification, file conversion, application version.
  • the functional components of the inventive system are written in a computer readable code.
  • Various software development systems provide the tools to create the computer readable code of the inventive system in accordance to the possibilities and needs of the used operating system.
  • the code may be written, for instance, in the commonly known computer language Java.
  • an exemplary development system may, for instance, be Netshow.
  • the databases DblO, Db21-Db2N and/or the processing program may be installable on the computers Coll-ColN, Co21-Co2N in the form of:

Landscapes

  • Engineering & Computer Science (AREA)
  • Library & Information Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)
  • Computer And Data Communications (AREA)
  • Document Processing Apparatus (AREA)

Abstract

L'invention concerne un système conçu pour plusieurs utilisateurs et servant à créer, éditer, reproduire et visualiser des documents constitués par des croquis dessinés à la main. Ce système capture le processus de création avec des informations verbales et/ou visuelles fournies par chaque utilisateur et met automatiquement ces informations en corrélation afin de les reproduire ultérieurement de façon synchronisée. Ce système englobe plusieurs outils et caractéristiques permettant principalement de combiner l'activité de croquis avec des images existantes, d'extraire sélectivement des informations de support corrélées à des entités individuelles de croquis et de collaborer pratiquement simultanément à un document commun. On peut régler l'architecture du système sur différents paramètres dans l'infrastructure de communication. On peut mettre ce système en application dans tout programme logiciel, dans un service basé sur Web, un explorateur Web ou un système d'exploitation d'ordinateurs et/ou de dispositifs de communication.
PCT/US2000/012833 1999-05-12 2000-05-09 Systeme et procede servant a effectuer l'indexation, l'acces et l'extraction de donnees audio/video presentant une activite simultanee d'execution de croquis WO2000068759A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU48367/00A AU4836700A (en) 1999-05-12 2000-05-09 System and method for indexing, accessing and retrieving audio/video with concurrent sketch activity

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13378299P 1999-05-12 1999-05-12
US60/133,782 1999-05-12

Publications (2)

Publication Number Publication Date
WO2000068759A2 true WO2000068759A2 (fr) 2000-11-16
WO2000068759A3 WO2000068759A3 (fr) 2001-02-22

Family

ID=22460279

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/012833 WO2000068759A2 (fr) 1999-05-12 2000-05-09 Systeme et procede servant a effectuer l'indexation, l'acces et l'extraction de donnees audio/video presentant une activite simultanee d'execution de croquis

Country Status (2)

Country Link
AU (1) AU4836700A (fr)
WO (1) WO2000068759A2 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004056083A1 (fr) * 2002-12-18 2004-07-01 Orange S.A. Dispositif graphique mobile et serveur
EP1655678A1 (fr) * 2004-07-21 2006-05-10 GiveMePower GmbH Procédé pour l'enregistrement, en vue de leur recouvrement, de données audio dans un système informatique

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5608859A (en) * 1993-12-28 1997-03-04 Nec Corporation Scenario editing apparatus
US5675752A (en) * 1994-09-15 1997-10-07 Sony Corporation Interactive applications generator for an interactive presentation environment
US6072479A (en) * 1996-08-28 2000-06-06 Nec Corporation Multimedia scenario editor calculating estimated size and cost

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5608859A (en) * 1993-12-28 1997-03-04 Nec Corporation Scenario editing apparatus
US5675752A (en) * 1994-09-15 1997-10-07 Sony Corporation Interactive applications generator for an interactive presentation environment
US6072479A (en) * 1996-08-28 2000-06-06 Nec Corporation Multimedia scenario editor calculating estimated size and cost

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004056083A1 (fr) * 2002-12-18 2004-07-01 Orange S.A. Dispositif graphique mobile et serveur
JP2006511112A (ja) * 2002-12-18 2006-03-30 オランジュ エス．アー． モバイルグラフィック表示装置
EP1655678A1 (fr) * 2004-07-21 2006-05-10 GiveMePower GmbH Procédé pour l'enregistrement, en vue de leur recouvrement, de données audio dans un système informatique

Also Published As

Publication number Publication date
AU4836700A (en) 2000-11-21
WO2000068759A3 (fr) 2001-02-22

Similar Documents

Publication Publication Date Title
US6724918B1 (en) System and method for indexing, accessing and retrieving audio/video with concurrent sketch activity
US7458013B2 (en) Concurrent voice to text and sketch processing with synchronized replay
CN107534704B (zh) 一种经由通信网络连接的信息处理方法、设备和介质
EP2940940B1 (fr) Procédés pour envoyer et recevoir des messages courts vidéo, appareil et dispositif électronique de poche associé
US9131059B2 (en) Systems, methods, and computer programs for joining an online conference already in progress
JP2006146415A (ja) 会議支援システム
CN111741324B (zh) 录制回放方法、装置和电子设备
CN112399132A (zh) 虚拟现实技术于远程会议系统中的应用方法
CN112035195A (zh) 应用界面的展示方法、装置、电子设备及存储介质
JP2004015750A (ja) ライブ配信サーバ、及びライブ配信方法
CN112751681A (zh) 图像处理方法、装置、设备以及计算机可读存储介质
CN111818383A (zh) 视频数据的生成方法、系统、装置、电子设备及存储介质
JP4696480B2 (ja) 遠隔会議システム、拠点サーバ及びプログラム
KR20130108684A (ko) 스마트 ｔｖ의 비디오 어노테이션 및 증강 방법 및 그 시스템
KR100258119B1 (ko) 대화형 멀티미디어시스템에 있어서 유저정보 편집 및 편집된정보 재생방법
JP2016063477A (ja) 会議システム、情報処理方法、及びプログラム
CN1029064C (zh) 自动音频及视频的呈现
KR20000054715A (ko) 인터넷상의 동영상컨텐츠 서비스시스템과 그 방법, 및동영상파일의 생성 및 전송방법과 그 기록매체
WO2000068759A2 (fr) Systeme et procede servant a effectuer l'indexation, l'acces et l'extraction de donnees audio/video presentant une activite simultanee d'execution de croquis
US20230362460A1 (en) Dynamically generated interactive video content
KR20080083490A (ko) 스케쥴링에 의하여 개인 방송국 서비스를 제공하는 시스템,장치 및 방법
WO2021073313A1 (fr) Procédé et dispositif de commande de conférence et de participation à une conférence, serveur, terminal et support de stockage
EP0848879B1 (fr) Annotation d'image dynamique au moyen de la telephonie
JP4244545B2 (ja) 情報作成方法、情報作成装置およびネットワーク情報処理システム
JP6344731B1 (ja) コンテンツ評価システム

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AU CA JP

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

121 Ep: the epo has been informed by wipo that ep was designated in this application
AK Designated states

Kind code of ref document: A3

Designated state(s): AU CA JP

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP