WO2002023350A1

WO2002023350A1 - Technique for superimposingly display additional information on display screen

Info

Publication number: WO2002023350A1
Application number: PCT/JP2000/006164
Authority: WO
Inventors: Hiroaki Haga
Original assignee: Digital-Knowledge Co., Ltd
Priority date: 2000-09-08
Filing date: 2000-09-08
Publication date: 2002-03-21
Also published as: JPWO2002023350A1; JP4574113B2

Abstract

A method for superimposingly displaying various objects and additional information including sound and animation on a screen, comprising a file creating step of moving or placing an image object to be added while inputting sound or animation in displaying a second screen being virtually present on a first screen and recording a sound or animation file and an information file where information concerning the position of the object on the second screen correlated with time and a display step of calling the files, displaying the first screen while linking the object with the input sound or animation and moving the object with time or placing the object.

Description

Description Technology for superimposing additional information on the display screen

The present invention belongs to a technology for superimposing and displaying additional information such as various objects and audio / video on a display screen, and for example, relates to a technology for improving the expressive power of the contents of an Internet homepage. Background art

With the development of the Internet, personal computers have spread rapidly. And it is used by a very large number of people, both young and old, mainly for sending and receiving e-mails and browsing websites. In recent years, new ways of using the Internet (business models) have been created one after another, and the internet has been called the second industrial revolution because it has unlimited potential.

By the way, as such new usages appear one after another, existing files, especially document files (typically HTML format files) that are exchanged via the Internet, are becoming too bland and interesting. Chip. On the other hand, there is a document file that contains some sort of object or voice, but it is difficult for a layman to create it, and at present it is rarely done. Under such circumstances, it would be extremely useful if there was a technology to easily incorporate various additional information into the file, and the use of the Internet would become more and more popular 1

No doubt. For example, when browsing the homepage, it would be useful to be able to input circled re-voices for important or questionable parts. It would also be more attractive if drawings and audio could be easily incorporated into existing still-image-only homepages. For users who are unfamiliar with the operation of personal computers, for example, if there is a homepage that explains the contents of the description by voice following the movement of a character such as a finger, it will be very easy to understand and familiar.

However, when a file written in such a way on a home page is transferred to a server using a conventional graphic tool, for example, if the file format is a document file, after drawing on the document file image, The problem is that it cannot be restored to the original document file format because it is saved in the image file format.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique for superimposing various objects, additional information such as audio and moving images on a display screen extremely easily. Disclosure of the invention

The present invention includes a step of calling a first file and displaying the image on a first screen; and a step of inputting a voice or a moving image while virtually inputting a second file virtually present on the first screen. Moving or arranging the image object to be added in the screen; obtaining the position information of the image object on the second screen in association with time; input audio information or moving image information Based on Creating an audio file or a moving image file; creating an object information file based on the information for specifying the image object, the location information, and the time information; Associating the audio file or the video file with the object information file; and displaying the first file on the first screen while linking the input audio or video to the second file. (1) a method of superimposing and displaying additional information on a display screen, the method including a step of moving or arranging the image object over time in a screen and displaying the image object.

The present invention also provides a step of calling a first file and displaying the image on a first screen; and a step of adding the image in a second screen virtually existing on the first screen. Moving or arranging the image object; obtaining information on the position of the image object on the second screen; and acquiring the image object based on the information for specifying the image object and the position information. Creating a project information file; associating the first file with the object information file; displaying the first file on the first screen and displaying the image in the second screen. And (b) displaying a supplementary object on the display screen.

Further, the present invention includes a step of calling a first file and displaying the image on a first screen; at a predetermined position of the second screen virtually existing on the first screen, Displaying an image object to be added that is linked to the audio file or the video file; obtaining position information of the image object on the second screen; and the audio file or the video. Specifying the output order of the files; outputting the audio file or the video file according to the specified output order, and displaying the image object linked to the audio file or the video file in the second screen. And (3) a method of superimposing and displaying additional information on a display screen.

The present invention also relates to (4), wherein (4) or (3), wherein no link information is added to the first file.

Furthermore, the present invention moves or arranges an image object in the second screen virtually existing on the screen of the first screen while inputting a sound or a moving image. A step of obtaining position information of the image object on the second screen in association with time; a step of creating a sound file or a moving image file based on the input sound or moving image; Creating an object information file by associating the information specifying the object with the position information and the time information; and transmitting the file related to the screen display of the first screen to the audio file or the moving image. A method of creating a source file that superimposes additional information on a display screen, including a step of associating a file with the above object information file Concerning the law (5).

The present invention also provides a step of, in a second screen virtually existing on a screen of the first screen, moving or arranging an image object to be added in the second screen; A step of obtaining the information of the position of the image object on the second screen; and a step of creating an image object file based on the information specifying the image object and the position information; Copy the file related to the screen display of the first screen to the object Relating to a method of creating a source file that superimposes additional information on a display screen, including a step of associating with a display information file.

(6).

Further, the present invention provides a step of displaying an image object to be added, which is linked to an audio file or a moving image file, at a predetermined position of the second screen virtually existing on the first screen. A step of obtaining position information of the image object on the second screen; and a step of specifying an output order of the audio file or the moving image file. How to make (7).

The present invention also relates to the methods (5) to (7), wherein any link information is not added to the file on the first screen (8).

Furthermore, the present invention relates to a magnetic recording medium storing a program for executing the steps defined in the above methods (1) to (8) (9).

The present invention also relates to a computer system having the magnetic recording medium defined in the medium (10) (10).

First, terms used in the above and claims will be described. “First file” refers to the file (processed file) in which additional information is embedded. The type of file is not particularly limited, and includes a document file, a spreadsheet file, an audio file, an image file, and a video file. The format is not particularly limited, and a document file is exemplified by an HTML format file. "Movie" includes not only video without sound but also video with sound. The “virtual second screen” is defined as the first screen and the i

A standing virtual screen that is transparent on the first screen (including translucent and colored transparent as long as the first screen can be seen through the second screen). “Audio file” and “video file” are not particularly limited, and include various types of files. “Image object” refers to image information added to the first file. For example, there are moving information of still images (movable images), drawing information, telops, fixed images, and the like. The phrase “while inputting audio or video” refers to recording audio and video when inputting an image, and existing audio and video files when inputting an image. Playing. The “object information file” is a file containing information necessary for displaying an image object on a file image to be processed as instructed. Generally called a data file. "Associating the first file with the audio file or the moving image file and the object information file" means that the link information is written in the first file and the first file link information is not written. For example, this includes recording in the database data in which the URL of the object information file, the URL of the audio file or the video file, and the URL of the file to be processed are associated. "Specifying the output order of the audio file or the moving image file" means specifying the output order of the audio file or the moving image file, and specifying the output order of the image object linked to the audio file or the moving image file. This includes indirectly specifying the output order of audio files or video files. In addition, according to the specified display order, the attached parts (set of object information file and image or video file) In addition to integrating the parts, a management file in which the display order of each part is recorded is preferably created without integrating the parts. BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a system configuration diagram of an embodiment according to the present invention. FIG. 2 is a hardware configuration diagram of an embodiment according to the present invention. FIG. 3 to FIG. 7 are flowcharts showing the internal processing of the first embodiment. FIG. 8 and FIG. 9 are internal processing flowcharts of the second embodiment. FIG. 10 is an operation flowchart of the third embodiment. FIG. 11 is an internal processing flowchart of part creation in the third embodiment. FIGS. 2 and 13 are flowcharts showing the internal processing in which the parts are incorporated in the third embodiment. FIG. 14 is a diagram illustrating a state where the input method selection window is displayed after the system according to the first embodiment is started. Fig. 15 shows the first mode in which the input instruction window for the file to be processed (target HTML) and the audio file (target sound) after "input new instruction" is selected in the input method selection window is displayed. FIG. FIG. 16 is a diagram showing a state in which the image (reference numeral 5) and the operation panel (reference numeral 10) of the file to be processed are displayed in the first embodiment. FIG. 17 is a diagram showing a state in which a list of movable images among the image objects is displayed. FIG. 18 is a diagram showing a state in which a list of basic figures that can be drawn among image objects is displayed. The color of the line can be selected with the code 6, and the thickness of the line can be selected with the code 7 (see Fig. 16). Figure 19 shows the telop. Characters, symbols, etc. to be displayed as telops are entered in the box of reference numeral 27. Figure 20 is a diagram showing a state where a list of fixed images is displayed. is there. In this figure, 12 fixed images are registered (reference numeral 28). FIG. 21 is a diagram showing a state where the control buttons are displayed. FIG. 22 is a diagram showing a state in which the movable image (finger) 30 is moved along the trajectory of the line 31 while inputting audio via an audio file in the first embodiment. FIG. 23 is a diagram showing a state in which a speed change list 32 is displayed from the operation panel in order to change the time axis of the audio file at the time of input. FIG. 24 is a diagram showing a state before the file to be processed in which the additional information is incorporated is reproduced. Reference numeral 3 3 indicates a playback button, and when this button is pressed, the playback operation starts. FIG. 25 is a diagram showing the movement of the movable image (finger) after the play button in FIG. 24 is pressed. In the figure, reference numeral 35 indicates a locus of a movable image (finger). Reference numeral 34 denotes operation buttons for fast forward, rewind, stop, and pause during playback. FIG. 26 is a diagram showing a screen displayed after the system is started in the second mode. Reference numeral 36 indicates a file to be processed. FIG. 27 is a diagram showing a trajectory after moving the image object (line) in the second embodiment. Reference numerals 37 and 38 denote trajectories of lines recorded as age object information. FIG. 28 shows a parts list used in the third embodiment. In this figure, five parts (No. 1 to 5) are displayed. FIG. 29 is a diagram showing a state where a plurality of parts (a plurality of sets of an object information file and an audio file) and a file to be processed (HTML file) are linked in the third mode. BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described in more detail with reference to the accompanying drawings. In the following, an HTML file is used as the file to be processed, but the present invention is not limited to this.

FIG. 1 is a system configuration diagram of an embodiment according to the present invention. In FIG. 1, reference numeral 1 denotes the Internet, reference numeral 2 denotes a user (operator) terminal (for example, a personal computer), and reference numeral 3 denotes a server. The user terminal and server are connected to the Internet via a communication line.

FIG. 2 is a hardware configuration diagram of an embodiment according to the present invention, which is basically the same as a commercially available personal computer. However, a sound recording unit for recording external sounds and a moving image recording unit for recording external moving images are provided. In this figure, the file to be processed (HTML file) and the audio (moving image) file are external, and the object information and programs are local. However, the locations of the files to be processed, audio (video) files, object information, and programs are not particularly limited.

The present invention is roughly classified into a first embodiment, a second embodiment, and a third embodiment. Therefore, in the following, description will be given for each mode.

1. First aspect

A first aspect of the present invention is a technique for easily creating a file to be processed in which audio and video information and intelligent object information are incorporated. As a creation method, existing audio file ゃ video file is used lb

In addition, there are two methods: a method of inputting object information (hereinafter referred to as method I), and a method of inputting object information while inputting audio and video (hereinafter referred to as method I). Hereinafter, each function, operation procedure, and internal processing in the system according to the first embodiment will be described.

(a) Description of each function

After activating the system and specifying the HTML file to be processed and deciding whether to create the file by method (1) or (2), the system is ready to enter the object information (Fig. 14 and Fig. 14). (Fig. 15 is a diagram related to the above specification and decision. However, since these diagrams assume only method (2), another panel will be displayed for method (2).) . At that time, an operation panel is displayed on the screen (see Fig. 16). On the operation panel 10, various input control functions (recording button 19, play button 20, pause button 21, stop button 22, delete button, etc.), various time control functions (time control Slide bar 18, playback speed change function 23, etc.), smart object selection function (movable image list 12, drawable basic figure list 13, telop input 14, fixed image) List 15), and a result creation support function (storage of saved files, preview, layout of play button, etc.) are provided. Since this operation panel is always displayed on the screen, these functions can be used at various times during input. Hereinafter, the various functions will be described.

★ Various input control functions (See Fig. 16)

* Record button 19: When this button is pressed, playback starts in the case of system ① (a system that uses existing audio and video files). Ί

In the case of the method (method of inputting audio or video), recording starts, and after pressing this button, recording of the specified object information starts.

* Play button 20: When this button is pressed, the input object information is read from the recording unit, and the object is arranged and operated according to the content according to the content.

* Pause button 21: Press this button to pause the recording of object information, the recording and playback of audio and video. This button is a toggle button and will not resume once pressed once it is pressed again. In this system, there are two other suspension methods. The first method is to press the cancel button (right button) on the mouse. The second is a method of automatically pausing and resuming depending on the position of the mouse cursor. For example, if the mouse cursor is outside the display area of the HTML file screen to be processed (here also on the operation panel, outside the display area of the HTML file screen to be processed), the recording and playback of audio and video are paused, If it is within the display area of the HTML file to be processed, release the pause of recording and playback of audio and video, and resume recording and playback. * Stop button 2 2: Press this button to stop recording and playback of audio and video, and stop recording of object information. In the case of method ① (method using an existing audio file or video file), even if this button is not pressed, the playback stops automatically when the playback of the existing file ends.

* Delete button (22 right button): Press this button to delete the input object information (saved in the temporary storage area). (1

★ Various time control functions (See Fig. 16)

* Slide bar 18: This is one of the time control functions of the operation panel. Like the existing audio and video playback tools, the file playback time is represented by a straight line, with the playback start and end positions at both ends. The knob indicates where the current playback position is. Dragging this knob changes the position of the input time. For example, if there is a time zone where it is not necessary to insert the object information during input, skip the unnecessary time zone; if the input timing does not match, return a little time; a certain time For example, when it is desired to insert a plurality of genius object information into a band, a certain genuine object information is recorded, then the time zone is returned again, and another genuine object information is recorded.

* Speed change function 2 3: This is used when you want to change the playback speed of these files in the case of method ① (method using existing audio and video files). For example, by setting the playback speed of the audio file to be slow, the time axis is extended, so that it is possible to input with a margin, and it is possible to calmly input the object information.

Female object selection function (See Fig. 17 to Fig. 20)

* List of movable images 1 2: A list of image data of movable images J registered in the system is displayed (reference numeral 25 in Fig. 17). If the operator adds new “movable image” image data to the system, it will be added to the list. Since the trajectory of the movement of the movable image does not remain, it points to the part explained and expresses the movement It is effective when you want to.

* List of drawable basic shapes 13: Free lines, straight lines, rectangles, etc. are registered as the basic drawable shapes (reference numeral 26 in Fig. 18). Since this drawing figure has a locus, it is effective when a marker is to be placed on a portion to be emphasized, or when writing and drawing freely on the processed HTML file screen like drawing software.

* Telop 1 4: When this is selected, it becomes possible to input a character string in a predetermined part (reference numeral 27) on the operation panel, and the input character string is pasted at the specified location on the HTML file screen to be processed. (See Figure 19). It can be used for supplementary explanations.

* Fixed image 15: By selecting a fixed image recorded in the system, it can be pasted into the image like a sticker (see Fig. 20).

★ Result creation support function

* Save button: By pressing this button, a smart object information file (data file) reflecting the input smart object information is created. In the case of simultaneous recording and recording, an audio file and a video file are created, respectively. In addition, an operation of linking the processed HTML file, the object information file, and the audio file or the moving image file is performed. * Play button layout button: Press this button to display the button layout window. A plurality of button candidates are listed in the button arrangement window, and the operator selects a desired button from the list and arranges the button at a desired position in the image of the HTML file to be processed. By pressing the play button placed here if

It can be operated according to the input contents.

How to enter information

At the time of recording (if not paused or stopped), the age object information is recorded only while the mouse's decision button (left button) is pressed. Note that the age object information is displayed at the position of the mouse cursor.

Input range of female intelligence object information

There is no restriction in the positive direction (that is, there is no restriction in the right direction and the downward direction) with respect to the base point (0, 0) from the upper left of the HTML file screen to be processed. Is restricted, so that movement in the negative direction is not possible (the project stops there).

(b) Explanation of operation procedure

First, the operator starts the system. Next, after starting up, the input method selection window will be displayed. “Use existing file and input new instructions” (method ①), “Simultaneous input, input new instructions” (method ②), “ Add an instruction to an existing file ”or“ Create HTML for web publishing J ”(Note that Figure 14 assumes that only method ② is input, so the above input method selection window In the following, it will be explained in detail when the method ① or ② is selected, and when the other “Add instructions to existing files” and 作成 Create HTML for web publishing ”are selected (Fig. 14) will be briefly mentioned at the end of this section.

Regardless of which method (1) or (2) is selected, the path of the HTML file to be processed ■ Enter the file name (see Fig. 15). Also I

May select a file from the displayed file selection window instead of inputting. The HTML file to be input (selected) may be on a local computer or on the Internet. As another method, if the system is specified to read the HTML file to be processed, the system automatically starts after reading the HTML file to be processed. (In this case, it is necessary to create an HTML file to be processed to which a line for calling this system has been added in advance). However, in the following description, it is assumed that an HTML file in which the address of the engine is not specified is used.

Then, in method (2), the user inputs (or selects) the name of the audio file or the video file to be used (note that Fig. 15 assumes only the audio file). In method (2), input the name of the audio file to be created, the name of the movie file, the path to create it, and the file format (including detailed settings for the audio and movie storage formats). Then, press the Finish button.

After pressing the Finish button, the processed H TML file image 5 and the operation panel 10 are displayed on the screen (see Fig. 16). At the time of initial display, the input object is set to the default (as shown by reference numeral 9, the movable image “finger” is selected as the image object). If you want to input an object that is different from the default object, select the desired object from the operation panel〗 0.

Next, by pressing the recording button 19, the input of the smart object information is started (Fig. 22). For method II, the specified audio a

Alternatively, since a moving image file is played, a smart object is displayed or moved using a mouse in accordance with the played sound or moving image. For example, in the case of FIG. 22, the movable image 30 is moved in the direction indicated by the reference numeral 31. In addition, in the case of this method, it stops automatically when the reproduction of the existing file ends, so the operator needs to complete the input before the end. In the case of method II, while recording or recording, use the mouse to display or move the object. In the case of simultaneous input, input is possible as long as the capacity of the recording unit (for example, auxiliary storage device) permits. In either case, since there are three types of temporary stop means as described above, it is possible to input calmly by properly using them.

When the input of the smart object information is completed, the operator presses the stop button 22. However, in the case of the method (2), the playback is automatically stopped after the playback (that is, when the slide bar 18 reaches the right end). In this case, it is not always necessary to press the stop button 22.

Next, the operator arranges a playback button for playback on the HTML file screen to be processed.

Finally, press the save button and close the browser to end this operation.

(c) Description of internal processing

Next, the internal processing will be described with reference to the flowcharts of FIGS.

First, after the system start-up, put out a selection instruction or to enter the Riobu Jeku Bok information by any of the method ① or ② (step 2 0 2) ₀ /?

Next, when the operator selects the method II (when using an existing audio file or video file), a file selection window (4 in FIG. 15) is displayed and the path of the HTML file to be processed is displayed. · File name and the path of the audio file or video file to be used · Enter the file name (selection) and give instructions (step 204). If the operator selects method (2) (input of audio or video at the same time), the path of the HTML file to be processed ■ File name and the audio file or video to be created Path to save the file · File name ■ Input (select) save format instruction (Step 207).

After inputting (selecting) the path, file name, etc. by the operator in these files, in the case of method (2), the path of the input audio file or video file. The file is sent to the playback unit, and the audio playback unit or the video playback unit reads the specified file (step 205). In the case of method II, the path, file name, and storage format are sent to the audio recording unit or video recording unit for the audio file or video file to be created, and the audio recording unit or video recording unit transmits the transmitted information. Prepare to create an audio or video file based on that (step 208).

Next, in both methods (1) and (2), the input HTML file to be processed is read and displayed by the browser used by the operator (step 209, see FIG. 16). Then, an operation panel 10 and a preview window are displayed on the processed HTML file image 5 (step 210).

After the operation panel 10 is displayed, the identification number of the default The number is sent to the recording unit (step 211). In the case of FIG. 16, since the “finger” of the movable image is selected as the default, the identification number is sent to the recording unit. When the operator has changed, the identification number of the changed smart object is sent to the recording unit (step 220), and the smart object corresponding to the identification number is read (step 222). At this stage, internal preparation is completed and the system waits for operator's instructions.

When the record button 19 is pressed, in step 2 32, in the case of method ① (in the case of using an existing file), the audio reproduction unit starts audio reproduction (in the case of using an audio file), and The playback section opens the video preview window and starts playing the video (if you have selected to use a video file). In the case of method II, in step 233, the voice recording unit starts recording (when simultaneously recording voice) and the video recording unit starts recording (when simultaneously recording moving images).

At the same time as playback or recording (recording), the pointing device detection unit starts detecting the position information of the mouse cursor (step 234). Thereafter, the position of the mouse cursor is constantly monitored. When the mouse cursor is outside the display area of the HTML file image 5 (here, the operation panel 10 is also outside the display area of the HTML file image), the recording of audio and video and the playback of the video are paused. (Step 2 3 6). When the mouse cursor moves into the display area of the HTML file image 5, the pause of the recording / playback of the audio / video is released, and the recording / playback is resumed (step 238). Even if the operator presses the cancel button (right button) of the re-mouse, recording and playback of audio and video are paused. ?

(Step 2 4 3). Then, when the pause is pressed again, the pause is released and the recording is resumed. ■ Playback is resumed (step 2 45) ₀

Then, unless the pause button is pressed after the recording button 19 is pressed until the stop button 22 is pressed, the current selection is made when the mouse's enter button (left button) is pressed. The selected object is displayed at the position of the mouse cursor (step 247). In addition to the display of the object, the current time is acquired at regular intervals from the audio playback unit or the video playback unit (in the case of method I) or the audio recording unit or the video recording unit (in the case of method I) at each sampling cycle ( Specifically, the current time is obtained based on the position on the time axis starting from 0 seconds during playback / recording), and sent to the recording unit as time information, and the current mouse cursor position information is obtained. (Specifically, the X and 丫 coordinates are obtained starting from the upper left of the HTML file to be processed. If the value of the X or Y coordinate becomes negative, it is truncated and recorded as 0. There is no limit and the value is recorded) and sent to the recording unit (step 249). Then, as long as the decision button (left button) of the mouse is kept pressed, the above acquisition of the current time and position information is repeated.

After that, when the stop button 22 is pressed by the operator (step 250), the audio playback unit or the video playback unit stops playback in the case of the method ①, and the audio recording unit or the movie recording unit in the case of the method ②. Stop recording (step 25 1). Further, in the case of the method (2), when the reproduction of the sound or the moving image is completed, the stop processing is performed even if the stop button 22 is not pressed (step 2411). The pointing device detector detects and monitors the position information of the mouse cursor. 1.0

Stop viewing (step 25). When the stop is released, it will be in the state of waiting for the operator's command.

Next, when the play button layout button is pressed by the operator (Step 2 26), a button layout window is displayed (Step 2 27), and after the type and position of the play button are determined by the operator, (Step 228), as in the case of the object, send the type (identification number) and position information of the playback button to the recording unit (Step 229) ₀

Then, when the save button is pressed (step 2 2 4), in the case of method (1), the video recorded by the video recording unit (for simultaneous recording) or the audio recorded by the audio recording unit (for simultaneous recording) Based on the information received in step 209, step 233, step 233, step 238 and step 245, a moving image file or an audio file is created, respectively. Also, an object information file (for example, a JS file format) is created based on the information written to the recording unit in Steps 212, 220, 229, and 249. (Steps 2 5 5). The input object information file contains the path and file name of the audio file or video file used (in the case of method）) or the path and file name of the created audio file or video file (in the case of method ②); input Information about the genuine object (for example, the identification number of the genuine object, information necessary for the genuine object (for example, the thickness and color of the line for a genuine object, and the telop for a telop) Write the identification number of the play button, position information, etc. Thus, as many objects as input objects Ά

Since the information is added to the project object information file on a project-by-project basis, the project objects are independent. Note that there is only one playback information file to be placed on one time axis (audio file or video file) and one play button to be placed. Next, a correction is made to the processed H TML file. The contents of the correction are information for linking the HTML file to be processed with the created object information file (for example, the path and file name of the object object information file) and an engine for reading the object object information file (driving part). ) (For example, engine path and file name) is written (step 260). In addition, since this correction only adds a few lines to the source of the processed HTML file, it is possible to correct the processed HTML file after inputting the operation. However, depending on the HTML file creation software to be processed, the contents added in this system may be invalidated, so correction at the source level is desirable. It is to be noted that a method that does not modify the HTML file to be processed is also possible (step 255), and the details of this method will be described in the item of the second embodiment. Further, it is determined whether or not a moving image is used as a time axis (step 256), and if so, the HTML file to be processed is corrected, and the image of the HTML file to be processed is displayed on the screen. In the horizontal part of the processed HTML file image, the video is arranged so that it can be browsed (step 257).

It should be noted that multiple time axes (audio files or video files) can be assigned by repeating the operations from step 201 to step 257 for one HTML file to be processed. Is a project object information file. There are also multiple regeneration buttons.

After inputting the instruction to close the browser, close the operation panel 10 (step 261) and end this program.

(d) Explanation of "Add instructions to existing file" and "Create HTML for public use of web"

Finally, other items in the input method selection window, "Add instructions to existing file" and "Create HTML for public web" are explained.

★ "Add instructions to existing file"

This item is selected when inputting additional instructions to the HTML file to be processed in which the subject information has already been incorporated by this system. To do so, select the “Add instructions to an existing file” item from the file selection window, and then select the HTML file to be processed for additional input. However, there is no need to select the audio file or video file that serves as the time axis because it is already linked to the HTML file to be processed, and there is no need for simultaneous input. Select the HTML file to be processed, read the file, read the linked object information file (send it to the recording unit), and input the audio file name or video file name into the audio playback unit or video playback unit. Send to After that, the process proceeds to step 209, and the rest is the same as in the case of method II.

★ "Create HTML file for web publication"

Normally, the location of the engine part file added to the processed HTML file is specified by a URL. Usually, the path of the local machine used is specified. So, select the "Create HTML file for web publishing" By specifying the URL of the part of the address on the Internet, the created HTML file to be processed can be used on the web. The HTML file used in the following second aspect (the URL of the engine is specified in advance) is also created by selecting this item.

2. Second aspect

A second aspect of the present invention is one in which the time axis of the first aspect is eliminated. In other words, without using or creating an audio file or a moving image file, the smart object information is written and the file to be processed is linked. Hereinafter, first, each function, operation procedure, and internal processing in the system according to the second aspect will be described.

(a) Description of each function

After activating the system and specifying the HTML file to be processed, the system is ready to enter the project object information. At that time, an operation panel is displayed on the screen. On the operation panel, various input control functions (delete button etc.), smart object selection function (still image, list of drawable basic figures, telop input, moving image, etc.), result creation support function ( Storage of saved files, preview, play button arrangement, etc.) are provided. The various functions will be described below.

* Various input control functions

* Delete button: Press this button to delete the input object information (saved in the temporary storage area).

* Display 'Hide display switching button: Processed HTML file image Switch between displaying and hiding the placed object.

Female genius object selection function

* Still image list: A list of image data of "still images" registered in the system is displayed. The operator selects a desired still image from these. If the operator adds new “still image” image data to the system, it is added to the list. The operator pastes the still image on the processed H TML file image like a sticker.

* List of basic shapes that can be drawn: Free lines, straight lines, rectangles, etc. are registered as basic shapes that can be drawn. Since this drawing figure has a locus, it is effective when a marker is to be placed on the part to be emphasized, or when writing and drawing freely on the processed HTML file screen like drawing software.

* Ticker: When this is selected, a character string can be input to a predetermined portion on the operation panel, and the input character string is pasted at a specified location on the processed HTML file screen. It can be used for supplementary explanations.

* Moving images: By selecting a moving image recorded in the system, it can be pasted into the image like a sticker.

★ Result creation support function

* Save button: By pressing this button, a smart object information file (data file) reflecting the input smart object information is created. In addition, an operation of linking the processed HTML file and the project information file is performed.

* Play button layout button: Press this button to display the button layout window. In the button arrangement window, A plurality of button candidates are listed, and the operator selects a desired button from the list and places it in a desired location in the image of the HTML file to be processed.

★ How to enter object information

At the time of recording (until the save button is pressed), the age object information is recorded only while the enter button (left button) of the mouse is pressed. Note that the age object information is displayed at the position of the mouse cursor.

Input range of female intelligence object information

There is no restriction in the positive direction (that is, there is no restriction in the right direction and the downward direction) from the upper left base point (0, 0) of the HTML file screen to be processed. Is restricted, so that movement in the negative direction is not possible (the project stops there).

(b) Explanation of operation procedure

First, the operator starts the system. Next, after startup, the path of the H TML file to be processed ■ Enter the file name. Alternatively, a file may be selected from the displayed file selection window instead of inputting. In this case, the HTML file to be input (selected) is preferably on a local computer. As another system startup method, it is also possible to automatically start up the system when loading the HTML file to be processed with a browser. In this case, the HTML file to be processed may be on a local computer or on the Internet.

After that, the processed HTML file image and operation panel are displayed on the screen. And, the object selection function of the operation panel To select the desired talent object.

At this stage, it is in a state of waiting for input of input object information. The operator places the smart object at a desired position using a mouse. When the drawing object is selected, move the mouse while pressing the left button of the mouse while drawing (see Fig. 27, but the operation panel is off-screen or closed in Fig. 27). When a genuine object other than a drawing object is selected, the position to be pasted is determined.

When the input of the input object information is completed, a playback button for playback is arranged on the HTML file screen to be processed.

Finally, press the save button and close the browser to end this operation.

(c) Description of internal processing

Next, the internal processing will be described with reference to the flowcharts of FIGS. In the following, the internal processing when the URL of the present system is written in the HTML file to be processed will be described.

First, the operator displays the image of the HTML file to be processed on the screen using a browser (step 401), and, in accordance with the URL of the system (engine part) written in the HTML file to be processed, the system Is started and the operation panel is displayed (Step 402). At this time, the operation panel should not cover the processed H TML file image.

After displaying the operation panel, the identification number of the smart object is sent to the recording section (step 407), and the smart object corresponding to the identification number is inserted. Then, the pointing device detection unit starts detecting the position information of the mouse cursor. If the drawing object is selected, and if the left mouse button is pressed in the processed HTML file image until the save button is pressed, the drawing object is selected. The position of the mouse cursor is acquired along with displaying the trajectory of the mouse on the screen (Steps 411 to 413). If the value of the X or Y coordinate becomes negative, it is truncated and recorded at 0, while the value is recorded without any restriction in the positive direction) and sent to the recording unit (step 41). 2). Then, as long as the mouse decision button (left button) is kept pressed, the display of the locus and the acquisition of position information are repeated. If any other object is selected, the identification number of the object and the information necessary for the object (for example, the content of the displayed character string in the case of a telop) are sent to the recording unit. (Step 407), the target object is displayed at the position of the cursor, and if the enter button is pressed (Step 409), the position information is sent to the recording unit (Step Ί0).

Next, when the play button layout button is pressed by the operator, a button layout window is displayed, and after the type and position of the play button are determined by the operator, the identification number is set in the same manner as in the case of the smart object. And position information to the recording unit (not displayed on the flowchart).

After that, when the save button is pressed, based on the information written in the recording unit in step 407, step 410, and step 412, the input object information file (data file) (Step 4 1 4). The genius object information file contains information about the genius object that was input (for example, the identification number of the object, the information necessary for the genius object (for example, the line thickness and color, In the case of, the contents of the text displayed in the telop), position information}; Write the identification number of the play button and the position information. The destination of the source object information file may be either on the local computer or on the server, and is appropriately specified depending on the application.

Next, data relating the smart object information file to the processed HTML file (specifically, link information between the smart object information file and the processed HTML file, their paths and addresses, and Information that identifies the operator) in a predetermined database (no modification is made to the processed HTML file). The location of this database is preferably located on the same computer as the system because it is read by the system. By adopting this method, if the information that identifies the operator is recorded in the database, the contents written in the HTML file to be processed by this system can be displayed only to the person himself / herself. It is also possible to show the written content or show it to an unspecified number of opponents.

The operation panel is closed by closing the browser displaying the HTML file to be processed, and the program ends (step 4 16).

In the above, the case where the URL of the engine is specified in the HTML file to be processed has been described in advance. However, in addition to this, the method of adding (extending) this system to the browser function and the use of a proxy n

There is a method (adding functions to the HTTP daemon). In these cases, the system can be used without any modification to the ^ processed HTML file. 3. Third aspect

The third mode is a technique in which an audio file or a moving image file and a smart object information file are set (parts), and the parts are incorporated into a file image to be processed.

The method of creating the parts is basically the same as that of the first mode. The method of inputting object information while using existing audio and video files, and the method of inputting audio and video, There are two ways to enter project information. Figures 10 and 11 show simplified flowcharts. However, when creating parts, unlike the first mode, there is no need to input the HTML file to be processed, prepare a template HTML file, set it to the default setting, and set it as necessary You just have to make the changes.

After the parts are created (after the save button is pressed), the object information file and the audio file or the moving image file are saved in a linked state (as a set). In addition, the part index file {for example, the final still image of the part, the name and description of the part (memo)} displayed in the parts playback order display window and the parts list display window is also a component of this part (Fig. 2 8). Furthermore, ID information for identifying this part is also a component of this part.

In the following, each function, operation procedure, and internal processing in the system that incorporates various created parts into the processed HTML file Let me explain why.

(a) Description of each function

After the system is started and the HTML file to be processed is specified, parts can be incorporated. At that time, an operation panel is displayed on the screen. The operation panel mainly has a parts list display window and a parts playback order display window. Hereinafter, these will be described.

★ Parts list display window

In the parts list display window, a list of parts (index of parts) recorded in the database is displayed (see Fig. 28). In this window, the operator selects a part to be included in the processed H TML file image.

★ Part playback order display window

The part playback order designation window has a function of managing the entire time of the HTML file to be processed. When a part is placed on the processed H TML file image, the index of the part (for example, the final still image of the part) is displayed in the playback order specification window. Each time a part is placed, it is added to this part playback order designation window in order. However, by dragging the index of the displayed part, the order of the part in the part reproduction order designation window can be changed.

(b) Operation procedure

First, the operator inputs (selects) the HTML file to be processed. The input (selected) HTML file to be processed may be either on a local computer or on the Internet. Next, the parts list window on the displayed operation panel (Fig.

2)), select the part you want to paste on the processed H TML file image. Then, refer to the size of the part displayed in the rectangle, move the mouse cursor to the desired position so that the rectangle does not protrude from the screen, and press the left button of the mouse at the paste position. Repeat this operation for the parts you want to paste. Next, the display order of the parts is determined. At the stage where all parts have been pasted, the markers for each part are arranged in the parts playback order display window in the order of pasting. Therefore, the operator drags the index of the part whose order is to be changed, and moves the index to a desired order in the part reproduction order display window.

Finally, press the save button to complete the installation work.

(c) Description of internal processing

The internal processing for pasting parts will be described with reference to the flowchart of FIG.

First, the input HTML file to be processed is read and displayed by a browser. Then, the operation panel is displayed so as not to be exposed to the processed HTML file image.

In the operation panel, for the parts registered in the database, the index data of the parts to be displayed in the parts list display window {for example, the name and description of the parts (memo)}, the parts to be displayed in the playback order display window The index data (for example, final still image), the file information file, and the audio file-The file name of the video file is sent to the recording unit (step 641). Then, based on the information sent to the recording unit, a parts list is displayed on the display (see step 642, Fig. 28). Also, A window for specifying the playback order is also displayed (step 6 43). When a part is selected by the operator (step 644), the object information file of the selected part is read and the position information (X, Y coordinates) written in the object information file is read. The size (operation range) of the part is grasped based on the maximum value of (Step 645), and the size is displayed in a rectangle in accordance with the position of the mouse cursor (Step 646).

Next, when the left mouse button is pressed, the position of the base point of the part is obtained from the HTML file (step 648). Also, the index of this part (for example, the final still image) is added after the index of the currently placed part in the part playback order display window (step 649). Then, the information (ID) specifying the arranged parts, the base point position information, and the reproduction order information are sent to the recording unit. This process is repeated by the number of parts.

Next, if there is a change in the part playback order, the change is reflected in the part playback order display window (step 652), and sent to the changed playback order recording unit (step 651). ).

Then, when the save button is pressed, the files (audio files and video files) on the time axis are combined in the order of playback, and these files that have been separated for each part are combined (step 6). 5 5). Specifically, since the time information contained in the audio file and video file of each part is also different, the time information is also recalculated in the order of playback. Also, regarding the object information file, the data (position data) of the recording section is recalculated, and these files that have been separated for each part are synthesized (step 6554). Specifically, object information of multiple parts is Based on the base information of the packet, it is converted to the XY axis of the current HTML file, and the time information is converted to one time axis in the order of part playback. Finally, the HTML file to be processed is modified (step 6556), which is the same as in the first embodiment. Then, after the instruction to close the system, the displayed windows are all closed (step 658), and the system is terminated.

4. Usage example of this system

) Internet learning

The teacher (client) uses this system to add audio and video to the teaching materials created in HTML format and then uploads them on the server. (Note that the URL of the engine is specified in this file.) Has been). The student (client) browses the HTML file on the server. Then, by pressing the play button on the HTML, the specified engine is started, and based on the data written in the smart object information file, the operation of the smart object or the moving object is synchronized with the sound or video. Placement is performed. In this way, for example, teachers can receive explanations on a blackboard as many times as they want, regardless of time or physical distance.

(2) Writing on bulletin boards and H TML files

HTML files on the Internet can only be edited by the creator, so if a third party wants to add information, the main method is to write it on a bulletin board. However, by using this system, it is possible to add information (writing, image attachment) to the HTML file.

Means for using the system will be described. First, on the homepage The creator adds the URL of the engine to the HTML and reflects it on the home page. As a result, all users who read the HTML file with the engine URL specified by the browser can write to the HTML file. In this case, the HTML file and the URL of the linked object information file are also registered in the database read by the engine to be started, but by registering and managing the user information here, the content written It is also possible to make settings that allow an unspecified number of people to view the file, that only the person can view the file, and that only a specific person can view the file. In addition, by managing users, settings are made such that the written content can be viewed by an unspecified number of people, that only the user can view the content, and that only a specific person can view the content. It becomes possible.

(3) Product description

The number of stores that sell products on the web is increasing, but, of course, only by photos and explanations, the push is weaker than when clerks actually explain the products at the storefront, If you do not have a high willingness to purchase, you will not be able to purchase. Therefore, by using this system, the sales side can explain the points of the product etc. in detail with sound, so that the purchase motivation can be further increased.

(4) Manual, help

For example, if there is something that you do not understand by operating a video or operating a personal computer, until the problem is solved, the user searches for the place described in the instruction manual and further searches for the corresponding button. · · A lot of hard work is needed. However, by using this system, in the case of a simple explanation, the buttons are It is possible to explain by pointing directly to the screen, such as "Press here · · · ·" on the photo of the operating unit that is operating. Also, in the case of PC help, the help screen is usually opened in a separate window, and the screen is displayed and explained. However, by using this system, the help screen can be displayed on the user's screen without opening a separate window. Can be explained. In both cases, explanations can be made with sound on one screen, which makes it very easy to understand.

(5) Video, video description

Since it is possible to explain along with the video, websites that introduce movies and videos, sign language teaching materials, how to use the fingers of musical instruments, etc. However, using this system, it is possible to separately explain video and text, so that text and pictures can be displayed without disturbing the video and explained in detail. It becomes possible to do.

(6) Explanation of music teaching materials

Because explanations can be made according to audio and video, it is possible to explain while actually playing music and writing music.

Claims

The scope of the claims

1. calling a first file and displaying the image on a first screen;

Moving or arranging an image object to be added in a second screen virtually existing on the first screen while inputting a voice or a moving image;

Obtaining position information of the above-mentioned object on the second screen in association with time; creating an audio file or a moving image file based on the input audio information or moving image information;

Creating a smart object information file based on the information specifying the image smart object, the position information, and the time information;

Linking the first file with the audio file or the moving image file and the object information file; linking the input sound or the moving image while displaying the first file on the first screen; Moving or arranging the above objects over time in the second screen and displaying the objects.

And a method for superimposing and displaying additional information on a display screen.

2. Retrieve the first file and display the image on the first screen;

Moving or arranging the image object to be added in the second screen virtually existing on the first screen; and determining the position of the image object on the second screen. Obtaining information;

Based on the information for specifying the image object and the position information, Creating a smart object information file; and associating the first file with the smart object information file; and displaying the first file on the first screen while displaying the second file on the first screen. Displaying the above-mentioned image object in the display, wherein the additional information is superimposed on the display screen.

3. calling the first file and displaying the image on the first screen;

Displaying, at a predetermined position of the second screen virtually existing on the first screen, an image object to be added, which is linked to an audio file or a moving image file;

Obtaining position information of the image object on the second screen;

A step for specifying the output order of the audio file or the video file;

A step of displaying the image object linked to the audio file or the moving image file on the second screen while outputting the audio file or the moving image file in accordance with the specified output order.

4. The method specified in claims (1) to (3), without adding any link information to the first file.

5. moving or arranging an image object in the second screen, which is virtually present on the screen of the first screen, while inputting a voice or a moving image;

Obtaining position information of the image object on the second screen in association with time; 3g

An audio file based on the input audio or video. Creating a file or video file;

Creating an image object information file by associating the information for identifying the image object, the position information, and the time information with each other; Steps for associating with an audio file or the above-mentioned video file and the above-mentioned project information file

A method for creating a source file that superimposes and displays additional information on a display screen.

6. In a second screen virtually existing on the screen of the first screen, moving or arranging an image object to be added in the second screen;

Obtaining information on the position of the image object on the second screen;

Creating an object information file based on the information specifying the image object and the position information; associating the file related to the screen display of the second screen with the object information file; Steps

7. displaying an image object to be added, which is linked to the audio file or the moving image file, at a predetermined position of the second screen virtually existing on the second screen;

Obtaining position information of the image object on the second screen;

A switch that specifies the output order of the audio file or video file One ^

Tep

How to create a source file that superimposes additional information on the display screen.

8. The method defined in claims (5) to (7), wherein no link information is added to the file for the first screen.

9. A magnetic recording medium for storing a program for executing the steps defined in claims (1) to (8).

10. A computer system having the magnetic recording medium defined in claim (9).