JP2006172304A - Multimedia memorandum system and method for recording and reproducing multimedia memorandum - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To dissolve burden of tagging when multimedia data to be referred later to look back on a user himself/herself in the past are stored. <P>SOLUTION: This multimedia memorandum system 10 has a pen position data input part 12 by which an input position of a pen is acquired, a stroke information storage part 15 which stores information by linking input position data of the pen to time information as stroke information, a multimedia data storage part 14 which stores the inputted multimedia data by linking it to the time information, an input time computation part 17 which reads the stroke information from the stoke information storage part 15 and computes input time of the pen and a multimedia data retrieval part 16 which retrieves multimedia data corresponding to the computed input time from the multimedia data storage part 14. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an apparatus and method for accumulating and processing multimedia data such as video and audio, and in particular, a user can easily refer to past data in order to look back on the user's own experience later. The present invention relates to a multimedia memo device and method.

  When taking notes (note taking) or taking notes for memorandum, if you can record the surrounding situation that occurred at that time together, Will help. Therefore, a method has been proposed in which surrounding information is recorded together with note-taking, and data corresponding to the writing of the note is reproduced when looking back later. As an example of such a proposal regarding sound, there is a research example called “Audio Notebook” (Non-Patent Document 1).

  By the way, the user records his / her actions as multimedia data using an audio recorder, a video camera, etc., and the multimedia data is recorded later (for example, hours, days, months, or years later). ) By playing, the user can look back on the behavior and situation at the time of recording. However, it is difficult to obtain necessary information from a large amount of multimedia data without any clue. Therefore, for easy reference later, it is necessary to tag the multimedia data when recording the multimedia data. However, it is bothersome to tag each time during a conversation, often forgetting to tag, and later searching for places that you forgot to tag. Is difficult.

For this purpose, it is considered to use an action actively performed by a user such as note-taking as a tag so that recorded multimedia data can be referred to later based on such a tag. . However, in order to refer to and remember later, it is necessary to take notes firmly, and the user may not be able to concentrate on conversation or work.
Lisa Stifelman, Barry Arons, Chris Schmandt; "The Audio Notebook," Proc. Of CHI, Vol. 3, No. 1, pp. 182-189, ACM CHI 2001

  In this way, when storing multimedia data such as video / audio that will be referred to later, it is necessary to tag the multimedia data for searching. It is quite complicated. Although it is considered that tagging is performed by using note-taking that is actively performed by the user, it is still a burden on the user.

  Therefore, an object of the present invention is a multimedia memo device used for storing multimedia data to be referred to at a later date in order for the user to look back on his / her own experience, and for the purpose of referring to the act actively performed by the user. It is an object of the present invention to provide a multimedia memo device that has a small burden on a user for tagging, and a recording method and a reproduction method in such a multimedia memo device.

  A first multimedia memo device according to the present invention includes a pen position data input unit that can acquire a pen input position, an image output unit that graphically displays pen position data on a display screen, and pen input position data. A stroke information storage unit for storing information linked to time information as stroke information, a multimedia data input unit for inputting multimedia data, and multimedia data for linking and storing input multimedia data with time information And a storage unit.

  The second multimedia memo device of the present invention reads stroke information obtained by linking pen input position data with time information from a stroke information storage unit, calculates an input time of the pen, time information, Multimedia data retrieval unit for retrieving multimedia data corresponding to the determined input time from the multimedia data storage unit for storing the linked multimedia data, and multimedia data for reproducing the retrieved multimedia data In the initial state, the information written by the user is displayed on the screen as an input history area as it is, and the input time is determined by the user operating the input history area. Media data is retrieved and played.

  The multimedia recording method of the present invention includes a step of acquiring a pen input position in response to a pen input, a step of graphically displaying pen position data on a display screen, and a link of the pen input position data with time information. Storing the obtained information as stroke information in the stroke information storage unit and linking the input multimedia data with the time information and storing it in the multimedia data storage unit.

  In the initial state, the multimedia memo playback method of the present invention displays the information written by the user as it is as the input history area on the screen, and when the user operates the input history area, from the stroke information storage unit, Read stroke information by linking pen input position data with time information, determine pen input time, and input from indexed multimedia data storage unit that stores multimedia data linked to time information Searching for multimedia data corresponding to time, and playing back the searched multimedia data.

  In the present invention, “note writing” as an active active action of the user is used as a tag to the multimedia data, so that the user has to bother to perform a tagging action for later reference. Is missing.

  Since the action of “memo writing” is performed by the user himself, there is an advantage that it is easy to remember as a tag when referring to it later. Therefore, paying attention to the memory of this action, shooting the hand that the user is aware of as multimedia data from the position of the line of sight, playing back the multimedia data, evoking the tag of storage for the information that the user wants to remember It became the mechanism which becomes. This chain makes it possible to reach the information required by the user himself / herself even if he / she has written an appropriate memo or only thinks of vague information.

  According to the present invention, even when talking with the other party or when there is work to be immersed in, if you leave a simple note, you can play back the situation information later by referring to the note. In order to be able to do so, it is possible to concentrate on conversation and work without worrying about the memo on the spot.

  Next, a preferred embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 shows the overall configuration of a multimedia memo device according to an embodiment of the present invention. The multimedia memo device 10 includes a main body part in which a pen input unit 101 that is input by a user's pen 110 or a stylus and a screen unit (display unit) 102 that displays an image or the like to the user are integrated. A camera 103 and a microphone 104 connected to each other. The pen input unit 101 is made of a transparent member and is provided on the front surface of the screen unit 102. When an input is made to the pen input unit 101 with the pen 110, the history data of strokes by the pen 110 is displayed on the screen unit 102. Therefore, the user can freely draw a figure or a handwritten character on the screen unit 102 using the pen 110.

  The camera 103 has an optical axis in a direction perpendicular to the screen unit 102 and is arranged so that the shooting center is the center of the screen unit 102. The camera 103 shoots the state of the screen unit 102 as an image. At this time, the microphone 104 records ambient sounds. It is also possible to mount a plurality of cameras. For example, the camera 105 can be attached to the main body and a surrounding image can be taken separately.

  FIG. 2 shows the internal configuration of such a multimedia memo device 10. As shown in FIG. 2, a multimedia data input unit 11 for inputting video from the cameras 103 and 105 and sound from the microphone 104, and pen position data input for acquiring the pen input position at the pen input unit 101. Unit 12, a time information output unit (that is, a built-in clock) 13 that generates and outputs time information, a multimedia data storage unit 14 that stores multimedia data, a stroke information storage unit 15 that stores stroke information, Based on the stroke information stored in the stroke information storage unit 15, the input time calculation unit 17 that calculates the time (input time) at which the pen was input is stored in the multimedia data storage unit 14 based on the determined time. A multimedia data search unit 16 for searching for the searched multimedia data; A multimedia data playback unit 18 for playing back media data, a history data playback unit 19 for playing back history data of strokes based on the stroke information stored in the stroke information storage unit 15, and among the played back multimedia data An audio output unit 20 that outputs an audio part to a speaker or the like, and a video output unit 21 that outputs an image part of the reproduced multimedia data to a display or the like are provided.

  The multimedia memo device 10 can be broadly divided into a portion related to recording of multimedia data (recording device) and a portion related to reproduction of recorded multimedia data (playback device). Since the multimedia data storage unit 14 and the stroke information storage unit 15 are portions where data is stored, it can be considered that the multimedia data storage unit 14 and the stroke information storage unit 15 belong to both the recording device and the playback device. The position data input unit 12 and the time information output unit 13 belong to a recording device, and are a multimedia data search unit 16, an input time calculation unit 17, a multimedia data playback unit 18, a history data playback unit 19, an audio output unit 20, and a video. The output unit 21 belongs to the playback device. Although not explicitly shown in FIG. 2, the pen position data is graphically displayed on the screen unit 102 shown in FIG. 1 when the multimedia data is recorded, that is, the pen input is displayed as a stroke as it is. When the multimedia data is reproduced, the information written by the user is displayed on the screen unit 102 as it is as an input history area.

  In the multimedia memo device 10, on the recording device side, the input position data determined by the pen position data input unit 12 is linked to the time information from the time information output unit 13 and is used as stroke information as a stroke information storage unit. 15 is stored. Similarly, multimedia data input to the multimedia data input unit 11 is also stored in the multimedia data storage unit 14 in a form linked to time information. FIG. 3 shows the data structure of each data in the stroke information storage unit 15 and the multimedia data storage unit 14. The stroke information represents the position coordinates of the pen 110 and the writing time when the user inputs, and the position coordinates (Xi, Yi) (input stroke data) of the pen and time data (ti) are linked. Is. In the stroke information storage unit 15, the position coordinates of the pen and the writing time are structured together as stroke information and managed in ascending order of writing. On the other hand, the multimedia data is managed by the multimedia data storage unit 14 with a time code indicating the recording time of the multimedia data.

  Next, the operation of this multimedia memo device will be described. Since this multimedia memo device has a function of recording multimedia data and a function of playing back the recorded multimedia data, the operation when recording the multimedia data is referred to as a recording mode. When the operation of reproducing media data is performed, it is called a reproduction mode.

  In the recording mode, when video from the camera or sound from the microphone is input to the multimedia data input unit 11, the multimedia data input unit 11 attaches a time code based on the time information output from the time information output unit 13. The multimedia data is stored in the multimedia data storage unit 14. On the other hand, the pen position data input unit 12 acquires the pen input position, links the pen position data and the time, and stores them in the stroke storage unit 15 as the above-described stroke information.

  As described above, the multimedia data and the stroke information are stored in the multimedia data storage unit 14 and the stroke information storage unit 15, respectively. In the playback mode, the information written by the user is displayed on the screen as an input history area. In addition, the input time calculation unit 17 calculates the input time from the stroke information storage unit 15 when the user touches or traces the input history area on the screen with a pen. The multimedia data search unit 16 searches the multimedia data storage unit 14 for multimedia data having the time code indicated by the determined time. The retrieved multimedia data is reproduced and output via the multimedia data reproducing unit 18, the audio output unit 20, and the video output unit 21. Based on the stroke information, the history data reproducing unit 19 reproduces the history data of the stroke previously input by the user, and the reproduced stroke is sent to the video output unit 21, and as a result, the stroke input by the user first. Is displayed on the screen.

  FIG. 4 shows a multimedia data playback procedure in the playback mode. First, in step 151, stroke data by pen input is displayed on the screen. When the playback mode is started, the final screen in the recording mode is displayed. In step 152, when the user touches or traces an arbitrary point on the screen (for example, (Xm, Ym)), in step 153, the same data as the combination of the X coordinate and Y coordinate of the place is obtained. A search is performed to determine whether or not the stroke information storage unit 15 is present. As a result, if there is no such data, the process proceeds to step 154 to stop the reproduction of the multimedia data, and the process returns to step 151.

  On the other hand, if there is a corresponding combination of X and Y coordinates in the stroke information storage unit 15 in step 153, the stroke writing time of the combination is extracted in step 155, and the time is set to In in step 156. The multimedia data set as a point (starting point) is searched, and in step 157, the searched multimedia data is reproduced. If the user does not do anything during the reproduction and the multimedia data is reproduced up to the final frame, the process proceeds to step 154 to end the reproduction, returns to step 151, and again displays the stroke display of the final screen in the recording mode. I do. On the other hand, if the user touches or traces the screen with a pen during playback, the process returns to step 152 and is the same as the coordinates touched or traced in the same manner as when touched or traced by the user. When there is a combination of coordinates in the stroke information storage unit 15, multimedia data corresponding to the input time of the coordinates is reproduced. In addition, when the user clicks a point where there is no corresponding coordinate value during reproduction of the multimedia data, a stroke display corresponding to the time at which the reproduction multimedia data is currently reproduced is displayed.

  In this way, according to the multimedia memo device of the present embodiment, it becomes possible to perform past information operations using writing history data.

  Furthermore, the multimedia memo device can be improved in usability by devising a display method for the user.

  For example, in the recording mode, as an automatic display erasing function, display information (stroke data) is erased after a certain period of time has elapsed since the time when data written by the user was written, and a new blank area is created. Can be provided. In this case, the displayed stroke data is gradually displayed lightly according to the elapsed time from the input.

  FIG. 5 shows a display example when the automatic display erasing function is executed. The default density of the color used to display the stroke data is a numerical value divided by the time from the time when the stroke was written, and after a certain time has elapsed, the color density is set to zero, and as time passes, The display becomes lighter and a new blank area is provided. Even in the data recorded in this way, in the playback mode, it is possible to refer to the stroke data by repeating the playback and stop from the stroke data of the final screen, and the user can obtain the necessary information. Can be earned.

  In the playback mode, the multimedia data searched by the multimedia data search unit 16 is overlaid on the screen. When the playback is stopped, the automatic display erasing function at the time when the playback is stopped is performed as described above. The writing (stroke data) information before the display is erased may be presented on the display screen.

  The multimedia memo device of the present embodiment described above can also be realized by reading a computer program for realizing it into a computer such as a tablet computer or a personal digital assistant (PDA) and executing the program. . Such a program is read into a computer by a recording medium such as a CD-ROM or via a network. Such a computer generally has a CPU, a hard disk device for storing programs and data, a main memory, an input device such as a keyboard and a mouse, a display device such as a liquid crystal display, and a recording such as a CD-ROM. It is composed of a reading device for reading a medium and a communication interface used for connection with a camera or a microphone. In this computer, a recording medium storing the above-described program is loaded into a reading device, the program is read from the recording medium and stored in the hard disk device, or the program is stored in the hard disk device via a network, and then the hard disk device. When the CPU executes the program stored in the above, it functions as the multimedia memo device 10 described above.

1 is a perspective view showing an overall configuration of a multimedia memo device according to an embodiment of the present invention. It is a block diagram which shows the internal structure of a multimedia memo apparatus. It is a figure which shows typically the data structure in a stroke information storage part and a multimedia data storage part. It is a flowchart which shows the procedure of the multimedia data reproduction | regeneration processing in reproduction | regeneration mode. It is a figure which shows the example of a display in which the written information disappears from a display screen with progress of time.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Multimedia memo apparatus 11 Multimedia data input part 12 Pen position data input part 13 Time information output part 14 Multimedia data storage part 15 Stroke information storage part 16 Multimedia data search part 17 Input time calculation part 18 Multimedia data reproduction part 19 history data reproduction unit 20 audio output unit 21 video output unit 101 pen input unit 102 screen unit 103, 105 camera 104 microphone 110 pen

Claims (10)

A pen position data input unit that can acquire the pen input position;
An image output unit that graphically displays the pen position data on the display screen;
A stroke information storage unit that stores information obtained by linking the input position data of the pen with time information as stroke information;
A multimedia data input unit for inputting multimedia data;
A multimedia data storage unit for storing input multimedia data linked to time information;
A multimedia memo device.   2. The multi of claim 1, wherein when there is an input with the pen, the display density of the pen position data decreases as time passes from the input, and a new blank area is provided on the display screen. Media memo device. From the stroke information storage unit, the stroke information obtained by linking the input position data of the pen with the time information is read, and the input time calculation unit that calculates the input time of the pen;
A multimedia data search unit for searching multimedia data corresponding to the determined input time from a multimedia data storage unit for storing multimedia data linked to time information;
A multimedia data playback unit for playing back the searched multimedia data;
Have
In the initial state, information written by the user is displayed on the screen as an input history area as it is, and when the user operates the input history area, the input time is determined and corresponding multimedia data is searched. A multimedia memo device to be played.   The retrieved multimedia data is overlaid on the screen, and when the reproduction is stopped, the stroke data is reproduced from the stroke information read from the stroke information storage unit and the stroke data is presented on the screen. 4. The multimedia memo device according to 3. Acquiring a pen input position according to the pen input; and
Graphically displaying pen position data on the display screen;
Storing information obtained by linking the input position data of the pen with time information in a stroke information storage unit as stroke information;
Linking the input multimedia data with time information and storing it in the multimedia data storage unit;
A method for recording multimedia memos.   The multi of claim 5, wherein when there is an input by the pen, the display density of the pen position data becomes lighter as time elapses from the input, and a new blank area is provided on the display screen. Media memo recording method. In the initial state, the information written by the user is displayed on the screen as the input history area as it is,
When the user performs an operation on the input history area, reading stroke information obtained by linking pen input position data with time information from a stroke information storage unit, and determining an input time of the pen;
Searching multimedia data corresponding to the determined input time from a multimedia data storage unit storing multimedia data linked with time information;
Playing the retrieved multimedia data; and
A multimedia memo playback method comprising:   The retrieved multimedia data is overlaid on the screen, and when the reproduction is stopped, the stroke data is reproduced from the stroke information read from the stroke information storage unit and the stroke data is presented on the screen. 8. The multimedia memo playback method according to 7.   The program which makes a computer perform the multimedia memo recording method of Claim 5 or 6.   A program for causing a computer to execute the multimedia memo reproduction method according to claim 7 or 8.

Images