JPH08320940A - Composite information processor - Google Patents

Abstract

PURPOSE: To receive integrated information from both of eyes and ears by allowing stroke input time and sound information recorded during the input period of the stroke to correspond to each other to generate composite information. CONSTITUTION: A packet data constituting part 3 annexes current time data inputted from a clock/counter 10 to stroke information inputted from a stroke input part 1. A writing processing part 4 outputs the stroke data with time information to a composite information file 5 together with voice data recorded at the time of inputting the stroke. The composite information recorded in the file 5 is read out by a reading processing part 6 and each time information- added data constituting the composite information are generated at the same time interval as recording time interval. Each time-added data generated at the generation time are transmitted to a stroke plotting part 8 and a sound reproducing part 9 and stroke plotting and voice reproduction are executed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a composite information processing apparatus for inputting both an image and a sound to generate composite information including the image and the sound, and an image and a sound based on the composite information. The present invention relates to a composite information processing device that outputs both

[0002]

2. Description of the Related Art Conventionally, visual information such as sentences, images and charts, and audio information such as speech, music and sound effects are
It has been handled by dividing it into each information mode. For example, visual information typified by paper media is distributed through communication channels such as physical mail, facsimile, electronic mail on LAN, telex, telegram, etc., and it is printed on paper or electronically or optically to users. Has been provided on the display screen. On the other hand, the voice information is
It has been transmitted and distributed mainly through communication channels for broadcasting such as TV and offline media such as CDs and tapes.

However, in recent years, a multimedia medium (or system) on a personal computer has begun to spread as a composite information, that is, a combination of both visual information and audio information which is digitally recorded, stored and reused. For example, in Apple Computer, software called Quick Time can play back audio and video stored on the Macintosh, and application software for making some videos has been released.

Particularly in recent years, a technique for accessing both visual information and audio information from a communication terminal on a network is being developed. In the device described in Japanese Patent Application Laid-Open No. 4-105114, a communication terminal capable of capturing image information in addition to voice with a scanner and bidirectionally operating with a pointing device such as a tablet has been reported.

Software for editing a simple animation with sound for presentation
Tools are also being developed.

[0006]

However, in the conventional multimedia communication, the image information and the audio information created by the user are separated and stored in relatively large units such as page units and sentence units. , Has been transmitted. Therefore, in e-mail with voice, etc., the voice can be inserted into the document, but the process of creating the image and the explanation of the voice cannot be associated with each other. Therefore, for example, when explaining directions using a map or explaining the operation of a machine using a blueprint, it is not clear where the audio information points in the image information. Is not fully effective.

The above-mentioned animation editing software
Some tools also have the function of finely controlling the image, but in order to use this, it is necessary to spend a lot of time performing detailed editing work, and the user can record in real time. It is completely unsuitable as a tool for communication. In view of the above circumstances, it is an object of the present invention to provide a composite information processing device suitable for a system that can receive unified information from both eyes and ears.

[0008]

Means for Solving the Problems A first composite information processing apparatus of the present invention that achieves the above object is: (1-1) Line drawing input operator for inputting a line drawing consisting of a plurality of strokes (1-2) Voice (1-3) Measuring means for measuring the input order or input time of each stroke input by the operation of the line drawing input operator (1-4) Each input by the operation of the line drawing input operator Each stroke information indicating a stroke, the input order or input time of each stroke measured by the above-mentioned measuring means, and each voice information indicating each voice recorded by the voice recording means during the input period of each stroke are mutually related. It is characterized in that it is provided with a composite information generating means for generating composite information associated with each other.

A second composite information processing apparatus of the present invention which achieves the above object is: (2-1) Line drawing means for drawing a line drawing composed of a plurality of strokes on a display screen (2-2) Sound producing means for producing voice (2-3) Each stroke information representing each stroke constituting the line drawing, the input order or input time of each stroke, and each voice information representing each voice corresponding to each stroke correspond to each other. While corresponding to each other based on the input order or input time configuring the composite information that is attached, while displaying each stroke based on each stroke information configuring the composite information on the display screen,
It is characterized by further comprising output control means for causing the sounding means to generate each sound based on each sound information forming the composite information.

Here, in the second composite information processing apparatus of the present invention, the output control means draws each stroke on the display screen at a drawing speed according to the duration of each voice corresponding to each stroke. It is preferably one. Alternatively, in the second composite information processing apparatus of the present invention, the output control means displays on the display screen corresponding to a sound being produced while each sound corresponding to each stroke is being produced by the sound producing means. It is also a preferable aspect that the stroke is displayed with an attribute different from the attributes of other strokes displayed on the display screen.

Here, the "input time" typically refers to the input start time of the stroke, but it does not necessarily have to be the input start time, but the input end time of the stroke or the input start of the stroke. A representative time corresponding to the stroke, such as an intermediate time between the time and the input end time, may be used. Also, the "input time" is
It may be an absolute time, but it is not limited to this. For example, the relative time based on the input start time of the first stroke in the series of strokes, or the input order of the series of strokes, and the immediately preceding input What kind of information can be used to understand the relationship between stroke input timings, such as the combination of the stroke input start time or input end time and the time difference between the stroke input start time input this time. May be.

The first composite information processing device of the present invention is a device in charge of input of composite information, and the second composite information processing device of the present invention is a device in charge of output of composite information. Alternatively, the first and second composite information processing devices may be integrated to form one composite information processing device that handles both input and output of composite information. However, it is not necessary to integrate them into one device, and the first composite information processing device and the second composite information processing device may be connected to each other via a communication line, or they may be installed independently, The composite information may be transmitted and received via a portable storage medium such as a floppy disk or MO.

[0013]

According to the first composite information processing apparatus of the present invention, the input order or input time of each stroke is measured, and each stroke, the input order or input time of each stroke, and each voice are associated with each other. Since the generated composite information is generated, in outputting information, it is possible to receive information in which the eyes and ears correspond to each other, and to realize a communication means with high expressiveness and comprehension.

The second composite information processing apparatus of the present invention outputs strokes and voices in correspondence with each other based on the input order or input time in the composite information, and the above-mentioned expression Communication with high power and understanding is realized. In the second composite information processing apparatus according to the present invention, the correspondence between the stroke and the voice is not limited to a specific correspondence, but, for example, at a drawing speed according to the duration of the voice, When the stroke corresponding to the voice is drawn on the display screen, it is immediately recognized that the stroke being drawn corresponds to the sound being produced. Alternatively, instead of responding at the drawing speed, for example, displaying a stroke corresponding to the sound being sounded with a different attribute from other strokes, such as displaying in a different color,
The stroke corresponding to the voice being sounded can be easily recognized also by displaying it thick and blinking the stroke.

In the present invention, since the stroke and the voice are associated with each other, even if the voice is input later than the start of the stroke at the time of input, the output start time of the stroke and the voice are output at the time of output. The output start time can be easily aligned with each other, and when inputting, it is not necessary to precisely manage the input time point of the corresponding voice assuming the output time, and an easy-to-use device or system can be realized.

[0016]

Embodiments of the present invention will be described below. Here, as an embodiment of the present invention, an apparatus for recording freehand drawing by a pointing device and commentary input by voice in synchronization will be described. FIG. 1 is a block diagram of an embodiment of a composite information processing apparatus including the configurations of both the first composite information processing apparatus of the present invention and the second composite information processing apparatus of the present invention.

A detailed description will be given later, and here,
First, with reference to FIG. 1, an outline of the entire embodiment will be described. The stroke input unit 1 is provided with a mouse (not shown), and by operating the mouse, stroke data representing each stroke forming a line drawing is input. Further, the voice input unit 2 records voice and inputs voice information.

The stroke information input from the stroke input unit 1 is attached with data of the current time obtained from the clock / counter 10 in the packet data construction unit 3. The stroke data with time information is output to the composite information file 5 by the writing processing unit 4 together with the voice data recorded when the stroke is input. The composite information file 5 is a sequence data in the form of a queue with drawing time data and voice data together with time data. The composite information recorded in the composite information file 5 is read by the read processing unit 6, and each data with time information forming the composite information is generated at the same time intervals as when recording. This is the time adjustment unit 7
Is performed by managing each data with time information while referring to the clock / counter 10. The data with each time information which is activated at the occurrence time is transmitted to the stroke drawing unit 8 or the voice reproduction unit 9.

FIG. 2 is a diagram showing a file structure of stroke data and voice data which composes the composite information file shown by blocks in FIG. First, the stroke data area for storing the input stroke data will be described. The packet index is a serial number indicating the order of occurrence of each stroke. This is also attached to the voice data packet, and is a serial number of two systems. The stroke drawing start time records the time when the stroke is input as the elapsed time from the start of the stroke input work. Regarding timing, it is assumed that the stroke input is generated by clicking the left mouse button. The route point list is a set of coordinate points forming a stroke, and the stroke point number is the number of coordinate points forming the stroke. The audio reference pointer designates the address (packet index) of audio data packets to be reproduced simultaneously. A null pointer is recorded if there is no corresponding voice.

Next, the voice data area will be described. The packet index is a serial number indicating the occurrence order of each audio data. The drawing reference pointer specifies the index of the stroke data packet to be displayed at the same time. A null pointer is recorded if there is no corresponding stroke. The byte length of the audio signal data is recorded in the audio data length, and the required time for audio reproduction is easily calculated from this. Although a compression means is not used in the present embodiment, when audio data compression is performed in order to efficiently use the storage capacity, a device is similarly configured by giving an audio duration in addition to this audio data length. .

Next, the operation of this embodiment will be described. FIG.
FIG. 9 is a flowchart showing a processing sequence for recording drawing data and audio data in association with each other in a composite information file. When the recording work is started (S21), the process is in a waiting state for mouse click and subsequent voice input. At the time when the first stroke data is input, that is, here, the first depression of the mouse button is used as the starting point of the recording operation to initialize the clock. All subsequent data entry will be recorded by the time elapsed from this origin. When the left mouse button is pressed (S22), the process proceeds to stroke coordinate point input processing (S23), in which a data set of a stroke start point and a path coordinate point sampled at fixed time intervals is obtained. Complete the input process. Completion of the stroke information collection process is performed by releasing the left mouse button (S24).

Simultaneously with the start of collecting stroke information, sampling of voice data is started in parallel. First, a data input waiting state (S25) is entered, and when a signal having a predetermined threshold intensity Ath or more is continuously input for a threshold period Tth1 or more, data sampling from an A / D converter (not shown) is started. Yes (S26). Regarding the end of voice sampling, similarly to the start, when the signal with the threshold intensity Ath or less continues for the threshold interval Tth2 or more, it is determined to be a silent interval, and the data sampling is ended (S).
27).

A data packet consisting of stroke data and voice data is written in the packet buffer 11 on the RAM (S29). When both the stroke data collection and the voice data collection are completed, the counter of the clock / counter 10 shown in FIG. 1 is incremented (S28). When the mouse is turned off before detecting the silent section and the silent section is not detected thereafter, even if the next stroke input, that is, the left mouse button is pressed, the pressing is invalidated.

A stroke input without voice (utterance) is expressed by referring to a voice data packet whose voice packet is empty. On the other hand, in order to express voice information without drawing,
When the right mouse button is pressed at the input waiting time (S25), the audio signal is sampled only during the time when the right button is pressed. This voice data packet is combined and stored to reference the empty stroke packet.

When stroke input or voice input is completed, the counter is incremented as described above (S
28), the stroke packet and the voice packet in the packet buffer 11 are output to the composite information file 5 (see FIG. 1) on the disk along with the occurrence time, using this counter value as the packet index (S29).

The process of inputting data and writing the input data to the composite information file 5 is repeated until an operation end signal is received (S30). This signal is generated by clicking the end button icon with the left mouse button. When the operation end signal is received, the end of file (EOF) is written in the composite information file 5 and the process ends (S31).

Next, a reproduction output mode for reproducing and outputting the composite information recorded in the composite information file 5 will be described. In the reproduction output mode of the composite information in this embodiment,
A real-time mode and a step mode are prepared. In the real-time mode, the elapsed time at the time of stroke recording is reproduced as it is, and in the step mode, drawing / audio is reproduced and output stepwise according to the reproduction order of stroke recording.

FIG. 4 is a flowchart showing reproduction in the real time mode. In real-time reproduction, each packet must be reproduced at the time input at the time of recording, so the clock is first initialized at the starting point of reproduction processing (S41). Reproduction of each packet thereafter proceeds by comparison with this clock.

First, a stroke packet is read from the composite information file 5 (S43), and then a voice packet corresponding to this packet is read (S44) and held in the packet buffer 10. Next, the stroke drawing start time described in the packet is compared with the clock,
It waits until the data transmission start time to the output device is reached (S45). At the start time, the data necessary for drawing and voice reproduction are transferred to the stroke drawing unit 8 and the voice reproduction unit 9.
(See FIG. 1), stroke drawing and voice reproduction are performed (S46, S47). Does this real-time reproduction detect the EOF indicating the end of the packet (S4
2) The process is repeated until an interrupt end signal from the user is received by a specific key (S48).

Next, the reproduction process in the step mode will be described with reference to FIG. As with real-time playback,
First, the packet data is fetched from the composite information file 5 and held in the packet buffer 11 (S52, S5).
3). Stroke drawing unit 8 for packet data by type
Then, it waits for a signal sent from the user to the voice reproducing unit 9 (see FIG. 1) and instructing the progress (S54). Instructions are made by key input or pressing the left mouse button. When the instruction is input, stroke drawing and voice reproduction are performed (S55, S56). The above steps are repeated until all the data of the packet is read from the composite information file 5 (S51) or until there is a stop instruction from the user (S57). The reception of the stop signal is the same as in the case of real-time reproduction.

FIG. 6 is a flowchart showing details of stroke drawing. In drawing a stroke, first, the voice data length and the stroke score are read from the packet buffer 11 (S61, see FIG. 2). 1) When the display mode (S63) is the drawing time control, the time interval Tin obtained by equally dividing the audio duration Ts.
The route points are connected at t 1 and drawn. Therefore, first, the voice data length Lv (byte) is calculated as the real time Ts (se
It is converted into c) (S62). This conversion can be easily obtained by the equation Ts = Lv / (N × Wb) using the sampling frequency N (Hz) of A / D conversion and the data width Wb. This voice duration is evenly divided by the number L of stroke path points (S64), and the obtained average value Tint = Ts /
A curve is formed while spline interpolating the route points with L as the drawing speed (S65).

2) When stroke special drawing is selected as the display mode, one or a combination of special attributes {blink, line width / line type change, color change} is set (S66), and the stroke is drawn. Do (S
67). The route point at this time is stored in the parameter buffer 70. When the voice output corresponding to the drawn stroke ends, the attribute of the same stroke as the drawn stroke is set to the standard value (S68), and the same stroke is read from the parameter buffer and drawn again (S69).

FIG. 7 is a diagram showing a state in which the composite information processing apparatus according to this embodiment is arranged on the network. The present composite information processing apparatus can be realized by using the high-performance workstation 70 or can be mounted on the portable information terminal 71. By communicating with each other through a LAN or a public line, it becomes possible to organically communicate the complex information, which is difficult even with only an image, only a sound, or an image with a simple sound as in the past.

Here, the above-described embodiment is a composite information processing having both the function of the first composite information processing apparatus according to the present invention and the function of the second composite information processing apparatus according to the present invention. Although it is an apparatus, the first composite information processing apparatus according to the present invention and the second composite information processing apparatus according to the present invention need not be integrated into one apparatus. For example, the workstation 70 shown in FIG. 7 is in charge of only inputting line drawing and voice (an example of the first composite information processing apparatus according to the present invention), and the portable information terminal 71 outputs only line drawing and voice. Person in charge (an example of the second composite information processing apparatus according to the present invention)
It may be.

[0035]

As described above, according to the present invention,
With simple operability, it is possible to record and transmit line drawing information and voice information that are finely synchronized with each other over time, and communication with higher expressiveness and comprehension is possible.

[Brief description of drawings]

FIG. 1 is a block diagram of an embodiment of the present invention.

FIG. 2 is a configuration diagram of a data packet.

FIG. 3 is a flowchart showing a recording process.

FIG. 4 is a flowchart showing a reproduction process in a real-time mode.

FIG. 5 is a flowchart showing a reproduction process in step mode.

FIG. 6 is a flowchart showing a detailed operation of stroke drawing.

FIG. 7 is a diagram showing a state in which a composite information processing device is arranged on a network.

[Explanation of symbols]

 1 Stroke Input Section 2 Voice Input Section 3 Packet Data Composition Section 4 Write Processing Section 5 Complex Information File 6 Read Processing Section 7 Time Adjustment Section 8 Stroke Drawing Section 9 Audio Playback Section 10 Clock / Counter 11 Packet Buffer 12 Parameter Buffer

Claims (4)

[Claims] 1. A line drawing input operator for inputting a line drawing consisting of a plurality of strokes, a voice recording means for recording a voice, and a measurement for measuring an input order or an input time of each stroke input by the operation of the line drawing input operator. Means, and stroke information representing each stroke input by the operation of the line drawing input operator, the input order or input time of each stroke measured by the measuring means, and the stroke during the input period of each stroke. A composite information processing apparatus comprising: composite information generation means for generating composite information in which each piece of voice information representing each voice recorded by the voice recording means is associated with each other. 2. A line drawing drawing means for drawing a line drawing composed of a plurality of strokes on a display screen, a sounding means for producing a sound, stroke information representing each stroke forming the line drawing, and stroke information for each stroke. The input order or input time and the voice information representing each voice corresponding to each stroke are associated with each other to form composite information, and the composite order is obtained while corresponding to each other based on the input order or the input time. The output control means displays each stroke based on each stroke information that constitutes information on the display screen, and causes each sound based on each voice information that constitutes the composite information to be sounded by the sounding means. A featured complex information processing device. 3. The output control means, at a drawing speed corresponding to a duration of each of the sounds corresponding to each of the strokes,
3. The composite information processing apparatus according to claim 2, wherein each stroke is drawn on the display screen. 4. The output control means, on the display screen, strokes on the display screen corresponding to the sound being sounded while the sounds corresponding to the strokes are being sounded by the sounding means. The composite information processing apparatus according to claim 2, wherein the composite information processing apparatus is displayed with an attribute different from the attributes of the other strokes displayed in (3).