JP3552338B2 - Complex information processing device - Google Patents

Description

[0001]
[Industrial applications]
The present invention relates to a composite information processing apparatus that inputs both an image and a sound to generate composite information including both the image and the sound, and a composite information that outputs both the image and the sound based on the composite information. It relates to a processing device.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, visual information such as texts, images, and charts, and audio information such as uttered voice, music, and sound effects have been exclusively handled in respective information modes. For example, visual information represented by paper media is distributed by physical mail, facsimile, communication routes such as e-mail over the LAN, telex, telegram, etc., and is available to users on paper or electronically or optically. Has been provided on the display screen. On the other hand, audio information has been transmitted and distributed mainly through communication channels such as broadcasting of telephones, radios and TVs, and off-line media such as CDs and tapes.
[0003]
However, in recent years, multimedia media (or systems) on personal computers has begun to spread as digital information that is combined with digital information, that is, both visual information and audio information, and stored and reused. For example, at Apple Computer, software called Quick Time can play audio and moving images stored on a Macintosh, and application software for creating several moving images has been released.
[0004]
In particular, recently, a technology for accessing both visual information and audio information from a communication terminal on a network has been developed. In the device described in Japanese Patent Application Laid-Open No. 4-105114, a communication terminal capable of capturing image information in addition to sound with a scanner and operating bidirectionally with a pointing device such as a tablet is reported.
[0005]
Software tools for editing simple animations with sound have also been developed for presentations.
[0006]
[Problems to be solved by the invention]
However, in conventional multimedia communication, image information and audio information created by a user have been separated, stored, and transmitted in relatively large units such as pages and sentences. For this reason, in an electronic mail with a voice or the like, it is possible to insert a voice into a document, but it is not possible to associate a process of creating an image with a description of the voice. Therefore, for example, when trying to explain a route using a map or explaining the operation of a machine using a design drawing, it is not clear where the audio information points in the image information. Is not fully effective.
[0007]
Some of the animation editing software tools mentioned above also provide a function to control the image in detail, but in order to use this, it is necessary to spend a lot of time performing detailed editing work, It is completely unsuitable as a tool for users to record and use in real time.
The present invention has been made in view of the above circumstances, and has as its object to provide a composite information processing apparatus suitable for a system capable of receiving uniform information from both eyes and ears.
[0008]
[Means for Solving the Problems]
A first composite information processing apparatus of the present invention that achieves the above object,
(1-1) Line drawing input operator for inputting a line drawing composed of a plurality of strokes (1-2) Voice recording means for recording voice (1-3) Input order of each stroke input by operating the line drawing input operator Or measuring means (1-4) for measuring an input time, stroke information representing each stroke input by operating a line drawing input operator, an input order or input time of each stroke measured by the measuring means, The information processing apparatus further includes a composite information generating unit configured to generate composite information in which audio information representing each voice recorded by the voice recording unit during the input period of each stroke is associated with each other.
[0009]
A second compound information processing apparatus of the present invention that achieves the above object is: (2-1) a line drawing drawing unit that draws a line drawing composed of a plurality of strokes on a display screen; (2-3) Each stroke information representing each stroke constituting the line drawing, the input order or input time of each stroke, and each voice information representing each voice corresponding to each stroke are associated with each other. Each stroke based on the stroke information constituting the composite information is displayed on the display screen while associating with each other based on the input order or the input time constituting the composite information. An output control means for causing each sound based on the sound information to sound by the sound means is provided.
[0010]
Here, in the second compound information processing apparatus according to the present invention, the output control means causes each stroke to be drawn on the display screen at a drawing speed corresponding to a duration of each sound corresponding to each stroke. Is preferred.
Alternatively, in the second compound information processing apparatus according to the present invention, the output control means may control the display screen corresponding to the sound being generated while each sound corresponding to each stroke is being generated by the sound generation means. It is also a preferred embodiment that the stroke is displayed with an attribute different from the attributes of other strokes displayed on the display screen.
[0011]
Here, the “input time” typically indicates the input start time of the stroke, but does not necessarily need to be the input start time, and the input end time of the stroke or the input start time of the stroke and the input Any representative time corresponding to the stroke, such as an intermediate time from the end time, may be used. The “input time” may be an absolute time, but is not limited thereto. For example, a relative time based on the input start time of the first stroke of a series of strokes, or a time of a series of strokes Information that indicates the relationship between strokes and the input timing of the stroke, such as the combination of the input order and the time difference between the input start time or the input end time of the stroke input immediately before and the input start time of the stroke input this time. Any type may be used.
[0012]
Note that the first composite information processing apparatus of the present invention is an apparatus in charge of inputting composite information, and the second composite information processing apparatus of the present invention is a device in charge of output of composite information. The first and second composite information processing apparatuses may be integrated to form one composite information processing apparatus that performs both input and output of composite information. However, it is not necessary to integrate them into one device, and the first composite information processing device and the second composite information processing device may be connected to each other via a communication line, or may be installed independently. The composite information may be transmitted and received via a portable storage medium such as a floppy disk or MO.
[0013]
[Action]
According to the first composite information processing apparatus of the present invention, the input order or input time of each stroke is measured, and each stroke, the input order or input time of each stroke, and each voice are associated with each other. Since the information is generated, it is possible to receive information corresponding to eyes and ears when outputting the information, thereby realizing a communication means with high expressiveness and understandability.
[0014]
The second composite information processing apparatus according to the present invention outputs strokes and voices in correspondence with each other based on the input order or input time in the composite information. Highly communicative communication is realized.
In the second compound information processing apparatus of the present invention, the correspondence between the stroke and the sound is not limited to a specific correspondence, but, for example, at a drawing speed corresponding to the duration of the sound, When the stroke corresponding to the sound is drawn on the display screen, it is immediately recognized that the stroke being drawn corresponds to the sound being sounded. Alternatively, instead of responding at the drawing speed, for example, displaying the stroke corresponding to the sound being pronounced with a different attribute from the other strokes, for example, displaying it in a different color, By blinking, the stroke corresponding to the sound being generated can be easily recognized.
[0015]
In the present invention, since the stroke and the voice are associated with each other, even when the voice is input later than the start of the stroke, for example, at the time of input, the output start time of the stroke and the voice output start time are output. It is not necessary to precisely control the input time of the corresponding sound at the time of input, assuming the time of output, and an easy-to-use device or system is realized.
[0016]
【Example】
Hereinafter, examples of the present invention will be described. Here, as an embodiment of the present invention, an apparatus for synchronizing and recording freehand drawing by a pointing device and commentary input by voice will be described.
FIG. 1 is a block diagram of an embodiment of a composite information processing apparatus including both the configuration of the first composite information processing apparatus of the present invention and the configuration of the second composite information processing apparatus of the present invention.
[0017]
The detailed explanation will be described later, and here, the outline of the entire embodiment will be described first with reference to FIG. The stroke input unit 1 is provided with a mouse (not shown), and by operating the mouse, stroke data representing each stroke constituting the line drawing is input. In the voice input unit 2, voice is recorded and voice information is input.
[0018]
The data of the current time obtained from the clock / counter 10 in the packet data forming unit 3 is attached to the stroke information input from the stroke input unit 1. The stroke data with time information is output to the composite information file 5 by the writing processing unit 4 together with the voice data recorded when the stroke is input. The composite information file 5 is a file in which drawing stroke data and audio data are converted into queue-format sequence data with time data. The composite information recorded in the composite information file 5 is read by the read processing unit 6, and the data with time information constituting the composite information is generated at the same time interval as that at the time of recording. This is performed by the time adjusting unit 7 managing the data with the time information while referring to the clock / counter 10. The data with each time information activated at the occurrence time is transmitted to the stroke drawing unit 8 or the sound reproduction unit 9.
[0019]
FIG. 2 is a diagram showing a file structure of stroke data and audio data, which constitute the composite information file indicated by the blocks in FIG.
First, the stroke data area for storing the input stroke data will be described.
The packet index is a sequential number indicating the order of occurrence of each stroke. This is similarly attached to the voice data packet, and has two serial numbers. The stroke drawing start time records the time at which the input of the stroke has occurred as the elapsed time from the start of the stroke input operation. Regarding timing, it is assumed that the input of a stroke has occurred by clicking the left mouse button. The path point list is a set of coordinate points forming a stroke, and the number of stroke points is the number of coordinate points forming the stroke. The audio reference pointer specifies an address (packet index) of an audio data packet to be reproduced at the same time. If there is no corresponding voice, a null pointer is recorded.
[0020]
Next, the audio data area will be described.
The packet index is a sequential number indicating the order of occurrence of each audio data. The drawing reference pointer designates the index of the stroke data packet to be displayed at the same time. If there is no corresponding stroke, a null pointer is recorded. The audio data length records the byte length of the audio signal data, and the time required for audio reproduction can be easily calculated from this. Although no compression means is used in the present embodiment, if audio data compression is performed for efficient use of storage capacity, a device is similarly configured by giving an audio duration in addition to the audio data length. .
[0021]
Next, the operation of this embodiment will be described.
FIG. 3 is a flowchart showing a processing order for recording drawing data and audio data in the composite information file in association with each other.
When the recording operation is started (S21), the apparatus is in a state of waiting for a mouse click and a subsequent voice input. The clock is initialized when the first stroke data input, that is, here, the first depression of the mouse button is the starting point of the recording operation. All subsequent data inputs are recorded by the elapsed time from this starting point. When the left mouse button is pressed (S22), the process proceeds to a stroke coordinate point input process (S23) where a data set of a stroke start point and a path coordinate point sampled at fixed time intervals is obtained. Complete the input process. The stroke information collection process ends when the left mouse button is released (S24).
[0022]
Sampling of audio data is started in parallel with the start of stroke information collection. First, a data input wait state (S25) is entered, and when a signal having a predetermined threshold strength Ath or more is continuously input for a threshold section Tth1 or more, data sampling from an A / D converter (not shown) is started. (S26). As to the end of the voice sampling, similarly to the start, when a signal having the threshold strength Ath or less continues for the threshold section Tth2 or more, it is determined that the section is a silent section, and the data sampling ends (S27).
[0023]
The data packet including the stroke data and the voice data is written to the packet buffer 11 on the RAM (S29). When both the stroke data collection and the voice data collection are completed, the counter of the clock / counter 10 shown in FIG. 1 is incremented (S28). The mouse is turned off before the silent section is detected, and at the time when the silent section is not yet detected, even if the next stroke input, that is, the left mouse button is pressed, the pressing is invalidated.
[0024]
A stroke input without voice (utterance) is expressed by referring to a voice data packet whose voice data is empty. On the other hand, if the right mouse button is pressed at the time of waiting for input (S25) in order to represent audio information without drawing, the audio signal is sampled only while the right button is pressed. The audio data packets are stored in combination to refer to empty stroke packets.
[0025]
When the stroke input or the voice input ends, the counter is incremented as described above (S28), and the stroke packet and the voice packet in the packet buffer 11 use this counter value as a packet index, and generate a composite packet on the disk together with the occurrence time. The information is output to the information file 5 (see FIG. 1) (S29).
[0026]
The process of inputting data and writing the input data to the composite information file 5 is repeated until an operation end signal is received (S30). This signal is generated by clicking the end button icon with the left mouse button. When the operation end signal is received, the end of the file (EOF) is written into the composite information file 5 and the process ends (S31).
[0027]
Next, a reproduction output mode for reproducing and outputting the composite information recorded in the composite information file 5 will be described.
A real-time mode and a step mode are prepared as the reproduction output mode of the composite information in the present embodiment. In the real-time mode, the time lapse at the time of stroke recording is reproduced as it is. , And reproduce / output the drawing / voice in a step manner.
[0028]
FIG. 4 is a flowchart showing the reproduction in the real-time mode.
In real-time reproduction, since each packet must be reproduced at the time input at the time of recording, first, the clock is initialized at the time when the reproduction process starts (S41). Subsequent packet reproduction proceeds by comparison with this clock.
[0029]
First, a stroke packet is read from the composite information file 5 (S43), and subsequently, an audio packet corresponding to this packet is read (S44) and held in the packet buffer 10. Next, the stroke drawing start time described in the packet is compared with the clock, and the process waits until the data transmission start time to the output device is reached (S45). At the start time, data necessary for drawing and sound reproduction is transmitted to the stroke drawing unit 8 and the sound reproducing unit 9 (see FIG. 1), and stroke drawing and sound reproduction are performed (S46, S47). This real-time reproduction is repeated until an EOF indicating the end of the packet is detected (S42) or an interrupt end signal from the user is received by a specific key (S48).
[0030]
Next, the reproduction process in the step mode will be described with reference to FIG.
As in the case of the real-time reproduction, first, the packet data is fetched from the composite information file 5 and held in the packet buffer 11 (S52, S53). The packet data is transmitted to the stroke drawing unit 8 and the sound reproducing unit 9 (see FIG. 1) for each type, and waits for a signal for instructing the user to proceed (S54). The instruction is made by key input or pressing the left mouse button. When the instruction is input, stroke drawing and sound reproduction are performed (S55, S56). The above steps are repeated until all the data of the packet is read from the composite information file 5 (S51) or until the user gives a stop instruction (S57). The reception of the stop signal is the same as in the case of the real-time reproduction.
[0031]
FIG. 6 is a flowchart showing details of stroke drawing.
In drawing a stroke, first, the audio data length and the number of stroke points are read from the packet buffer 11 (S61, see FIG. 2).
1) When the display mode (S63) is the drawing time control, drawing is performed by connecting the path points at time intervals Tint obtained by equally dividing the duration Ts of the sound. For that purpose, first, the audio data length Lv (byte) is converted into real time Ts (sec) (S62). This conversion can be easily obtained by the equation Ts = Lv / (N × Wb) using the sampling frequency N (Hz) of the A / D conversion and the data width Wb. This voice duration is equally divided by the number L of stroke path points (S64), and a curve is formed while the path points are spline-interpolated using the calculated average value Tint = Ts / L as the drawing speed (S65).
[0032]
2) When the special drawing of the stroke is selected as the display mode, one or a combination of the special attributes {blink, line width / line type change, color change} is set (S66), and the stroke is drawn (S67). ). The path point at this time is stored in the parameter buffer 70. When the audio output corresponding to the drawn stroke is completed, the same stroke attribute as the drawn stroke is set to a standard value (S68), and the same stroke is read from the parameter buffer and drawn again (S69).
[0033]
FIG. 7 is a diagram illustrating a state in which the composite information processing apparatus according to the present embodiment is arranged on a network.
The present composite information processing apparatus can be realized by using a high-function workstation 70 or can be mounted on a portable information terminal 71. By communicating through a LAN or a public line, it becomes possible to perform organic communication of complex information, which has been difficult even with only an image or only a sound, or even a simple image with sound, as in the related art.
[0034]
Here, the above embodiment is a composite information processing apparatus having both the function of the first composite information processing apparatus according to the present invention and the function of the second composite information processing apparatus according to the present invention. However, the first composite information processing device according to the present invention and the second composite information processing device according to the present invention need not be integrated into one device. For example, the workstation 70 shown in FIG. 7 is in charge of inputting only a line drawing and an audio (an example of the first composite information processing apparatus according to the present invention), and the portable information terminal 71 only outputs the line drawing and the audio. The person in charge (an example of the second compound information processing apparatus according to the present invention) may be used.
[0035]
【The invention's effect】
As described above, according to the present invention, under simple operability, it is possible to record and transmit line drawing information and audio information that are finely synchronized with time, and have higher expressiveness and understandability. Communication becomes possible.
[Brief description of the drawings]
FIG. 1 is a block diagram of an embodiment of the present invention.
FIG. 2 is a configuration diagram of a data packet.
FIG. 3 is a flowchart illustrating a recording process.
FIG. 4 is a flowchart showing a reproduction process in a real-time mode.
FIG. 5 is a flowchart showing a reproduction process in a step mode.
FIG. 6 is a flowchart illustrating a detailed operation of stroke drawing.
FIG. 7 is a diagram illustrating a state in which the composite information processing apparatus is arranged on a network.
[Explanation of symbols]
REFERENCE SIGNS LIST 1 stroke input unit 2 voice input unit 3 packet data configuration unit 4 write-out processing unit 5 composite information file 6 read processing unit 7 time adjustment unit 8 stroke drawing unit 9 audio reproduction unit 10 clock / counter 11 packet buffer 12 parameter buffer

Claims (4)

A line drawing input operator for inputting a line drawing composed of a plurality of strokes,
Voice recording means for recording voice,
Measuring means for measuring the input order or input time of each stroke input by the operation of the line drawing input operator; and each stroke information representing each stroke input by the operation of the line drawing input operator, and the measuring means Generates composite information in which the measured input order or input time of each stroke and each piece of voice information representing each voice recorded by the voice recording means during the input period of each stroke are associated with each other. A composite information processing apparatus comprising composite information generating means. A line drawing drawing means for drawing a line drawing composed of a plurality of strokes on a display screen;
Each stroke information representing each stroke constituting a line drawing, the input order or input time of each stroke, and each voice information representing each voice corresponding to each stroke are associated with each other. While corresponding to each other based on the input order or the input time constituting the composite information, each stroke based on the stroke information constituting the composite information is displayed on the display screen, and the composite information is displayed. A composite information processing apparatus, comprising: output control means for causing each of the sounds based on the respective sound information to sound by the sounding means. 3. The composite information according to claim 2, wherein the output control means draws each stroke on the display screen at a drawing speed corresponding to a duration of each sound corresponding to each stroke. Processing equipment. The output control means displays, on the display screen, a stroke on the display screen corresponding to the sound being sounded while the sound corresponding to each stroke is being sounded by the sounding means. 3. The composite information processing apparatus according to claim 2, wherein the display is performed with an attribute different from an attribute of another stroke.

Images