JPH02288600A - Information processing system - Google Patents

Abstract

PURPOSE:To inform the location of an operation object to the operator in voice and to enhance the presence acoustically by providing a means outputting a stereosonic sound recorded and synthesized. CONSTITUTION:The information processing system is provided with a sound input means 9, an A/D converter 20, D/A converters 21-23 converting recorded and synthesized discrete acoustic data into an analog sound signal, acoustic output means 10-12 such as a speaker, a signal processing processor 28 and an acoustic data processing means 24. Not only the reproduction of the sound of stereo recording but also the sound collected by a single microphone and the synthesized sound is controlled for its volume and output point of time to attain a simulated stereosonic sound and the resulting sound is outputted by using plural acoustic output devices 10-12. Thus, the sentiment and the presence are enhanced so as to progress the understanding of the content and image for a document or a catalog or the like expressed in multi-medium displayed on a display means 1 and the location of an object desired for operation or notice is informed in sound by utilizing the stereosonic audible distance sense. Thus, the system with more convenience of use is realized.

Description

[Detailed description of the invention] [Industrial application field]

The present invention relates to an information processing system that processes multimedia documents and the like containing audio information. [Prior art] For example, Cypuis' Voice Station (Robe)
rt T, N1cholson, “Integr.
Ating Voicein the office
Word”, BYTE Ilagazine, 1
983. (December issue) system is capable of adding audio annotations to 2 editable documents. This system displays the content of the voice message as a bar graph in a window separate from the document, and displays the length of the voice message. The sound parts and silent parts of the voice message are visually displayed. Then, the user edits the audio using soft keys displayed in the operation menu field common to the system. The soft keys are provided with the same operating commands as on a tape recorder. in particular. record”, “5top”, “play”, +re
"wind" and "fast forward".Furthermore, a number is displayed for each audio sentence in this window, and you can insert, delete, and record additional audio messages for each audio sentence by specifying this number using the keys. In addition, an icon representing an abstracted speaker is marked at the place where the audio annotation is added in the two documents, and when this icon is pointed, the above-mentioned window opens.

[Problem to be solved by the invention]

However, the traditional voice station system. It does not have a means to enhance the sense of presence acoustically or to notify the operator of the position of the operation target through sound. The system does not take full advantage of the quality of the sound. The object of the present invention is to solve these problems of the prior art.

[Means to solve the problem]

In order to solve the above problems, the information processing system of the present invention includes means for outputting recorded and/or synthesized stereo sound. It goes without saying that it is possible to playback sound recorded in stereo by installing multiple microphones, but also to control the volume and output point of sound recorded with a single microphone or synthesized sound to create pseudo-stereo, and to use multiple sound output devices. A means for outputting is provided.

[Effect]

In the information processing system of the present invention, it is possible to enhance the emotion and sense of presence through stereo playback, and facilitate understanding of the contents and images of documents, catalogs, etc. expressed in multimedia displayed on the display means. Using the auditory sense of distance, it is now possible to use sound to notify the location of the object you want to operate or focus on, taking advantage of the goodness of sound. A more user-friendly system can be realized.

Example 1 The present invention will be described below with reference to nine drawings. FIG. 1 is a diagram showing an information processing system according to an embodiment of the present invention. 1 is a display means and is a character. Graphics 9 Display images and/or sounds. Reference numeral 2 denotes a display processing means, which displays each piece of information on the screen of the display means 1 in accordance with a display command from the information processing means 5, which will be described later. 3
4 is a means for inputting a position on the screen of the display means 1; 4 is a means for inputting characters; and 8 is a means for inputting an image. Data input from these input means is displayed on the display means 1 via the display processing means 2. Reference numeral 5 denotes an information processing means that processes the input data and creates, edits, and stores documents. 6 is a means for storing various data (including the above-mentioned documents) used in the information processing, and 7 is a filing means for accumulating the above-mentioned documents. 9 is a sound input means such as a microphone, telephone, VTR, etc.; 14 is a sound input means 9 for amplifying the analog sound signal inputted from the sound input means 9;
A means of frequency filtering. 19 samples the output signal of the means 14; i1 AD converter for converting into negative acoustic data, 20°21
．． 22 and 23 are DA converters that convert recorded and/or synthesized 1IWl sound data into analog sound signals; 1;
5, 16, 17 and 18 are means for amplifying the output signal of the above means and filtering it with two frequencies; 10, 11, 12;
and 13 are sound output means such as speakers. Further, 29 is a means for generating a control signal for repeating the AD and DA conversion at a constant cycle, and 28 is a means for compressing and encoding the AD-converted discrete audio data and decoding the encoded audio signal. This is a signal processing processor that performs conversion processing, sound synthesis processing, etc. 24 is the AD converter 19
Transfer control of the discrete acoustic data from the information processing means 5 or the signal processing processor 28 and DA of the discrete acoustic data from the information processing means 5 or the signal processing processor 28
This is acoustic data processing means that controls transfer to the transducers 20, 21, 22, and 23. 32 is a CD (combination disc) device. Each of the above means is connected by a main bus 30 or sound bus 31 consisting of data, address and control lines. Furthermore, the acoustic data processing means 24 includes an output volume control means 27 for controlling the individual output volumes of the plurality of acoustic output means, and an acoustic data storage means 26 for temporarily storing the discrete acoustic data. A dead time memory readout control means 25 is provided which performs dead time control and reads discrete acoustic data from the storage means 26. Further, in this embodiment, the plurality of sound output devices IQ, 1
It is assumed that 1.12 and 13 are arranged on the left and right, and on the top and bottom of the display means 1, as shown in FIG. The information processing means 5 and the acoustic data processing means 24 are implemented using a general microprocessor (for example, Motorola's 6
8000. It is assumed that the signal processing processor is a digital signal processing device such as Intel's TMS 32020 or the like. The display processing means 2 divides the screen of the two display means 1 into a plurality of partial areas (windows), operates the windows as independent display means, and creates a virtual screen having a logical screen larger than the screen of the actual terminal consisting of the display means 1. Multi-window is realized by controlling and associating the windows displayed on the screen of a real terminal and the real terminal. Specifically, the display processing means uses the information processing means 5 using the technology disclosed in Japanese Patent Application Laid-Open No. 62-281074 and Japanese Patent Application Laid-Open No. 63-153625.
Create a virtual terminal and edit a logical virtual screen using the command. Then, a part of the virtual screen is cut out as a field of view, and a window is created on the screen of the display means 2, which is the real terminal.
Display as . Furthermore, moving windows on the screen of a real terminal. A means 3 (such as a mouse) for inputting the position of the display means on the screen of the operator with respect to the virtual terminal or a means 4 (such as a keyboard) for inputting characters, which has functions such as changing the size and moving the field of view on the virtual plane. The information processing means 5 inputs
The information processing means 5 can freely set the configuration and size of the virtual screen, display a mixture of 2 characters, 9 figures, 9 images, etc., and display multiple windows at the same time. . Next, a method of communication between a user and the system in the information processing system of the present invention will be described. The communication method includes a method of displaying a command menu on the display means 1 and selecting it using the position input means 3, and a method of typing a command from the single character input means 4. In the former method, the information processing means 5 displays a command menu field required for inputting 2 characters, 2 figures, 9 images and/or sounds, 9 editing, storing and/or outputting, etc., according to a procedure described in advance in a program. The position on the virtual screen to be displayed is determined, and the display area information (for example, area origin coordinates, width,
(vertical width, etc.) is stored in the information processing storage means 6. Then, the display area information and the command menu character code are sent to the display processing means 2, which displays the command menu on the screen of the display means 1. In such a state, the user selects the command in the command menu displayed on the screen of the display means 1.
When an instruction is given using the position input means 3, this position information is transmitted to the information processing means 5 by the display processing means 2. The information processing means 5 controls the information processing storage means 6 to read the area information in the command menu column, performs matching with the position information, and recognizes the command selected by the user. In the information processing system of the present invention described above, an embodiment of recording will first be described. When the information processing means 5 receives an instruction to record from the user using one of the above-mentioned methods, it issues an initialization command for recording to the audio data processing means 24 via the main bus 3°. In response to the initial command, the acoustic data processing means 24 issues a reset signal to the signal processing processor 28 via the sound bus 31, thereby setting the state of the means to a reset state. Thereafter, the acoustic data processing means 24 notifies the information processing means 5 of the completion of the reset via the main bus 30. When the information processing means 5 receives notification of the completion of reset from the audio data processing means 24, it secures a storage area for the recorded sound in the information processing storage means 6. Then, a recording start command is issued to the audio data processing means 24. When the acoustic data processing means 24 receives the instruction. The signal processing processor 28 is started, and the discrete acoustic data (having a length of 8 bits or more ) is captured using the input command of the means 24 and continues to be transferred to the signal processing processor 28 using the output command of the means 24. The signal processor 28 compresses and encodes this discrete audio data into a length of several bits using the international standard CCITT G, 721 ADPCM method, and outputs it to the information processing means 5 via the audio bus 31. The information processing means 5 sequentially stores the compression-encoded recorded audio data sent from the signal processing processor 28 in the storage area secured within the information processing storage means 6. In this state, when the information processing means 5 detects a recording stop instruction from the user using the method described above. A recording stop command is issued to the audio data processing means 24. When the audio data processing means 24 receives the recording stop command, it stops taking in the discrete audio data from the AD converter 19 and stops transferring the audio data to the audio data processing means 24. A processing stop command is issued to the signal processor 28 to stop the compression encoding process. after that. The information processing means 5 is notified of the completion of recording stop. When the information processing means 5 receives the notification of the completion of the stop, it stores the recorded audio data stored in the storage area in the information processing storage means 6 in the file means 7. In addition,
When recording sound with high quality, signal processing processor 2
The acoustic data processing means 24 directly transfers the discrete acoustic data taken in from the AD converter 19 to the information processing means 5 without performing compression encoding according to 8. The sound recording processing in the two embodiments of the present invention has been described above. Next, the means for stereophonizing and outputting the thus recorded sound or synthesized sound will be described in detail. First, the principle of pseudo stereo conversion will be explained using FIGS. 3 and 4. FIG. 3 is a top view of the positional relationship between the user 101, the sound output equipment 11.12 in FIG. 2, and the sound source 100. on the other hand. FIG. 4 shows the user 101 and the sound output device 12 in FIG.
．． 13 and a sound source 100 viewed from above. FIG. The sound source localization function of humans using both ears has been studied in the field of psychophysiology, and it is known that humans perceive the location of sound sources based on the time difference (phase difference) and sound pressure difference between the sounds that enter the ears. For example, in FIG. 3, the time taken for the sound emitted from the sound source 100 to reach the position of the sound output device 10 is T.
Engineering. If so. The sound Y1°(1) at the position of the sound output device 10 is expressed as the following equation. Yxa(t) = xirxa X x(t r
xa) (1) In 22, is a constant, t is time and
(t) represents the sound g100. Similarly, if the time taken for the sound emitted from the sound source 100 to reach the position of the sound output device 11 is T, then the sound output device 1
The sound Y1□(1) at position 1 is expressed as follows. r□□(t) = K/T□□ xx(t-r□□)
(2) Therefore, the sound output devices 10 and 11 are arranged at equal intervals on the left and right with the user at the center, and near the user. By outputting the sound of equation (1) from the audio output device 1o and the sound of equation (2) from the audio output device 11, the user can aurally perceive the position of the sound source. That is,
It is possible to give the sound depth in the horizontal direction. Furthermore, as shown in FIG. 4, the sound output devices 12 and 13 are placed above and below the front of the user 101, at equal intervals above and below with the user at the center, and close to the user. Sound output device 12 outputs the sound of , and sound output device 1 outputs the sound of equation (4).
By outputting from 3, it is possible to make the user perceive an auditory stereo sensation. Y, z(t) = x/r□z x X(t
Lx) (3) YJz(t) = x/r, s
x x(t-r□7) and 2 equations (3
) in Ti2 and T in formula (4). The sound emitted from the sound source 100 is transmitted to the sound output devices 12 and 13.
represents the time to reach the position. The processing described above is carried out by the acoustic data processing means 2 in FIG.
Do it within 4. A detailed example thereof is shown in FIG. The circuit of FIG. 5 consists of three parts: an acoustic data storage means 26 consisting of one means 49-51, an output volume control means 27 consisting of two means 52-55, and a dead time memory read-out control means 25 consisting of a section other than these. It consists of First, a means for realizing the term X (t-T) in Equation 2 (1) will be described. In the present invention, this item is realized using the acoustic data storage means 26. The acoustic data storage means 26 is a means for temporally storing the discrete acoustic data X (t). Said means 2
6 for the address set in the memory address register 51 in the memory 49. Write the contents of the memory data register 50. Alternatively, the contents of the address are read into the memory data register 50. The memory address register 51 consists of multiple bits and handles values in a positive range. Therefore, when the address is incremented by 1 and the most significant pit overflows, the value of the register increases again from 0. That is, ring-like access is possible as shown in FIG. The contents of the memory address counter 56 represent the address corresponding to the time t, and the memory address counter 56 is incremented by 1 in accordance with the signal from the periodic signal generating means 29. In such an environment, the acoustic data storage means 26
First, the selector 60 is controlled to write the contents of the register 61 into the memory data register 5o.The information processing means 5 writes the value of X(t) into the register 61 as appropriate. In parallel with this, the selector 59 is controlled to write the contents of the memory address counter 56 to the memory address register 51. Then, it controls the acoustic data storage means 26 and writes the contents of the memory data register 50 to the address in the memory 49 indicated by the memory address register 51. After writing the value of X (t) into the memory 49 in this manner, the value of X (t-T) is read out from the memory 49 by performing the following processing. first. The acoustic data processing means 24 controls the subtracter 58 to subtract the T8° set in the register 73 from the contents of the memory address counter 56 by means described later, and controls the selector 59 to store the output of the subtracter 58 in the memory. Write to address register 51. Then, the acoustic data storage means 26
and controls the memory 4 indicated by the memory address register 51.
9, that is, T□. The value of X before the time is read into the memory data register 5o, and is set in the register 55 by controlling the selector 60. The means for realizing the term X(t-T1°) in Equation 9 (1) has been described above. Next, the N/T construction in formula 9 (1). Output volume control means 27 for calculating xX(t-T□.)
An example will be described. The acoustic data processing means 24 reads out and sets the contents T□ of the register 70 by means described later. is divided by the contents of the register 53 by controlling the divider 52 by 2. The contents of the register 53 are set by the information processing means 5. Further, the 2 multiplier 54 is controlled to multiply the output of the 2 divider 52 by the content X (t-T) of the register 55, and the output, ie, Yxx(t), is set in the register 74. The thus obtained Y
The signal is sent to the A converter 20, converted to an analog signal via the filter 15, and outputted from the audio output means 10. An example of formula 9 (1) has been described above, but - numerically, the above T□. is not a constant, but a function T that changes over time. It is expressed as (1). Next, an embodiment for realizing this will be described using FIG. 5. In this embodiment, as described above, X(1) is written into the memory 49 at time t and T operation is performed from time t. Reads the value previously written. That is, X(1)
T□ than when I wrote it. Only later Tyu. is used to process X (t-T). For this reason, in the present invention, the dead time control memory 62 shown in FIG. 5 is provided,
The above processing is realized. The acoustic data processing means 24 first reads the contents of the memory address counter 56 and the T set in the register 72 by the information processing means. Add the value of (1). The selector 69 is controlled to set the memory address register 67. In parallel with this, the contents of the register 72 are set in the memory data register by controlling the selector 68. Thereafter, the contents of the memory data register are written to the address in the dead time control memory 62 indicated by the memory address register 67. When this operation is completed, the contents of the memory address counter 56 are further set in the memory address register 67 by controlling the selector 69, and the contents of the address in the dead time control memory 62 indicated by the memory address register 67 are set in the memory data register. , read out. Then, the content is set in the register 70 by controlling the selector 68. The detailed means of an embodiment related to 2 equations (1) have been explained above using FIG. The control means 27 and the dead time memory readout control means 25 are used in a temporally divided manner to perform the same processing as in Equation 9 (1), thereby realizing stereophonic sound. Note that the signal processing processor 28 converts the compressed and encoded recorded audio data transferred from the information processing means 5 into CC.
ITT G, 721 Restored by ADPCM method. The register 61 shown in FIG. 6 is set in accordance with the output timing of the means 29. In addition, the Japanese Patent Publication No. 60-14360 synthesizes sound using the method described in the Japanese Patent Publication No. 47-15402, etc., and stores the said means 29 in the register 61 of FIG.
Set according to the output timing. Switching of these processes in the signal processing processor 28 is performed by the information processing means 5 via the main bus 30. Furthermore, as shown in FIG. 6, a plurality of stereo conversion means shown in FIG. The stereo sound is converted into stereo under the specified conditions, the total sum of the stereo sound is determined by the sum-of-products calculator 80, and the output of the sum-of-products calculator 80 is output to the DA converter. Next, one application example of the present invention will be described with reference to FIG. FIG. 7 shows a layout diagram of the display means 1 when viewed from the front. In FIG. 7, reference numeral 90 represents a virtual screen of the virtual terminal, which is generated by the display processing means 2 in response to a command from the information processing means 5. Displays 1 character, 2 graphics, 2 images and/or sounds. As described above, the 9 display processing means 2 cuts out a part of the virtual screen as a field of view according to the command from the information processing means 5, and displays it as a window 91 on the screen of the display means 2, which is a real terminal. Reference numeral 91 in FIG. represents this window. Further, reference numeral 92 represents an icon indicating the presence of audio data. However, since the icon 92 is not displayed on the window 91, the user does not know its location. In such a situation, a process will be described in which the means of the present invention is used to notify the location of the icon in FIG. 7, that is, the acoustic data, using the sense of distance provided by stereo sound. The information processing means 5 manages the coordinates of the display object shown in FIG. 7 in both the real screen coordinate system of the display means 1 and the individual virtual screen coordinate systems. Here, the information processing means 5 calculates the difference between the display coordinate value of the icon 92 and the origin 93 of the real screen coordinate system of the display means 1, and calculates the difference between the origin 93 and the icon 92, and calculates the difference between the origin 93 and the icon 92 to give a sense of distance. As shown in FIG. 8, a virtual depth distance aL is introduced to generate a stereo effect. In FIG. 8, numerals 10 and 11 represent the audio output devices 1o and 11 shown in FIG. 3. The information processing means 5
The magnitude of L is checked and an operation expressed by a linear equation is performed to obtain T1° and T□1 explained in FIG. 3, and set in the register 72 in FIG. (1) When XL<O. Txo=fi 7(5) T□□=mπ−F(6) (2) When XL≧0. T□2=Jr5T finished ear 7 (7) T□3= (XL-m) 2+I, 2 (8
) The information processing means 5 outputs sound to the vertical direction of the two screens, that is, to the sound output devices 12 and 13 shown in FIG. XL in the above formulas (5) to (8) is replaced with YL. Calculate using the same procedure as above, register 72 in FIG.
Set to . By performing such processing, stereo sound is output from the plurality of audio output devices to notify the positions of various objects on the two display means. Furthermore, by setting L in FIG. 7 to the position in the depth direction of the object, the movement of the object in the three-dimensional space can be audibly conveyed to the user. In the embodiment of the present invention shown in FIG. 1, an example is shown in which a filter and a DA converter are used as a set, and a plurality of sets are provided for stereo output. However, as shown in FIG. A converter 93 is provided, which is used under time-division control, and the output is sent to the filters 15 to 15 shown in FIG. 1 using a distributor 94.
It is also possible to implement a configuration in which the output is output to 18. [Effects of the Invention] The information processing system of the present invention enhances emotion and sense of presence through stereo reproduction, and promotes understanding of the contents and images of documents, catalogs, etc. expressed in multimedia displayed on display means. At the same time, it became possible to make use of the auditory distance sensation provided by stereo to notify the location and movement of the object to be operated or focused on, making full use of the goodness of sound. A more user-friendly system can be realized.

[Brief explanation of the drawing]

Fig. 1 is a block diagram showing the overall configuration of an information processing system according to an embodiment of the present invention, Fig. 2 is a front view showing the arrangement of audio output equipment, and Figs. 3 and 4 are explanations of the principle of the stereo system. Fig. 5 is a block diagram showing detailed means for performing stereo processing, Fig. 6 is a block diagram showing the configuration of an embodiment for stereophonizing multiple sounds, and Figs. FIG. 9 is an explanatory diagram showing an application example of stereo output in the information processing system of the invention. FIG. 9 is a block diagram showing the configuration of another embodiment for stereophonizing sound. Explanation of symbols 1...Display means, 2...Display processing means, 3...Screen position input means of display means, 4...Character input means,
5... Information processing means, 6... Various data storage means,
7... Filing means, 8... Image input means, 9
...Sound input means, 10.11.12 and 13...
Sound output means, 14... frequency filtering means,
15.16.17 and 18...frequency filtering means, 19...AD converter, 20.21.22 and 23...DA converter, 24.
...Acoustic data processing means, 25...Dead time memory reading control means, 26...Acoustic data storage means, 27
... Output volume control means, 28 ... Signal processing processor, 29 ... Control signal generation means, 31 ... Sound bus,
30...Main bus, 32...CD (combined disc) device. Figure ♀

Claims (1)

[Claims] 1. In an information processing system that performs processing such as inputting, editing, storing and/or outputting characters, figures, images and/or sounds, means for outputting recorded and/or synthesized stereo sound is provided. An information processing system comprising: 2. An information processing system comprising a plurality of sets of means for converting recorded and/or synthesized discrete sound data into analog signals, a filter, and sound output equipment. 3. A means for converting recorded and/or synthesized discrete sound data into an analog signal, a plurality of filters and sound output devices, and a means for switching and outputting the analog signal to the plurality of filters and sound output devices. Information processing system. 4. An information processing system comprising means for controlling the volume of a single sound and means for temporally delaying the sound. 5. An information processing system comprising a means for controlling the volume of a plurality of sounds, a means for temporally delaying the plurality of sounds, and a means for synthesizing and outputting a plurality of sounds processed by the two means. . 6. In an information processing system that inputs, edits, stores, and/or outputs characters, graphics, images, and/or sounds, means for displaying characters, graphics, images, and/or sounds, and means for displaying the characters, graphics, images, and/or sounds; A display processing means for controlling a screen area (window), drawing control, etc. is provided, and the various objects are displayed in a window on the display means by controlling the display processing means, and the various objects and the window are displayed in a window on the display means. An information processing system characterized in that the sense of distance is changed by stereo sound according to the positional relationship of the objects, and the positions of the various objects are notified. 7. The information processing system according to claim 1, characterized in that the audio output devices are arranged on the left and right and/or above and below the means for displaying characters, figures, images and/or sounds.