JP3525817B2 - Operation reaction sound generation device and recording medium storing operation reaction sound generation program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an operation reaction sound generation device and a recording medium recording an operation reaction sound generation program, which is one of user interfaces of a computer system, and more particularly to a user input operation and an attribute of an object. The present invention relates to an operation reaction sound generation device that parametrically generates an operation reaction sound, and a recording medium recording an operation reaction sound generation program.

[0002]

2. Description of the Related Art Conventionally, a method of outputting a reaction sound in response to some operation by a user has been used for the purpose of operation feedback in a dialogue system. FIG. 11 is a block diagram showing an example of the configuration of a conventional operation reaction sound output device, and the configuration of the conventional operation reaction sound output device will be described with reference to FIG.

The conventional operation reaction sound output device has an input device 11 such as a mouse and a keyboard for the user to input to the interactive system, and a graphical user interface (hereinafter referred to as GUI). An event-driven type interactive system 12 that performs internal processing according to the above, an event acquisition unit 13 that acquires event information (button press, icon click, menu open, etc.) that occurs in the interactive system 12, and digital audio data. Operation for acquiring operation reaction sound database 15 in which operation reaction sound data is accumulated and operation reaction sound data corresponding to the event information acquired by event acquisition unit 13 from operation reaction sound database 15, and outputting as acoustic data Reaction sound selection unit 14 and operation reaction sound selection unit 14
And an acoustic output device 16 such as a speaker that outputs the acoustic data output from.

That is, in the conventional operation reaction sound output device, operation reaction sound data (for example, operation reaction sound database) prepared in advance for event information (for example, "button press") occurring in the interactive system 12 is generated. 15
If the digital sound file "kachi.wav" is stored in association with the event information "button pressed", this "kachi.wav") is selectively output.

Further, one example of further development of this operation reaction sound generating device is "April 1993, Proceedings
Of Inter-Kay '93 pp.228-235 "Synthesizing Auditory Icons""(" Synthesizin
g Auditory Icons. ", Proceedings of INTER-CHI'93. p
p.228-235 April 1993). In this paper, using the attribute of the operation target object as a parameter,
A technique of processing the operation reaction sound parametrically and outputting it is described.

Specifically, when a file having a large file size is clicked, a large operation reaction sound is output, the tone of the drag sound is changed according to the background when the icon is dragged, and a sound is generated when the process becomes faster (heavy). Get faster, and so on. By using this technology, it is possible to represent icons and system information using operation reaction sounds.

On the other hand, a technique for generating sound or music in response to pointing device movement information is disclosed in Japanese Patent Laid-Open No. 10-4.
It is described in Japanese Patent Publication No. 0067. According to this, by acquiring movement information such as the movement amount and movement direction of the pointing device and outputting sound, voice, or music,
The operation of the pointing device can be understood auditorily, and the operability of a personal computer or the like is improved.

A keyboard capable of detecting the pressing force of a key on the keyboard and outputting the information as pressing force information is disclosed in Japanese Patent Laid-Open No. 62-62.
No. 70897. It is stated that this keyboard is suitable for inputting information on music performance, since information on the strength of keystrokes can be input.

Further, there is a technique in which a keyboard can detect the presence / absence of a contact, a pressing pressure, a speed, and a depth, and a signal sound is changed according to the information, as disclosed in Japanese Patent Laid-Open No. 9-29200.
No. 305286. According to this, malfunction is reduced by feeding back the pressed state of the key of the keyboard to the user.

On the other hand, in the field of animation generation, a device for generating a predetermined sound according to the motion information of a visual object displayed on a display is disclosed in Japanese Patent Laid-Open No. 6-.
No. 243229. according to this,
The feature of automatically generating a predetermined sound when an object approaches or collides with another object facilitates the generation of animation.

An invention for controlling the movement and timing of a visual object so as to be synchronized with an output audio signal is described in Japanese Patent Laid-Open No. 232601/1993. According to this, at least one of the timing and order of reproduction or output of a moving image is controlled based on the image control signal included in the audio signal.

[0012]

However, the above-mentioned conventional technique has the following problems.

First, the first problem of the prior art is that the operation reaction sound output in the operation on the object in the virtual world is unrealistic, and the user hears the operation reaction sound so It is impossible to know whether the operation has been performed.

The reason for this is that regarding the output of the operation reaction sound,
The point is that neither “detailed information of user operation (type of input device, operation strength and operation speed)” nor “attribute information of object to be operated” is taken into consideration. In addition, the operation strength and the operation speed of the input device are not acquired as continuous numerical values, and the acquired continuous numerical values are not reflected in the operation reaction sound output.

The generation of sound in the real world depends on the states of both the effecting side and the receiving side. For example,
When tapping a cup of water with a stick, the state of the tapping side (eg:
Various sounds are generated depending on the relationship between both the material of the rod, the strength of tapping, the angle of tapping, etc.) and the state of the side to be hit (for example, the amount of water, the material of the cup, etc.).

On the other hand, a conventional example relating to a GUI operation reaction sound (the method shown in FIG. 11, and "April 1993, Proceedings of Inter-Kay '93 pp.228-235").
In the operation reaction sound generator described in “Synthesizing Auditory Icons”), detailed information of the user operation (operation intensity, operation speed, type of input device) is reflected in the output operation reaction sound. There is no such thing. More specifically, the same operation reaction sound is output regardless of whether the input device is a mouse or a touch panel, and the GUI object is strongly or weakly clicked.

The above-mentioned conventional example, Japanese Patent Laid-Open No. 10-400.
67, JP-A-62-70897, JP-A-9
As shown in Japanese Patent Laid-Open No. 305286, there has been a device that changes a sound according to movement information of a mouse, which is one of operation information, and pressing force of a keyboard, but the cause of the sound change is limited to only the operation information. However, there is no relation with the object that is the operation target. Therefore, the first problem of the above-mentioned conventional technique cannot be solved.

Further, the conventional examples shown in JP-A-6-243229 and JP-A-5-232601 are as follows.
This is an example in which the motion of an object and sound are interlocked in a closed world called a virtual world, and the relation between the user's operation in the real world and the generated sound is not taken into consideration. Can not solve the problem of.

Next, the second problem of the prior art is that when outputting various sounds, it is necessary to create all of these sounds in advance and store them in the recording device.

The reason for this is that in the conventional embodiment, the sounds prepared in advance (in the example of FIG. 11, the operation reaction sound database 1
This is because it has only the function of playing back the file stored in 5) at a certain timing.

In view of the above-mentioned problems of the prior art, the present invention provides an operation reaction sound generation device and an operation reaction sound generation program for generating an operation reaction sound that increases the reality and naturalness of GUI operation. It is an object of the present invention to provide an operation reaction sound generation device and a recording medium in which an operation reaction sound generation program is provided, which can provide a recorded recording medium and improve the naturalness of a user interface.

[0022]

According to the present invention, both "detailed information of user operation which is a phenomenon in the real world" and "attribute information of an operation target object in the virtual world" are used,
Generates various operation reaction sounds in real time.

More specifically, based on the operation information (type of input device, operation strength and operation speed) acquired from the input device, and the object attribute information in the virtual world, the tone color, height, and size of the acoustic data are set. It is processed and edited, or the sound data is subjected to effect processing and output.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION [First Embodiment] (Description of Configuration) Next, a first embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing an example of the configuration of the first embodiment of the present invention.

In FIG. 1, the input device 11 outputs the action of the user, particularly the intensity of the operation, as a continuous numerical value. Specifically, in addition to the basic functions of an input device such as a mouse, a keyboard, and a touch panel used in a normal dialogue system (output of click presence, drag speed, drag direction, etc.), operation strength ( For example, “click strength”) is measured and a continuous numerical value (= operation parameter) is output. The dialogue system 12 is an event-driven system that has a GUI and performs internal processing according to an input from the input device 11.

The event acquisition unit 13 acquires UI-related events (button press, icon click, menu open, etc.) that occur in the interactive system 12 and outputs them to the operation reaction sound synthesizer 4.

On the other hand, the operation parameter acquisition unit 2 acquires the operation parameter of the input device 11 through the dialogue system 12 or directly from the input device 11 and outputs it to the operation reaction sound synthesizer 4.

The object attribute acquisition unit 3 acquires the attribute information of the GUI object to be operated from the interactive system 12 and outputs it to the operation reaction sound synthesizer 4. The object attribute information referred to here is the size, color, data capacity, update date, caption,
And so on.

The operation reaction sound synthesizer 4 uses the event information,
Using operation parameters and object attribute information,
The operation reaction sound based on each of these pieces of information is synthesized and output to the sound output device 16 as sound data.

The sound output device 16 is a device for outputting the input sound data as a sound, and is specifically composed of a D / A conversion device, an amplifier, a speaker and the like.

Further, the operation reaction sound synthesizer 4 in the present embodiment has a basic reaction sound selection section 41, an operation reaction sound database 42, a sound editing apparatus 43, a first parameter conversion section 44, and a second parameter conversion section 44. And the parameter conversion unit 45 of FIG.

The operation reaction sound database 42 stores sound data.

Event information is input to the basic reaction sound selection unit 41 from the event acquisition unit 13. The basic reaction sound selection unit 41 holds in advance the correspondence information between the input event information and sound data. Sound data corresponding to the input event information is acquired from the operation reaction sound database 42 and output to the sound editing device 43 as operation reaction sound data.

The first parameter conversion unit 44 converts the operation parameter input from the operation parameter acquisition unit 2 into the first sound editing parameter and outputs it to the sound editing device 43.

The second parameter conversion unit 45 converts the object attribute information input from the object attribute acquisition unit 3 into a second sound editing parameter, and the sound editing device 4
Output to 3.

The sound editing device 43 is a device having a function of editing and processing sound waveforms such as various sound effectors, sound filters, and equalizers, and waveform editing software and various effect devices for musical instruments can be used. The sound editing device 43 includes the first and second parameter conversion units 4
Using the first and second sound editing parameters input from 4 and 45, the operation reaction sound data input from the basic reaction sound selection unit 41 is processed and edited, and the processed and edited sound data is output as an audio output device. Output to 16.

(Explanation of Operation) Next, the operation of the operation reaction sound generating apparatus in the first embodiment shown in FIG. 1 will be described with reference to the flowcharts of FIGS. 2 and 3.

The user uses the input device 11 to perform an input operation such as clicking or dragging on the GUI object of the interactive system 12 (S1). Input device 11
In addition to the operating parameters output by a normal input device,
Numerical value of at least click strength (= operating parameter)
Output as.

The interactive system 12 receives an input operation from the input device 11 and advances processing in an event driven manner (S
2).

The event acquisition unit 13 acquires an operation-related event generated in the dialogue system 12 and outputs it to the operation reaction sound synthesizer 4 as event information (S).
3).

On the other hand, the event acquisition unit 13 outputs the acquired event to the operation parameter acquisition unit 2 and the object attribute acquisition unit 3 as well, and the operation parameter acquisition unit 2 receives the event via the interactive system 12 or The operation parameters are directly acquired from the input device 11 and output to the operation reaction sound synthesizer 4 (S4).

When the event information is input from the event acquisition unit 13, the object attribute acquisition unit 3 acquires the attribute information of the GUI object related to the event information from the interactive system 12 (S5). Further, the acquired object attribute information is output to the operation reaction sound synthesizer 4.

The operation reaction sound synthesizer 4 uses the event information,
The operation reaction sound is synthesized using the operation parameter and the object attribute information (S6).

The synthesized operation reaction sound is output from the acoustic output device (S7).

The synthesis of the operation reaction sound (S6 of FIG. 2) performed by the operation reaction sound synthesizer 4 will be described in detail with reference to FIG.

Event information is input from the event acquisition unit 13 to the basic reaction sound selection unit 41. The basic reaction sound selection unit 41 holds the correspondence relationship between the event information and the operation reaction sound data in advance and inputs it. The operation reaction sound data corresponding to the event information is acquired from the operation reaction sound database 42 (S61) and input to the sound editing device 43 (S6).
2).

On the other hand, the first parameter conversion unit 44 converts the operation parameter input from the operation parameter acquisition unit 2 into the first sound edit parameter based on the correspondence table or conversion rule held in advance ( S6
3), and output to the sound editing device 43 (S64).

The second parameter conversion unit 45 converts the object attribute information input from the object attribute acquisition unit 3 into a second sound editing parameter based on a correspondence table or conversion rule held in advance (S6).
5), and output to the sound editing device 43 (S66).

The sound editing device 43 uses the first and second sound editing parameters input from the first and second parameter converting sections 44 and 45 to perform the operation input from the basic reaction sound selecting section 41. An acoustic editing process is performed on the reaction sound data (S67), and the processed acoustic data is output to the acoustic output device 16.

[Example] Here, the first embodiment of the present invention will be described in more detail with reference to a specific example.
Further, here, a detailed description will be given with reference to FIG. 1, FIG. 4, and FIG. 7, which are block diagrams of one configuration example of the first exemplary embodiment.

In the present embodiment, an example in which a mouse with a pressure sensitive sensor (71 in FIG. 4) is used as the input device 11 and the push buttons 72 and 73 displayed on the display are clicked with the mouse will be described. To do.

The mouse 71 with a pressure-sensitive sensor used as the input device 11 is, for example, in September 1998, 14th Human Interface Symposium Proceedings, 231-231.
The mouse reported in FIG. 1 of “Analysis of click pressure during mouse positioning work” on page 235 can be used. This mouse can output the pressure when the mouse button is clicked as digital data. Similar to a normal mouse, it also outputs click-on / off signals, drag direction, and drag speed.

When the user uses the mouse 71 with the pressure sensitive sensor to move the cursor on the GUI onto the push button 72 and clicks the mouse button, the mouse 71 with the pressure sensitive sensor generates a click pressure with a click-on signal. Output as a numerical value. All of these pieces of operation information output from the input device are operation parameters. Dialog system 1
2 interprets what operation was performed on which GUI object from the coordinates of the cursor and the input click-on signal, and performs an event (in this case, "push button pressed").

The event acquisition unit 13 acquires GUI operation-related events among the events generated in the dialogue system 12, and outputs them to the basic reaction sound selection unit 41, the operation parameter acquisition unit 2 and the object attribute acquisition unit 3. .

The basic reaction sound selection section 41 holds in advance a table (basic reaction sound selection table) showing a correspondence relationship between events and operation reaction sounds as shown in FIG. 7, and corresponds to an input event. The data of the operation reaction sound to be played is acquired from the operation reaction sound database 42, and the sound editing device 43
To enter. In this case, referring to the event field 75 of the basic reaction sound selection table shown in FIG. 7, the operation reaction sound field 76 corresponding to “push button pressed” is displayed in “File3.wa”.
v ”can be obtained. The basic reaction sound selection unit 41 displays “Fil
The sound data “e3.wav” is acquired from the operation reaction sound database 42 and output to the sound editing device 43. (Where File
3.wav is the sound data of the timbre that can be expressed by the onomatopoeia "Kachin". )

On the other hand, when an event of "button press" is input from the event acquisition unit 13, the operation parameter acquisition unit 2 receives the input event (here, "button pressed") through the interactive system (or directly from the input device 11). The operation parameter (for example, “click pressure = 80”) related to “press” is acquired. Next, the acquired operation parameter (“click pressure = 80”) is used as the first parameter conversion unit 44.
Output to. In this case, assuming that the click pressure is 80, the click pressure is a value between 0 and 99, and is a value indicating that the mouse has been clicked relatively strongly.

The first parameter conversion unit 44 converts the input operation parameter into the first sound editing parameter. This conversion is executed based on the conversion table or conversion rule described in the first parameter conversion unit 44. The operation parameter “click pressure = 80” is the first acoustic editing parameter (here, for example, the pass frequency value of the bandpass filter (parameter for changing the sound quality), the modulation depth value of the chorus effector (parameter that complicates the timbre)). , Volume gain value (parameter for volume change)).

The conversion table or conversion rule described in the first parameter conversion unit 44 is based on the relationship between "action and generated sound" in the real world. For example, in the real world, the stronger the action, the larger the volume of the generated sound and the more complicated the timbre, but such a correspondence is described in the conversion table. In this case, since the click pressure of 80 indicates that the mouse has been clicked relatively strongly, the volume is increased and the sound editing parameter is converted to complicate the sound. And the first obtained
The sound editing parameter of is output to the sound editing device 43.

On the other hand, when the event information is input from the event acquisition unit 13, the object attribute acquisition unit 3 acquires the attribute information of the GUI object related to the event. In this case, the "push button" to be operated
Attribute information such as button size, texture, and color. Then, the acquired object attribute information is output to the second parameter conversion unit 45.

The second parameter converter 45 converts the input object attribute information into the second acoustic parameter. In this case, for example, the size of the object is converted to a pitch shift value (a parameter that changes the pitch of the pitch),
The texture of the object is converted into the parameter value of the acoustic filter (parameter that changes the timbre). The conversion table or conversion rule described in the second parameter conversion unit 45 is based on the principle of sound generation in the real world. The principle of sound generation is that in the real world, the tone color usually differs depending on the material of the object being hit, and the pitch changes depending on the loudness. The conversion table or conversion rule described in the second parameter conversion unit 45 imitates this relationship. Converted second
The sound editing parameter of is output to the sound editing device 43.

The sound editing apparatus 43 uses the first and second sound editing parameters input from the first and second parameter converting sections 44 and 45 to perform the operation input from the basic reaction sound selecting section 41. Process and edit the reaction sound data (in this case, "file3.wav"). Specifically, the sound editing apparatus 43 can be realized by, for example, waveform editing software, and sets the operation reaction sound data “file3.wav (click)” which is the input from the basic reaction sound selection unit 41 as the sound to be edited. Waveform editing software, as its function,
It has various waveform editing functions such as pitch shift, waveform cut-and-paste, fade-in, fade-out, power envelope processing, and sound effect functions such as delay, chorus, reverb, and filter functions.
The waveform editing software processes and edits "file3.wav" by executing various functions using the acoustic editing parameters. As a result, new acoustic data “newfile3.wav (gatchin)” in which the timbre, pitch, volume, etc. are changed is generated based on file3.wav. The processed acoustic data is input to the acoustic output device 16, and the acoustic output device 16 outputs the input acoustic data as a sound.

As a result, the sound "click" selected as the push button press sound reflects "click", "click", "click", etc., reflecting the user's click strength, button size, color, etc. Such various sounds are edited and output (Fig. 4). As an event, even with the same operation of pressing a button, a more realistic operation reaction sound that reflects the state of the side performing the operation (operation parameter) and the side receiving the operation (object attribute information) may be output. It will be possible.

Here, another example of the first embodiment will be described with reference to FIG.

FIG. 5 is a diagram for explaining the operation reaction sound of the file icon. When an icon is clicked (77 in FIG. 5), an “icon click” event occurs, but here, for example, the operation parameter acquisition unit 2 operates “click strength (pressure)” as in the previous example. Get as a parameter. On the other hand, it is assumed that the object attribute information acquisition unit 3 acquires, as the object attribute information, information such as “file size”, “file update date”, “file type”, etc. that is not apparent to the GUI object. By editing the waveform using the acoustic editing parameters obtained based on the operating parameters and object attribute information, information (file size, file update date, etc.) that is not expressed by visual information can be used as the operation reaction sound. It becomes possible to express through.

On the other hand, reference numeral 78 in FIG. 5 is a diagram showing operation parameters and object attribute information when icon dragging is performed. When a drag operation is performed, an “icon drag” event occurs. At this time, the operation parameter acquisition unit 2 may acquire “drag speed” as an operation parameter. By converting the operation parameter called drag speed into an acoustic edit parameter and using the obtained acoustic edit parameter to edit and process the sound, for example, the tone or sound of the drag sound such as “smooth” is dragged according to the drag speed. The height of can be changed.

As described above, the operation parameters representing the operation characteristics such as the operation intensity and the operation speed by continuous numerical values are used for the operation reaction sound synthesis. By this,
The output sound can be changed according to the characteristics of the user operation, and as a result, the realistic and natural operation is improved.

Further, referring to FIG. 6, a description will be given of an example of the operation reaction sound to the operation of the menu button. When the menu button displayed on the menu bar is pressed, the menu button is visually depressed and at the same time a pull-down menu opens. Although the user's operation is only pressing of the menu button, opening of the pull-down menu is a reaction that is triggered by the pressing of the menu button, which is a user's operation. Therefore, button pressing and menu opening can be regarded as a series of operation series.

Therefore, in addition to the menu button pressing sound, the user operation parameter (in this case, click strength) is also used for pulldown menu open sound synthesis. Specifically, the operation parameter acquisition unit 2 is characterized in that the correspondence between the event information and the operation parameter related to the event information is held in advance, and the associated operation parameter is acquired according to the input of the event. To do. In this case, as a specific operation, for example, the operation parameter acquisition unit 2 acquires an operation parameter (click strength) for pressing the menu button, which is a user operation immediately before the pull-down menu open event occurs. On the other hand, the object attribute acquisition unit 3 acquires “the number of menu items” and “menu color” displayed in the pull-down menu as object attribute information. Then, the first and second parameter conversion units (44, 45) convert these operation parameters and object attribute information into first and second sound editing parameters. The sound editing device 43 uses the sound editing parameters to process and edit the sound data, and as a result, the menu button pressing sound and the pull-down menu opening sound are output.

In this embodiment, a mouse with a pressure-sensitive sensor is used as the input device 11 to acquire the click strength, but a mouse that employs a velocity measuring method can also be used. The velocity measurement method is a keystroke strength measurement method used for a keyboard of an electronic keyboard instrument. Specifically, it has a mechanism that holds two switches that are in contact with each other when the keyboard is normal (when not pressed) and when it is pressed, and measures the switching speed of the two switches. The keystroke speed has a strong correlation with the keystroke strength. Therefore, it can be considered that the higher the switch switching speed, the stronger the keystroke strength. As described above, the input device 11 may be any mouse that can indirectly measure the click strength.

It is also conceivable to use a mouse with scroll buttons or a mouse with wheels as the input device 11. The rotation speed of the wheel and the pressing pressure of the scroll button are used as operation parameters.

As the other input device 11, for example, as described in Japanese Unexamined Patent Publication No. 9-305286, a keyboard or a touch that can detect the presence / absence of key contact, the pressing pressure, the pressing speed, or the depth. It is also possible to use a touch panel that can detect pressure.

By utilizing the present invention, it is possible to output different operation reaction sounds depending on the type of the input device 11 (whether it is a mouse or a touch panel, etc.) even with the same operation.

In the first method, the input device 11 outputs its own "device type information" together with the operation parameters, and the operation parameters are output by the first parameter conversion unit 21.
Convert to sound editing parameters with. As a result, it becomes possible to process and edit the sound reflecting the type of the input device 11.

Alternatively, similarly, the event acquisition unit acquires "device type information" of the input device 11 together with the event information, and the basic reaction sound selection unit 41 is utilized by using both the information.
May select the basic operation reaction sound. By this,
The basic operation reaction sound can be selected in a form that reflects the type of the input device 11.

Further, the interactive system 12 having no GUI
The present invention can also be applied to the above. GUI
There is a personal computer interface for the visually impaired, etc., as a dialogue system without such information. For example, in October 1997, the 13th Human Interface Symposium Proceedings pp. 9
The dialogue system for visually impaired persons described in 7-102 "Sound Design for Interface -Screen Reader CV / SR for Blind Persons-" can be used. This interactive system holds a logical tree structure of GUI objects, and a user moves on the tree structure using a ten-key pad to operate a target object. At this time, the operation reaction sound (state change sound in the literature) is output in response to the operation, but here the operation reaction sound is processed and edited using the operation parameters and object attribute information (capacity and update date). By doing so, it is possible to output a sound that dynamically changes and follows the strength of the user operation or the attribute of the GUI object.

Regarding another operation of the operation reaction sound synthesizer 4 of the present embodiment, as shown in FIG. 8, the operation parameter and the first and second object attribute information are input to the parameter conversion section (44, 45). At the same time, it can be input to the basic reaction sound selection unit 41. Further, either the operation parameter or the object attribute information may be input only to the basic reaction sound selection unit 41.

At this time, the basic reaction sound selection unit 41 uses any combination of the input event information, operation parameters and object attribute information, event information and operation parameters, or event information and object attribute information. Then, basic operation reaction sound data is acquired from the operation reaction sound database 42.

As described above, by using not only the event information but also the operation parameter and the object attribute information for selecting the basic reaction sound, the operation reaction sound synthesizer 4 can generate the operation reaction sound which changes more dynamically. It will be possible.

As described above, in the parameter conversion executed by the first and second parameter converting units 44 and 45, for example, a stronger sound produces stronger sound, and a larger object produces lower and thicker sound. It is desirable to imitate the principle of sound generation in the real world such as. However, even if it is not imitation of the real world, it is possible to generate a new and effective operation reaction sound that reflects the operation intensity and operation speed of the user by linking with the operation of the user or the operation target object. .

[Second Embodiment] Next, the second embodiment of the present invention will be described.
Embodiments will be described with reference to the drawings.

FIG. 9 is a block diagram showing an example of the configuration of the second embodiment of the present invention.

Referring to FIG. 9, in the second embodiment, an input device 11 for the user to input to the dialogue system.
An interactive system 12 having a GUI for performing event-driven processing, an event acquisition unit 13 for acquiring an event that occurs in the interactive system 12, and an operation parameter acquisition unit 2 for acquiring an operation parameter output from the input device 11. And an object attribute acquisition unit 3 for acquiring information on a GUI object to be operated from the interactive system, and an operation reaction sound synthesizer 4 for synthesizing an operation reaction sound using event information, operation parameters, and object attribute information.
An animation generation device 5 for generating an animation using event information, operation parameters and object attribute information, an output synchronization control device 18 for adjusting output synchronization between audio data and video data, and an output for displaying video and audio. And device 19. Further, the same components as those in the first embodiment are designated by the same reference numerals, and those having the same functions as those in the first embodiment will not be particularly described in the present embodiment.

The animation generation device 5 is a device for accumulating various image data for animation and controlling animation reproduction. More specifically, the animation generation device 5 performs animation reproduction speed control, reproduction image size and position control, and image effect addition based on the control parameters.

The animation generating device 5 includes an animation database 52 for accumulating a group of image data for animation, a basic animation selecting section 51 for acquiring basic animation data from the animation database based on event information, and an operation parameter acquiring section 2. A third parameter conversion unit 54 that converts the input operation parameter into the first animation control parameter, and a fourth parameter that converts the object attribute information input from the object attribute acquisition unit 3 into the second animation control parameter. It is composed of a conversion unit 55 and an animation control unit 53 that controls the animation reproduction method (reproduction speed, position and size of reproduced image, image effect, etc.) based on animation control parameters.

The output synchronization control device 18 is a device for controlling the timing of output of sound and image, and synchronizes the input sound data and animation data and inputs them to the output device.

The output device 17 is a device capable of displaying both audiovisual information, and is composed of, for example, a display and a speaker.

The input device 11, the dialogue system 12, and the operation reaction sound synthesizer 4 are the same as those in the first embodiment. Event acquisition unit 13, operation parameter acquisition unit 2,
And the object attribute acquisition unit 3 has the same function as that of the first embodiment, but outputs each output to the animation generation device 5 in addition to the operation reaction sound synthesis device 4. More specifically, the event acquisition unit 13 outputs the output event information to the operation reaction sound synthesizer 4 and at the same time to the basic animation selection unit 51. The operation parameter acquisition unit 2 outputs the output operation parameter to the operation reaction sound synthesizer 4 and at the same time outputs it to the third parameter conversion unit 54. The object attribute acquisition unit 3 inputs the output object attribute information to the operation reaction sound synthesizer 4 and at the same time outputs it to the fourth parameter conversion unit 55.

[Embodiment] Next, the second embodiment will be described in detail with reference to a specific embodiment with reference to FIG. The example of FIG. 6 is an example relating to the menu button operation.

The user presses the menu button displayed on the menu bar using the input device 11 (for example, a mouse with a pressure sensitive sensor). The click signal and the click pressure at this time are output to the dialog system as operation parameters. The interactive system 12 interprets the operation from the operation parameter input from the input device and the position of the cursor, and generates an event (“menu button pressed” and “pull-down menu open”). The event acquisition unit 13 acquires these events and outputs them to the operation reaction sound synthesizer 4 and the basic animation selector 51 of the animation generator 5.

The operation parameter acquisition unit 2 is the interactive system 1.
The operation parameter (click pressure here) at the time of “pressing the menu button” is acquired via the input device 11 or via the input device 11, and is input to the operation reaction sound synthesizer 4 and the third parameter converter 54 of the animation generator 5. Output.

The object attribute acquisition unit 3 acquires the attributes of the menu buttons and pull-down menu objects to be operated from the dialogue system 12, and causes the operation reaction sound synthesizer 4 and the fourth parameter converter 55 of the animation generator 5 to acquire them. Output.

In the operation reaction sound synthesizer 4, as described in the first embodiment, the operation reaction sound corresponding to the input event information is processed and edited based on the operation parameter and the object attribute information. , Generates an operation reaction sound that dynamically adapts to the user's operation and the attributes of the object.

Next, the operation of the animation generating apparatus 5 will be described below by taking "menu open" as an example. When the user presses the menu button, the event information “menu open” is input to the basic animation selection unit 51. The animation database 52 stores image data representing changes in the GUI, but the basic animation selection unit 51 acquires the animation data for menu opening from the animation database 52 and outputs it to the animation control unit 53.

Here, the operation parameter acquisition unit 2 acquires the "click strength (pressure)" through the interactive system or directly from the input device. The third parameter conversion unit 54 converts the operation parameter (“click strength”) input from the operation parameter acquisition unit 2 into the first animation control parameter. In the present embodiment, it is assumed that the operation parameter “click strength” is converted into an animation control parameter for controlling the pull-down menu open speed and the pull-down menu display size.

On the other hand, the object attribute information acquisition unit 3 acquires the object attribute information, for example, “pull-down menu color” from the interactive system 12, and outputs it to the fourth parameter conversion unit 55. The fourth parameter conversion unit 55 converts these pieces of object attribute information into second animation control parameters. In this example, it is converted into an animation control parameter that controls the speed of opening the menu according to the “number of menu items”.

The animation control unit 53 controls the reproduction of the animation data input from the basic animation selection unit 51 according to the input first and second animation control parameters.

The video data whose reproduction is controlled by the animation control unit 53 is output to the output synchronization control device 18. Audio data is also input from the operation reaction sound synthesizer 4 to the output synchronization control device 18, but these data are output to the output device 19 in synchronization with the output of the video data and the audio data. The basic operation of the synchronization control is to control the reproduction start time of the audio data and the video data at the same time, but to perform more detailed synchronization control, the synchronization signal should be sent during the editing of the audio data and the video data. It is also possible to embed it and synchronize during reproduction based on this synchronization signal.

The output device 19 is a device such as a display and a speaker that can output and present video information and audio information, and outputs the input audio / video data.

In the case of this embodiment, for example, in the output operation reaction animation, the stronger the click and the smaller the number of menu items, the quicker the pull-down menu will open, and the stronger the click, the larger the menu display size will be displayed. The font size of menu items is also displayed large). At the same time, the operation response sound is a menu-open sound with a high volume, a slightly low pitch, and a complicated timbre. On the contrary, when there are many menu items and the click is weak, the pull-down menu opens slowly and the menu display size also becomes small. At the same time, the operation reaction sound that is output has a low volume, a high pitch, and a light timbre. In this way, not only the operation reaction sound but also the operation reaction animation is dynamically changed in response to the operation condition and the attribute of the operation target object.

Next, FIG. 10 shows another example of the second embodiment. The dialogue system 12 is usually a GUI in the GUI management unit 120 included in the dialogue system 12.
Is being generated. In the example of the second embodiment described above, the method of generating an animation without using the GUI data generated inside the dialogue system 12 has been described. It is also possible to use.

Specifically, the animation generation device 5 includes an animation control unit 53 and a third parameter conversion unit 5
The animation data to be output to the animation control unit 53 is output from the GUI management unit 120 of the dialogue system 12.

The GUI of the dialogue system in the first and second embodiments is not limited to the two-dimensional GUI used in the current personal computers and video games. For example, the present invention can be applied to a dialog system having a three-dimensional interface, such as a virtual reality environment or augmented reality.

A gesture input device can also be used as the input device 11. For example, there is a device in which a user holds a light source in his / her hand, shoots the light source with a camera, and analyzes the movement of the light source to perform gesture recognition. Here, the movement of the light source is regarded as the movement of the human hand, and the movement patterns such as “raised hand” and “pointed to the right” are extracted from the movement of the light source, and those movement patterns are commanded to the dialogue system. The technology used as has been used. Usually, as described above, the motion of the light source is compared with a preset finite motion pattern, and the motion pattern to be matched with the motion of the light source is determined. At this time, more detailed information such as the moving speed of the light source is acquired as a continuous numerical value and is used as the operation parameter in FIGS. 1, 9 and 10, and the operation parameter in the gesture input is processed and edited. And can be used to control the animation.

In the virtual reality space, it is possible to directly operate by grasping, moving or throwing a virtually existing 3D object, but the present invention can also be used as a feedback generation method for such a direct operation. Is.

Further, the operation reaction sound producing apparatus described in the first and second embodiments can be realized by a computer generally called a personal computer or a workstation. In this case, each function of the operation parameter acquisition unit 2, the object attribute acquisition unit 3, the operation reaction sound synthesis device 4, the animation generation device 5, the GUI management unit 12, the event acquisition unit 13, and the output synchronization control device 18 described above is realized. Create a computer program to perform the CD-RO
The program is recorded on a recording medium represented by an M, a floppy (registered trademark) disk, or a semiconductor memory, and the computer reads out the recording medium on which this program is recorded so that each function is generated in the computer. Then, the operation reaction sound generation device of the present invention can be realized by a computer. Further, this computer program may be recorded in a recording device in the server, for example, or may be in the form of providing the program included in the server via a network.

[0107]

The first effect of the present invention is that the reality of the GUI operation is increased, and the user can obtain a more natural operation feeling. For example, if you click the push button quickly and lightly, a short and light button press sound is output, and if you drag a large file slowly, a heavy and loose friction sound is output,
By strongly clicking, a large menu or window opens, and weakly opening slowly, a small menu or window opens, so that the user can obtain a more natural operation feeling.

The reason is that by using both the detailed input information acquired by the input device and the attribute information of the object for the operation reaction generation, a plurality of operations such as strength and speed of the operation and size and shape of the object can be obtained. This is because the operation reaction sounds and operation reaction animations that reflect the factors and their changes are output.

When considering the real world, a reaction occurs according to a certain action. To give an example of the reaction sound, for example, if you hit a cup containing water with a stick, a "sound of hitting the cup" will be generated, but it is very difficult to make exactly the same sound more than once. Let's do it. This is because what kind of sound is produced depends on the "property of the beating side (property of the cup)" and the "property of the striking side (property of the rod and how to strike)". Even if you hit the exact same cup with the same stick, different sounds will be produced depending on the strength, the angle, and the position of the hit. Even if the same stick can be hit in the same way, different sounds will be produced depending on the type of cup and the amount of water in it. With regard to the state change that can be visually confirmed, there is a phenomenon that the deeper the depression, the deeper the depression, or the deeper the depression, even if pushed with the same strength.

That is, the operation reaction caused by a certain action changes variously depending on both the information of "the property of the subject of the action" and "the property of the subject of the action". Not only is the change in sound dramatic, but there are many cases in which the timbre, volume, and pitch change subtly, but the subtle change in sound that follows a subtle difference in action is the reality. Or it creates naturalness.

In the present invention, the mechanism of such an operation reaction is modeled, and various operation reaction generations are possible. The action in the dialogue system is G displayed on the display.
A user operates a UI object through an input device. The object attribute acquisition unit 3 acquires the “property of the object to be acted on” and the operation parameter acquisition unit 2 “property of the input device exerting the action and the action itself (type of input device, strength of operation, speed)” To get. Then, by using both of these pieces of information, the operation reaction is processed and edited and controlled, so that various operation reactions can be generated and output. As a result, various operation reactions are generated dynamically with respect to the operation and sensitive to the slight difference in the operation. As a result, the sense of reality and naturalness in the operation are improved.

The second effect of the present invention is that it is not necessary to produce and prepare many sounds in advance, although various operation reaction sounds and reaction animations are output.

The reason is that the basic operation reaction sounds and animations are processed and edited and controlled in real time by using the operation parameters and the object attribute information and output.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an operation reaction sound generation device.

FIG. 2 is a flowchart showing the operation of the operation reaction sound generation device.

FIG. 3 is a flowchart showing the operation of the operation reaction sound generation device.

FIG. 4 is an explanatory diagram of the first embodiment.

FIG. 5 is an explanatory diagram of the first embodiment.

FIG. 6 is an explanatory diagram of the first embodiment.

[Figure 7] Basic reaction sound selection table

FIG. 8 is another configuration diagram of the operation reaction sound synthesizer.

FIG. 9 is a configuration diagram of an operation reaction generation device (including an animation generation device).

FIG. 10 is another configuration diagram of the operation reaction generation device (including the animation generation device).

FIG. 11 is a block diagram showing an example of a configuration of a conventional operation reaction sound output device.

[Explanation of symbols]

2 Operation Parameter Acquisition Unit 3 Object Attribute Acquisition Unit 4 Operation Reaction Sound Synthesizer 5 Animation Generation Device 11 Input Device 12 Dialog System 13 Event Acquisition Unit 14 Operation Reaction Sound Selector 15 Operation Reaction Sound Database 16 Sound Output Device 18 Output Synchronization Control Device 19 Output device 41 Basic reaction sound selection unit 42 Operation reaction sound database 43 Sound editing device 44 First parameter conversion unit 45 Second parameter conversion unit 51 Basic animation selection unit 52 Animation database 53 Animation control unit 54 Third parameter conversion Part 55 Fourth Parameter Converting Part 71 Mouse with Pressure Sensitive Sensor 72 Push Button 73 Push Button 74 Operation Parameter and Object Attribute Information when Push Button is Pressed 75 Event Column of Basic Reaction Sound Selection Table 76 Basic Reaction Sound Selection Operating parameters of the operating parameters and object attribute information 78 icon operating parameters and object attribute information 79 when a menu button click while dragging during table operation response sound field 77 icon click and object attribute information 120 GUI manager

Continuation of the front page (56) Reference JP-A-5-197355 (JP, A) JP-A-7-116348 (JP, A) JP-A-7-334307 (JP, A) JP-A-8-76919 (JP , A) JP 8-227341 (JP, A) JP 8-234896 (JP, A) JP 8-278781 (JP, A) JP 8-329315 (JP, A) JP 9-16365 (JP, A) JP-A-9-265299 (JP, A) JP-A-10-31481 (JP, A) JP-A-10-39992 (JP, A) JP-A-10-40067 (JP, A) A) JP 10-143155 (JP, A) JP 11-44703 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 3/00 G06F 3/033-3 / 037 G06F 3/14-3/153 G06F 3/16 G10K 15/00-15/06

Claims (11)

(57) [Claims] 1. A graphical user interface
Acquiring the operation parameter from an input device capable of obtaining the operation parameter consisting of either one or both of the strength of the operation and the speed of the operation with respect to the interactive system, and a graphical user object to be operated.
The attribute information of the object output on the user interface and the event information for the operation are acquired, the operation reaction sound corresponding to the acquired event information is selected, and the operation parameter and the attribute information of the object are set. An operation reaction sound generation device characterized by editing and outputting the selected operation reaction sound based on the above. 2. An event-driven interactive system having a graphical user interface, and an input operation for the interactive system, which comprises either one or both of operation intensity and operation speed. An input device that outputs operation parameters and the operation pattern that the input device outputs in response to the input operation.
Obtained by interpreting parameters by the dialogue system
An event acquisition unit that acquires event information, an operation parameter acquisition unit that acquires operation parameters output from the input device, and an object that acquires attribute information of a graphical user interface object to be operated from the interactive system. An attribute acquisition unit, an operation reaction sound synthesizer that synthesizes an operation reaction sound based on the event information, the operation parameter, and the object attribute information, and an output that outputs the operation reaction sound synthesized by the operation reaction sound synthesizer An operation reaction sound generation device comprising: a device. 3. The operation reaction sound synthesizing device holds in advance an operation reaction sound database in which sound data is accumulated, and correspondence information between the event information and the sound data, and the operation reaction corresponding to the event information. A basic reaction sound selection unit that selects sound data from sound data stored in the operation reaction sound database; a first parameter conversion unit that converts the operation parameter into a first sound editing parameter; and the object attribute information. Has a function of editing and processing an acoustic waveform by various acoustic effectors, acoustic filters, equalizers, etc., and a second parameter conversion unit for converting the first acoustic editing parameter and the second acoustic editing parameter. Sound editing device for processing and editing the sound data selected by the basic reaction sound selection section based on the sound editing parameters of If the operation response sound generating apparatus according to claim 2, characterized in that they are composed of 4. The output device is capable of outputting video and audio, and further using the event information, the operation parameter and the object attribute information, animation reproduction speed control, reproduction image size and position An animation generation device that controls animation reproduction such as control and addition of an image effect; acoustic data of the operation reaction sound synthesized by the operation reaction sound synthesis device; and image data in animation reproduction controlled by the animation generation device. The operation reaction sound generation device according to claim 2 or 3 , further comprising: an output synchronization control device that performs synchronization adjustment. 5. The animation generation device comprises: an animation database for accumulating a group of image data for animation; a basic animation selection unit for obtaining basic animation data from the animation database based on the event information; A third animation conversion parameter, a third animation conversion parameter, a third animation animation parameter, a fourth animation animation parameter, a second animation animation parameter, and a second animation animation parameter. reproduction speed of animation reproduction based on the animation control parameter, the operation of claim 4, position and size of the reproduced image, and animation control unit for performing control such as image effect, characterized in that it is composed of応音 generating device. 6. The third parameter conversion unit, wherein the animation generation device converts the operation parameter into a first animation control parameter, and a fourth parameter that converts the object attribute information into a second animation control parameter. A conversion unit; and an animation control unit that controls the reproduction speed of animation reproduction, the position and size of the reproduced image, the image effect, etc., based on the first animation control parameter and the second animation control parameter. The operation reaction sound generation device according to claim 4 , wherein the animation control unit controls animation reproduction for animation data acquired from the dialogue system. 7. An event-driven interactive system having a graphical user interface, an input operation for the interactive system, and an intensity of the operation,
Alternatively, a recording medium readable by a computer system including at least an input device that outputs an operation parameter consisting of one or both of operation speeds and an output device that outputs the operation parameter to a user, , The operation pattern output from the input device by the input operation.
Obtained by interpreting parameters by the dialogue system
An event acquisition function that acquires event information, an operation parameter acquisition function that acquires an operation parameter output from the input device, and an object that acquires attribute information of a graphical user interface object to be operated from the interactive system. An attribute acquisition function, and an operation reaction sound synthesizing function for synthesizing an operation reaction sound based on the event information, the operation parameter, and the object attribute information and outputting the operation reaction sound to the output device. A recording medium recording an operation reaction sound generation program. 8. An operation corresponding to the event information, wherein the operation reaction sound synthesizing function holds in advance an operation reaction sound database function in which sound data is accumulated, and correspondence information between the event information and the sound data. A basic reaction sound selection function of selecting reaction sound data from sound data stored in the operation reaction sound database; a first parameter conversion function of converting the operation parameter into a first sound editing parameter; and the object attribute. It has a second parameter conversion function of converting information into a second acoustic editing parameter, and a function of editing and processing an acoustic waveform with various acoustic effectors, acoustic filters, equalizers, and the like. The sound data selected by the basic reaction sound selection unit is processed and edited based on the sound editing parameter of 2. Recording medium recorded with claim 7, wherein the operation response sound generation program, wherein the sound editing function, in that it consists of. 9. The output device is capable of outputting video and audio, and the computer is further configured to utilize the event information, the operation parameter, and the object attribute information to control the animation playback speed and the size of the playback image. An animation generation function for controlling animation playback such as control of pod position and addition of image effect, acoustic data of operation reaction sound synthesized by the operation reaction sound synthesis function, and animation reproduction controlled by the animation generation function. 9. A recording medium recording the operation reaction sound generation program according to claim 7 , which realizes an output synchronization control function for adjusting synchronization of image data. 10. The animation generation function, an animation database function for accumulating a group of image data for animation, a basic animation selection function for acquiring basic animation data by the animation database function based on the event information, and the operation. A third parameter conversion function for converting a parameter into a first animation control parameter, a fourth parameter conversion function for converting the object attribute information into a second animation control parameter, the first animation control parameter and the An animation control function for controlling the reproduction speed of the animation reproduction, the position and size of the reproduced image, the image effect, etc. based on the second animation control parameter. A recording medium recording the operation reaction sound generation program to claim 9. 11. A third parameter conversion function for converting the operation parameter into a first animation control parameter, and a fourth parameter for converting the object attribute information into a second animation control parameter, by the animation generation function. And a conversion function, and an animation control function for controlling a reproduction speed of animation reproduction, a position and a size of a reproduced image, an image effect and the like based on the first animation control parameter and the second animation control parameter. The recording medium storing the operation reaction sound generation program according to claim 9 , wherein the animation control function controls animation reproduction for animation data acquired from the dialogue system.