JP2003296010A - Method of generating effective sound in pen input operation, terminal unit and two-way interactive system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method of generating effective sound in the pen input operation and the like for easily organizing or sharing an interactive communication environment between interlocutors. <P>SOLUTION: The pen input operation is performed via an input interface 920 with a prescribed input surface 940 for taking in the information on a coordinate on the input surface 940 equivalent to the contact point between an input pen 930 and the input surface 940 as an electric signal. The presence of the electric signal from the input interface 920 is detected with a prescribed sampling cycle, and the operation of the input pen 930 is specified based on the result of detection of the electric signal. The effective sound corresponding to the specified operation of the input pen 930 is selected from an effective sound group previously prepared by type of operation of the input pen 930 and is reproduced and outputted. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound effect of a pen input operation through an input interface for taking in coordinate information on an input surface corresponding to a contact point between a predetermined input surface and the input pen as an electric signal. The present invention relates to a generation method, a terminal device, a two-way interactive system, and the like.

[0002]

2. Description of the Related Art In recent years, with the spread of information processing devices such as personal computers, personal learning and counseling using these information processing devices have come to be performed. Further, with the rapid spread of communication infrastructure, a form of distance learning between a teacher and one or more students is steadily established between remote places via transmission means such as English conversation and supplementary training. In such distance learning, the teacher and the student proceed with the learning while watching the image of the interlocutor. Therefore, for example, it is suitable for the distance learning performed through the transmission means disclosed in Japanese Patent No. 3074677. An interactive system is known.

On the other hand, in distance learning as described above,
Teachers need to provide students with additional information such as presentation of tasks, scoring, and evaluation. For example, Japanese Patent No. 2970
As disclosed in Japanese Patent No. 661, there has been proposed a display method capable of presenting not only the image of the interlocutor but also various related information.

[0004]

DISCLOSURE OF THE INVENTION The inventor has found the following problems as a result of detailed examination of the above-mentioned prior art. That is, in a conventional interactive interactive system or the like, which uses an information processing device alone or indirectly through a transmission means (including learning, counseling, presentation, etc.), a monitor placed in front of the interlocutor. Multiple windows are displayed on the screen. These windows include various windows such as a window for the image of the other party's interlocutor, a window for slide display, and a whiteboard window for pen input. In particular, in an indirect dialogue via the monitor screen, the operation sound itself of the pen input, such as a pen input to a window displayed on the monitor screen, cannot be heard, which gives the interlocutor a feeling of strangeness. For example, in an actual lesson (a direct dialogue between the teacher and the student), the writing sound that occurs when the teacher draws letters with chalk on the blackboard is also an important factor in constructing a lesson-teacher interactive environment. Despite the elements, the conventional interactive interactive system lacks the acoustic environment related to the input actions of these interlocutors themselves.

The present invention has been made in order to solve the above-mentioned problems, and it provides acoustic data that contributes to the construction of a dialogue environment, directly or indirectly, especially the operation sound of pen input made by the person himself / herself. By providing the interlocutors with sound effects for each type of operation, a sound effect generation method for pen input operation that facilitates the construction or sharing of a dialogue environment among the interlocutors,
An object of the present invention is to provide a terminal device that realizes it, a two-way interactive system including the terminal device, a computer program that realizes a sound effect generation method of a pen input operation, and a recording medium in which the computer program is recorded. I am trying.

[0006]

A method for generating a sound effect of a pen input operation according to the present invention is particularly suitable for a sound input of a pen input made by an interlocutor among acoustic data which is an important factor in constructing a dialogue environment. Is supplied to the interlocutors as a sound effect. The sound effects provided in this way that correspond to the operation of the interlocutor effectively direct the psychological construction of the interactive scene in which they are placed to the interlocutor, not only visually but also auditorily. ,to support.

Specifically, the method for generating a sound effect of a pen input operation according to the present invention has a predetermined input surface and stores coordinate information on the input surface corresponding to a contact point between the input surface and the input pen. It is a method of generating a sound effect of a pen input operation via an input interface for capturing as an electric signal. In particular, the method detects the presence or absence of an electric signal from the input interface at a predetermined sampling period, and specifies the operation of the input pen based on the detection result of the electric signal. And
The sound effect corresponding to the specified operation of the input pen is selected from a sound effect group prepared in advance for each operation type of the input pen, and reproduced and output. The selected sound effect is reproduced and output when the operation of the input pen is specified or while the specified operation is continued.

In the sound effect reproducing method for pen input operation according to the present invention, at least the operation of the input pen is
An operation of bringing the input pen into contact with the input surface and an operation of moving the contact point while the input pen is in contact with the input surface are included. Furthermore, in the sound effect reproducing method of the pen input operation according to the present invention, different sound effects may be reproduced and output according to the moving speed level of the contact point on the input surface.

As described above, by appropriately providing the sound effect of the operation performed by the interlocutor during the dialogue, the interlocutor is placed on the interlocutor not only in the visual aspect but also in the auditory aspect. It effectively directs and supports the psychological construction of conversational situations. Further, by providing the sound effect corresponding to the action of the interlocutor himself, it becomes easy to convey the emotion of the interlocutor and the atmosphere of the dialogue environment to the other interlocutor,
The building of a common thinking space (dialogue environment) among the interlocutors is promoted.

The above-described effect sound generation method for pen input operation can be realized by an information processing device such as a personal computer which has been rapidly spread. In particular, the effect sound generation method for pen input operation has a predetermined method. The terminal device in the communication system that mutually transmits and receives the video data and the audio data via the transmission means also functions as an information processing device that realizes the effect sound generation method of the pen input operation according to the present invention.

That is, the terminal device according to the present invention includes a display unit capable of multi-window display, an input interface capable of pen input, and a data base (hereinafter, referred to as D / Called B)
And an output unit for outputting a sound effect, and a control unit. The display section displays one or more windows (multi-window display). The input interface is
It has a predetermined input surface and outputs coordinate information on the input surface corresponding to a contact point between the input surface and the input pen as an electric signal. The output unit is a speaker, a headphone, an earphone, or the like that outputs a sound effect corresponding to the operation of the input pen. The D / B stores a plurality of types of sound effect data prepared in advance for each operation of the input pen.
The control unit causes the output unit to reproduce and output the sound effect corresponding to the operation type of the input pen.

In particular, the control unit specifies the operation of the input pen based on the electric signal from the interface detected at a predetermined sampling period, and outputs the sound effect data corresponding to the specified operation to the D / B. The sound effect data thus read is reproduced and output by an output unit such as a speaker. In addition, the control unit, as an operation of the input pen,
At least, an operation of bringing the input pen into contact with the input surface and an operation of moving the contact point while the input pen is in contact with the input surface are specified.

In the terminal device according to the present invention, it is preferable that the data base stores a plurality of types of sound effect data prepared for each moving speed level of the contact point on the input surface. This is because it is possible to select a sound effect that is suitable for various presumed dialogue environments. Also,
A plurality of types of sound effect data prepared for each window displayed on the display unit may be stored in the database.

Further, the terminal device having the above-mentioned structure can be applied to a two-way interactive type system which realizes a remote place interactive via a predetermined transmission means. In particular, such a two-way interactive system is suitable for directing and supporting a dialogue scene that changes from time to time for each interlocutor, both visually and audibly. Then, in order to prepare sound effects suitable for various dialogue scenes, the D / B in the terminal device is
Prepared for at least one of the operator of the input pen and the terminal device included in the two-way interactive system,
It is preferable that a plurality of types of sound effect data are stored.

The window display method as described above may be a program executed by a computer or the like, and in this case, the program may be distributed via a network whether wired or wireless. Also, C
It may be stored in a recording medium such as a D, a DVD, or a flash memory.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of a method for generating a sound effect of a pen input operation, a terminal device, a two-way interactive system including the same, etc. according to the present invention will be described in detail below with reference to FIGS. explain. In the description of the drawings, the same parts will be denoted by the same reference symbols and redundant description will be omitted.

A terminal device according to the present invention is an information processing device such as a personal computer, and realizes the effect sound generation method of a pen input operation according to the present invention. The terminal device may be a stand-alone type used by one interlocutor, and realizes a remote dialogue between terminal devices connected via a predetermined transmission means, for example, Japanese Patent No. 30744677. It is also applicable to the two-way interactive system described in.

First, the bidirectional interactive system to which the terminal device is applicable will be described. Note that this two-way interactive system is a system that realizes remote dialogue in a state where the line-of-sight coincides with each other. Any interactive interactive display system can be applied.

FIG. 1 is a logical configuration diagram showing the entire interactive interactive system that enables line-of-sight matching between interlocutors at remote locations. This system realizes a two-way dialogue between an interlocutor 3A (first interlocutor) at point A (first point) and an interlocutor 3B (second interlocutor) at point B (second point). It is a system that does. In particular, this system is a counseling system that establishes intimate relationships of trust even with interlocutors in remote areas and enables close communication, and a system that aims at home learning. , Is supposed to be a counselor such as a psychologist. Therefore, this system has various functions and configurations so that these counselors can provide information necessary for counseling and an ideal counseling environment.

Specifically, at the point A, (a) a chair 4A on which the interlocutor 3A (for example, counselor) sits (having a height adjusting mechanism 40A), and (b) a table 2A to which the interlocutor 3A faces, (C) The terminal device 20A having a mechanism for matching the line of sight between the interlocutors via the transmission means 27 is installed, and the interlocutor 3A is isolated by the partition 22A. In addition, the interlocutor 3A has a CC for capturing an image of the hand separately from the CCD camera 5A for capturing the interlocutor.
A D camera 910A is attached. On the other hand, at the point B, (a) a chair 4B on which an interlocutor 3B (for example, a client who receives counseling) sits and a (height adjusting mechanism 40)
B), (b) Table 2B where the interlocutor 3B faces
And (c) a terminal device 20B equipped with a mechanism for matching the lines of sight between the interlocutors via the transmission means 27, and the interlocutor 3B is isolated by the partition 22B. In addition, the interlocutor 3B also has a CCD for imaging the interlocutor.
In addition to the camera 5B, a CCD camera 910B for picking up an image of the hand is attached. Here, the transmission means 27
Is the terminal device 20A at the point A and the terminal device 20B at the point B
An optical fiber transmission line 31 (main transmission line) and a satellite communication transmission line 32 (backup transmission line) that enable large-capacity transmission are included in order to enable transmission and reception of image information and audio information to and from. The transmission means 27 may be wired or wireless. In addition, the transmission means includes a network such as a telephone line network that has already been installed, and an electronic bulletin board system (BBS: Bulle
tin Board System).

At point A, the chair 4A described above functions to define the position of the interlocutor 3A. This chair 4A is preferably fixed in order to keep a constant distance from the display unit 21A of the terminal device 20A. However, even if the chair 4A is not fixed, the chair 4A can be installed by installing the table 2A on the interlocutor 3A and the display unit 21A.
It becomes possible to maintain a constant distance from the display unit 21A (the fact that the existence of the table 2A serves as a reference for the psychological distance of the interlocutor 3A has been confirmed). The configuration of the terminal device 20B at the point B where the other party's interlocutor 3B (for example, a person who receives counseling) is the same.

At the point A, the terminal device 20A (1)
A monitor TV arranged on the table 2A for displaying the image of the interlocutor 3B based on the image information transmitted from the terminal device 20B at the point B via the transmission means 27.
Display unit 21A equipped with 1A, (2) video of interlocutor 3A taken by CCD camera 5A and CCD camera 910A
The image pickup unit 6A for taking in each of the images captured at step 11 as image information and transmitting the image information to the terminal device 20B at the point B, and (3) being transmitted from the terminal device 20B at the point B via the transmission means 27. Voice output unit 23A for outputting the voice of the interlocutor 3B based on the received voice information
(Speaker), and (4) a voice input unit 24A (microphone) for collecting the voice of the interlocutor 3A as voice information and transmitting it to the terminal device 20B at the point B. In addition, this system, as a counseling system, realizes more detailed communication.
Input a character or a graphic, display the input character or graphic on the display unit 21A, and transmit the character or graphic information to the terminal device 20B at the point B as a character or graphic information (keyboard, pointing, (Device, interface such as a touch panel), and (6) a video processing device 26A provided between each of the above-described components and the transmission means 27 and performing signal processing, transmission control, video data editing work, and the like. The terminal device 20B at the point B also
The terminal device 20A at the point A has the same configuration as that of the terminal device 20A. Specifically, the image pickup device 600B including the display unit 21B, the CCD camera 5B and the image pickup unit 6B, the voice output unit 23B, the voice input unit 24B, and the character graphic. The input unit 25B and the video processing device 26B are provided.

Further, the terminal devices 20 at the points A and B
In A and 20B, the configurations of the imaging devices 600A and 600B (imaging means) will be described. For convenience of description, when referring to components common to the terminal devices 20A and 20B at points A and B, for example, the imaging device 60
The description will be made by omitting the letters A and B that distinguish each point like 0. In addition, unless specifically described, the terminal device 20A at the point A is described in principle, and the terminal device 20 at the point B having a common configuration is described.
Overlapping description of B is omitted.

This image pickup device 600 includes a microminiature CCD camera 5 (5A) as an image pickup section, and a support mechanism 10 (FIG. 2) for installing the CCD camera 5 in a supported state at a predetermined position.
Reference unit) and an image pickup unit 6 (6A) for controlling the CCD camera 5. The monitor TV 1 (1A) is placed on the table 2 (2A), and the interlocutor 3
(3A) is a chair 4 (4) provided with a height adjusting mechanism 40 (40A) placed at a distance L (m) from the monitor TV 1.
Sitting in A). This distance L (LA) is set to 0.5 m or more and 5 m or less. An outer diameter of 20 mm or less and a length of about 100 mm between the interlocutor 3 and the monitor TV 1 to be photographed.
A cylindrical microminiature CCD camera 5 is provided. Further, the CCD camera 5 is installed so that its imaging direction is directed to the interlocutor 3 who is imaged. Then, the image of the interlocutor 3 imaged by the camera 5 is transmitted as image information from the imaging unit 6 to the other interlocutor side (transmitted to the terminal device 20B at the point B via the transmission means 27).
Depending on the interlocutor, the space between the display unit 21 and the interlocutor may be weakened. Therefore, in order to cope with such a situation, it is preferable that the CCD camera 5 has a deep focal depth.

Next, a specific installation position of the microminiature CCD camera 5 will be described with reference to FIG.

First, the image pickup unit 6 of the image pickup apparatus 600 is installed on the display section 21 for displaying the transmitted image of the other interlocutor. An image of the interlocutor himself / herself taken by the camera 5 is captured in the body of the imaging unit 6 (captured by the imaging unit 6 via the camera cable 50)
In addition to the changeover switch 6a, various adjustment switches 6b are provided for switching between the display of 1) and the display of the transmitted image of the interlocutor 3B. The switch 210 includes a power switch for turning on / off the power of the monitor TV 1. The CCD camera 5 is installed by a support mechanism 10 (robot arm) at a predetermined position in front of the monitor screen 8 on which the transmitted image 9 of the interlocutor 3B is displayed. The monitor screen 8 can be sufficiently used if it has a size of about 4 inches.

Specifically, the CCD camera 5 is located at a position forward w (cm) from the monitor screen 8, and the monitor screen 8
It is arranged near the head 9a of the interlocutor image 9 displayed in FIG.
The central axis of the cylindrical CCD camera 5 is located at a position h (cm) above the eye position of the interlocutor image 9 shown by the dotted line.

As described above, since the microminiature CCD camera 5 is located near the head above the eye position of the interlocutor image 9 displayed on the screen 8 of the large monitor TV 1, it is particularly suitable for two-way dialogue. No trouble will occur. For example, the CCD camera 5 is provided above the eye position (the position shown by the dotted line in the figure) of the interlocutor image 9 displayed on the screen 8 at h = about 10 (cm), and between the monitor TV 1 and the interlocutor 3. In the case of the system configuration in which the distance L is set to about 2.5 (m), the parallax angle is 2.3 °.
It is possible to sufficiently fall below the detection limit parallax angle of 3 ° (the interval w between the monitor screen 8 and the CCD camera 5 is 10 (c
Even if there is about m), there is no particular effect on the variation of the parallax angle). That is, if the eyes of the other party (the eyes of the image of the other interlocutor 3B displayed on the monitor screen 8) can be clearly seen at the time of line-of-sight matching, even if the microminiature camera 5 is provided near the head, the monitor It has been confirmed that if the screen 8 of the TV 1 is large, there is almost no hindrance to the interactive dialogue. As a result of the experiment, the size of this screen 8 is 35 cm in width and 26 cm in height.
It has also been confirmed that good two-way dialogue can be realized if the size is at least a certain level. It should be noted that if mutual interlocutors are acquaintances, there is a result that there is a tendency that there are few psychological obstacles even if the size of the screen is small, and it is advisable to properly use the screen size according to the application.

Further, the parallax angle will be described with reference to FIG. In this specification, the parallax angle means, as shown in FIG. 3, the line of sight of the interlocutor 3 to be photographed, which is directed to the interlocutor image 9 displayed on the monitor screen 8, and the direction of the CCD camera 5. The angle θ formed by the line of sight of the interlocutor 3 is defined. In other words, it means the shift angle of the installation position of the CCD camera 5 with respect to the line of sight of the interlocutor 3. In addition, the line-of-sight matching here means the 4th Japan Electrical Engineers Union Meeting (N)
o. 1998), it means that the unnaturalness due to the installation position of the CCD camera 5 is below the allowable limit. Quantitatively, the parallax angle is 4.5 ° or less in the left and right direction (both temple sides of the image 9 with respect to the eye center A of the interlocutor image 9), and 12 ° directly above (with respect to the image center A, The head 9b side of the image 9) and 8 ° below (the body side of the image 9 with respect to the image center A) or less are within the line-of-sight matching range. Of course, the smaller the parallax angle, the better, and the detection limit is 3 ° or less in the horizontal and vertical directions. Therefore, the area in which the CCD camera 5 can be installed is the conical area shown in FIG. In addition,
This conical area is the center point A of the interlocutor image 9 displayed on the monitor screen 8 and the eye 30 of the interlocutor 3 (in this embodiment, the center of both eyes of the interlocutor image 9 is the center point of the interlocutor image 9). Defined as A), the line of sight 31 of the interlocutor 3 connecting
It is defined by the line-of-sight matching region 9b on the monitor screen 8 and the position of the eye 30 of the interlocutor 3 when the monitor is separated by a predetermined distance L (m). If the size of the interlocutor image 9 displayed on the monitor screen 8 is small, the CCD camera 5 and the head of the interlocutor image 9 do not necessarily have to overlap each other. Rather, the head of the interlocutor 9 is hidden by the CCD camera 5 and becomes difficult to see. As described above, the condition for line-of-sight matching is that the CCD camera 5 and the eyes of the interlocutor image 9 are present in the area with a parallax angle of 3 °, and specifically, about 3 cm at a position 50 cm away from the screen.
The degree (h in FIG. 2). Therefore, within this range, the line-of-sight matching state is maintained even if the CCD camera 5 and the head of the interlocutor image 9 are displaced. This interlocutor image 9 and CC
The D camera 5 can be displaced by, for example, adjusting the image display position on the monitor screen 8 or adjusting the support portion of the CCD camera 5 to change the position of the CCD camera 5 itself.

FIG. 5 is a view showing the external appearance of a terminal device for realizing the effect sound generation method for pen input operation according to the present invention. In the following description, the terminal device is applied to the two-way interactive system shown in FIG. 1, but the terminal device has a configuration corresponding to both the terminal devices 20A and 20B of FIG. Therefore, the subscripts are omitted for the reference numerals of the respective components.

In this terminal device, an image pickup device 600 for picking up an image of the interlocutor 3 located in front of the display part 21 is arranged on a display part 21 having a monitor screen 8 on which a plurality of windows are displayed. There is. The image pickup device 600 includes a CCD camera 5 which is an image pickup unit, a support unit 10 which holds the CCD camera 5 at a predetermined position, and an image pickup unit 6 which takes in an image picked up by the CCD camera 5 via a cable 50. The area indicated by A1 in FIG. 5 is an imaging area of the CCD camera 5.

On the other hand, the interlocutor 3 inputs characters and figures by bringing the input pen 930 into contact with the tablet 920 which is the input interface of the terminal device. In addition, the interlocutor 3 uses the headphones 9 for listening to the sound effect output according to the type of pen input operation performed by the interlocutor 3.
10 (or even a speaker) is attached.

Specifically, as shown in FIG. 6, this terminal device includes a display unit 21 such as a monitor (corresponding to the display units 21A and 21B in the bidirectional interactive system shown in FIG. 1). The image processing device 26 (corresponding to the image processing devices 26A and 26B in the bidirectional interactive system shown in FIG. 1), the external recording device 700, a pointing device 900 such as a mouse, a tablet 920 as an input interface, and pen input. An output unit 920 (headphones, speakers, etc.) for outputting acoustic data such as a male difference sound effect, and an imaging device are provided. The imaging device is composed of the imaging unit (CCD camera) 5 and the imaging unit 6 for imaging the interlocutor as described above.

The video processing device 26 takes in image information from the image pickup unit 6 included in the image pickup device and transmits / receives image information such as the image of the interlocutor on the other side through the transmission means 27. 261 (I / O in the figure),
In addition to the control unit 263 for controlling the window display operation and the sound effect output operation of the pen input operation performed by the interlocutor, the transmitted and received image information, character data, video data, and sound effect data prepared by the interlocutor himself / herself in advance. A database 262 (D / B in the figure) for storing information such as
A drawing unit 264 that displays a predetermined pattern in accordance with an instruction from the control unit 263, and data input / output that enables the reception of position information from the pointing device 900 such as a mouse and the tablet 920, and the exchange of data with the external recording device 700. A unit 265 (I / O in the figure), a tablet 920 that is an interface for pen input, and a sound source controller 950 for outputting acoustic data to an output unit 910 such as a speaker. The external recording device 700
Includes, for example, a magnetic tape, a CD, an optical disk, a hard disk, a DVD, a flash memory and the like. Also, D /
B262 stores sound effect data corresponding to a pen input operation performed by the interlocutor during the dialogue.

Specifically, the I / O 2 is transmitted via the transmission means 27.
When 61 receives the image data of the other party's interlocutor, the control section 263 displays the other party's interlocutor image on the monitor screen 8 of the display section.
Show window on top. That is, the control unit 263 first instructs the drawing unit 264 to display the monitor screen 8 of the display unit 21.
A plurality of windows are displayed at a position designated in advance by the mouse 900 or the like. At this time, one of the windows displayed on the monitor screen 8 is the received image of the other party's interlocutor, and the other windows are windows for presenting auxiliary information for dialogue progress, inputting characters, and the like. . In the display control by the control unit 263, the display position is set so that the head above the eyes of the other-side interlocutor image and the CCD camera (imaging unit) 5 previously installed in front of the monitor screen of the display unit 21 overlap. Is adjusted, and the line-of-sight matching state via the transmission means 27 is secured.

During the dialogue while displaying the image of the other party's interlocutor as described above, the control section 263 causes the output section 910 to reproduce and output the sound effect of the pen input operation performed by the interlocutor during the dialogue. The sound source controller 910 is controlled. Note that FIG. 7 shows a tablet 9 as an input interface used when the interlocutor inputs characters and figures.
20 is a perspective view showing the appearance of 20. FIG. This tablet 920
Is provided with an input surface 940 for detecting coordinate information of a point in contact with the input pen 930 as an electric signal. The control unit 263 moves the input pen 930 to the input surface 94.
When a pen input operation such as moving the contact point in the state of being in contact with 0 is performed, the monitor screen 8 is displayed based on the contact point coordinate information on the input surface 940 identified from the electric signal from the tablet 920. A desired character or the like is displayed in the displayed window (whiteboard window or the like in which handwritten characters can be input).

Specifically, the sound effect generating operation in the control unit 263 (the sound effect generating method of the pen input operation according to the present invention) is carried out, for example, as shown in the flow chart shown in FIG.

That is, the control unit 263 controls the I / O 265.
An electric signal sent from the tablet 920 via the (the input pen 9 on the input surface 940 of the tablet 920).
Presence / absence of coordinate information of a contact point with 30) is detected at a predetermined sampling period Δt (step ST1).

When the electric signal indicating the coordinate information of the contact point between the input surface 940 and the input pen 930 is detected from the tablet 920, the control unit 263 first detects the electric signal continuously from the previous sampling time. It is confirmed (step ST2). This is based on the detection result of the electric signal from the tablet 920, at least the input surface 940.
When the input pen 930 is brought into contact with the input surface 940,
This is for distinguishing the operation of moving the contact point with the input pen 930 in contact with the upper part (drawing operation by the interlocutor).

If the control section 263 has not detected the electric signal from the tablet 920 at the time of the previous sampling,
It is determined that it is the time when the input pen 930 contacts the input surface 940, and the sound effect data corresponding to the contact operation between the input surface 940 and the input pen 930 is selected from the sound effect data stored in the D / B 262 in advance. Read out. Then, the control unit 263 controls the sound source controller 950 to cause the output unit 910 to reproduce and output the read effect sound data. For example, in this contact operation, a sound effect “pi” is reproduced and output.

On the other hand, the control unit 263 controls the tablet 920 in the current sampling as in the previous sampling.
When an electric signal is detected from the above, it is determined that the drawing operation is performed subsequent to the above-described contact operation, and the moving speed of the contact point between the input surface 940 and the input pen 930 is calculated (step S).
T6). The calculation of the moving speed is performed by first calculating the distance ΔD (= ((X2-X2) between the contact point coordinate (X1, Y1) detected last time and the contact point coordinate (X2, Y2) detected this time.
1) ² + (Y2-Y1) ² ) ^1/2 ) is calculated (step ST5). Since the moving distance ΔD calculated in this way is the moving downward distance during the sampling period Δt, the moving speed ΔV of the contact point is given by ΔD / Δt.

In the D / B 262, the moving speed ΔV calculated as described above is divided into a plurality of levels, and the sound effect data corresponding to each speed level is stored.
The control unit 263 reads the sound effect data corresponding to the speed level calculated from the sound effect data stored in the D / B 262, and the read sound effect data is output by the output unit 9.
The sound source controller 950 is controlled so as to be reproduced and output at 10 (step ST6). For example, the control unit 263
When the moving speed level of the contact point on the input surface 940 is low (when the drawing speed of the input pen 930 is low),
When the moving speed level of the contact point on the input surface 940 is high, the low-pitched sound effect “Shoo” is reproduced and output.

The effect sound corresponding to the moving speed level of the contact point is stored in the D / B 262 in advance as described above.
It can also be generated by adjusting the pitch and tone color of the reference sound effect data with the sound source controller 950. Of course,
The sound effect of such a pen input operation is an environmental sound necessary for constructing a dialogue environment, but at a volume that makes it difficult to hear the dialogue and the BGM reproduced during the dialogue, each interlocutor feels uncomfortable. Therefore, it is preferable that the volume is as natural as possible for the interlocutor to recognize.

FIG. 9 shows a display state of the monitor screen 8 under the multi-window display control by the control unit 263. On the monitor screen 8 shown in FIG. 9, a window 800A for displaying the image of the other party's interlocutor is displayed so as to overlap with the CCD camera 5 (line-of-sight match state), while the tablet 92 is displayed.
A whiteboard window 800B in which handwritten figures and characters by a pen input operation from 0 are directly displayed is displayed.

As described above, a plurality of windows are displayed on the monitor screen 8 of the display unit 21, and the displayed windows also affect the psychological dialog environment given to the interlocutor. Therefore, if the sound effect data stored in the D / B 262 is prepared corresponding to the window that matches the subject of the dialogue that changes from time to time, or if the pitch and tone color are adjusted by the sound source controller 950, different dialogue scenes can be created. Easy to build. It should be noted that the variation of the sound effect variation such as the pitch and the tone color adjusted by the sound source controller 950 may be selected by the interlocutor himself, and in this case, a more individual dialogue effect is possible.

Further, FIG. 10 is a diagram showing a two-way interactive system to which a terminal device for realizing the above-described effect sound generation method for pen input operation is applied. In the terminal device in which one of the interlocutors 3A exists, like the two-way interactive system in FIG. 10, a tablet 920A is provided as an input interface, and the interlocutor 3A uses the input pen 9
By bringing 30A into contact with the input surface 940A of the tablet 940A, handwritten information such as desired characters and figures is sent to the video processing device 26A. The video processing unit 26A outputs a sound effect according to the type of pen input operation of the interlocutor 3A, and also transmits it to the terminal device of the interlocutor 3B via the transmission means 27.

On the other hand, in the terminal device of the other party's interlocutor 3B,
Similarly, the tablet 920 is used as an input interface.
B is provided, and the other party's interlocutor 3B uses the input pen 93
By touching 0B with the input surface 940B of the tablet 940B, handwritten information such as desired characters and figures is sent to the video processing device 26B. The video processing unit 26B outputs a sound effect according to the type of pen input operation of the other party's interlocutor 3B and also transmits it to the terminal device of the interlocutor 3A via the transmission means 27. Areas CA1 and CA2 in FIG. 10 are imaging areas of the CCD cameras 5A and 5B in the terminal devices of the interlocutors 3A and 3B, respectively.

The data base in the terminal device of each interlocutor is prepared for at least one of the operator of the input pen (interlocutor himself) and the terminal device included in the interactive system. Plural types of sound effect data may be stored.

[0049]

As described above, according to the present invention, a different sound effect is generated according to a pen input operation performed by the interlocutor himself / herself during the dialogue, so that the interlocutor can see not only the visual aspect but also the auditory aspect. , It effectively directs and supports the psychological construction of the dialogue scene in which the person is placed. Also,
By providing sound effects corresponding to the actions of the interlocutors themselves, the emotions of the interlocutors and the atmosphere of the dialogue environment can be easily communicated to the other interlocutors, and the common space (interaction This has the effect of promoting the formation of the environment).

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a two-way interactive system that realizes line-of-sight matching between remote places via transmission means.

FIG. 2 is a diagram showing the image pickup device shown in FIG. 1, and a diagram for explaining a method of installing a CCD camera for line-of-sight matching.

FIG. 3 is a diagram illustrating a parallax angle.

FIG. 4 is a diagram showing an area where a CCD camera is to be installed.

FIG. 5 is a diagram showing an external appearance of a terminal device (terminal device according to the present invention) that realizes a pen input operation sound generation method according to the present invention.

FIG. 6 is a diagram showing a configuration of a terminal device according to the present invention.

FIG. 7 is a diagram showing an overview of a tablet as an input means in the terminal device according to the present invention.

FIG. 8 is a flowchart for explaining the operation of the pen input operation sound generation method according to the present invention.

FIG. 9 is a diagram showing an example of a monitor screen in a terminal device (terminal device according to the present invention) that realizes a pen input operation sound generation method according to the present invention.

10 is a conceptual diagram for explaining an embodiment to which the pen input operation sound generation method according to the present invention is applied in the two-way interactive system shown in FIG.

[Explanation of symbols]

8 ... Monitor screen, 21 (21A, 21B) ... Display section, 3
(3A, 3B) ... Interlocutors A, B, 5A, 5B, 910
A, 910B ... CCD camera (imaging unit), 6 (6A, 6)
B) ... Imaging unit, 600 (600A, 600B) ...
Imaging device, 26 (26A, 26B) ... Image processing device, 2
0 (20A, 20B) ... Terminal device, 262 ... D / B, 2
63 ... Control unit, 900 ... Mouse (pointing device), 910 ... Headphones (speaker), 920,
920A, 920B ... Tablet, 930, 930A,
930B ... Input pen, 940, 940A, 940B ... Input surface.

Claims (11)

[Claims] 1. An effect of a pen input operation via an input interface which has a predetermined input surface and takes in coordinate information on the input surface corresponding to a contact point between the input surface and the input pen as an electric signal. A sound generation method, wherein presence / absence of an electric signal from the input interface is detected at a predetermined sampling period, operation of the input pen is specified based on a detection result of the electric signal, and the specified input pen is detected. A sound effect corresponding to the operation of the input pen is selected from a sound effect group prepared in advance for each operation type of the input pen, and the selected sound effect is specified when the operation of the input pen is specified or while the specified operation is continued, Method of generating sound effect of pen input operation for playback output. 2. The operation of the input pen is at least
An operation of bringing the input pen into contact with the input surface, and
The method for generating a sound effect of a pen input operation according to claim 1, further comprising an operation of moving a contact point of the input pen in a state of being in contact with the input surface. 3. The method for generating a sound effect of a pen input operation according to claim 1, wherein different sound effects are reproduced and output according to a moving speed level of the contact point on the input surface. 4. A display unit capable of displaying one or more windows and a predetermined input surface, and electric information indicating coordinate information on the input surface corresponding to a contact point between the input surface and the input pen. An input interface for capturing as input, an output unit that outputs a sound effect corresponding to the operation of the input pen, and a data base that stores a plurality of types of sound effect data prepared in advance for each operation of the input pen A control unit that causes the output unit to reproduce and output a sound effect corresponding to the operation type of the input pen, and the control unit, based on an electrical signal from the interface detected at a predetermined sampling cycle, The operation of the input pen is specified, the sound effect data corresponding to the specified operation is read from the data base, and the read sound effect data is forwarded. Terminal apparatus for reproduction output by the output unit. 5. The control unit includes, as the operation of the input pen, at least an operation of bringing the input pen into contact with the input surface, and a contact point with the input pen in contact with the input surface. The terminal device according to claim 4, wherein an operation of moving the terminal is specified. 6. The data base stores a plurality of types of sound effect data prepared for each moving speed level of the contact point on the input surface. 5. The terminal device according to item 5. 7. The data base stores a plurality of types of sound effect data prepared for each window displayed on the display unit.
7. The terminal device according to claim 6. 8. A two-way interactive system that realizes remote interaction via a predetermined transmission means, and includes the terminal device according to any one of claims 4 to 7. 9. The data base of the terminal device includes a plurality of types of sound effect data prepared for at least one of the operator of the input pen and the terminal device included in the two-way interactive system. 9. The interactive system of claim 8, wherein the interactive system is stored. 10. A computer program for performing the method according to claim 1 by a computer. 11. A recording medium on which a computer program for executing the method according to claim 1 is recorded by a computer.