JPH09190544A - Acoustic presentation method for image data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire the position and contents of invisible component elements existing out of a display area without scrolling display screen by outputting a specific sound that is previously made to correspond to the component element of a plotting area to an acoustic area correspond to the plotting area localizing a sound image in the acoustic area. SOLUTION: A portable information terminal 401 generates the stereophonic sounds and outputs them through the speakers 402 and 403. The component elements of a window system such as windows 406, 407, an icon 408, etc., are arranged in a plotting area 405. When the window system receives an instruction from a user to search for the position of the window 406, for example, the system outputs the sound that is previously assigned to the window 406 after localizing a sound image at the centroid position, for example. of the window 406. Thus the user can judge the position of the window 406 even when the window 406 is not displayed.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a computer,
The present invention relates to a data presentation method for information equipment and the like, and a method for acoustically expressing display image information and an apparatus using the same.

[0002]

2. Description of the Related Art Display screens, which are the main means of presenting information in information equipment such as computers and communication terminals, have recently become popular.
The GUI is becoming the mainstream to improve the usability for users. In addition, the proportion of image data included in display information tends to increase more and more with the increase in definition of displays.

On the other hand, with the development of communication infrastructure, there is an increasing need for an information terminal with improved portability for the purpose of exchanging information and moving information when moving or going out. However,
In a portable information terminal or the like, it is necessary to reduce the overall size in order to enhance portability, and accordingly, the display screen is also reduced, which limits the information that can be displayed.

As a conventional method for eliminating the smallness of the display screen, a virtual screen larger than the display size is set, the display is made to appear as one display area of the screen, and the screen is scrolled. There are a method of searching for a target object and a method of having a plurality of screens and switching between them.

There is also a method of presenting by converting the positions and shapes of constituent elements in a drawing area such as a display screen into auditory information. Some of these methods include
There is a method of preliminarily assigning a language expressing characteristics such as shape and color to the constituent elements and reading them out by using a voice synthesis technique, or a method of associating the constituent elements with a specific sound and outputting them.

[0006] As a conventionally proposed speech synthesis method, for example, J. Allen, MS Hunnicutt and D. Klat are used.
t, "From text to speech: The MITalk system" (Camb
ridgeUniversity Press, 1987). This is a method of parsing text and converting it into an audio signal using rules.

As a method of outputting sounds associated with the constituent elements, for example, Gaver, WW, "Using and Cr
eating Auditory Icons "(Auditory Display, SFI Stu
dies in the Sciences of Complexity, Proc. Vol. XV
III, Addison-Wesley, 417-446, 1994).

[0008]

In the method of virtually setting a screen larger than the display, the constituent elements of the non-displayed portion outside the display range cannot be seen, so that the necessary information cannot be displayed by scrolling the screen. There was a problem that it took time to search. To solve this, there is a way to display a help area that displays the arrangement of components included in the entire screen area.
There is a problem that it is difficult to set such an area when the display screen is small. Further, even in the system having a plurality of screens, there is a problem in that it is not possible to see the windows on the screens whose display is not selected.

There is a problem that the arrangement position cannot be determined by the method of reading out the constituent elements by voice or the method of associating the sounds.

[0010]

In order to solve the above problems, means for mapping a sound corresponding to the position of a constituent element in a drawing area into an acoustic area and an attribute of the constituent element in the drawing area are represented. A means for locating sound at a specific position in the acoustic area is provided, display image data is arranged by setting a drawing area that includes the display screen area, and image data outside the display area is allocated to the display image data. Sounds that have been assigned in advance
A method of presenting the sound image by locating and outputting the sound image to a position in the sound presentation region corresponding to the position in the drawing region of the image data is used.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 shows an embodiment of a hardware configuration for executing the method of the present invention. FIG. 1 has the configuration of a computer system having a stereophonic device for voice and sound output. That is, C that performs the operation
A PU 101, a RAM 102 for temporarily rewriting programs and data for arithmetic operations and sequentially rewriting, a ROM 103 for storing a system boot program, an I / O controller 104 for controlling system input / output,
Storage device 10 such as a magnetic disk for holding system programs, application programs, data, etc.
5, which perform instruction and data transfers via the system bus 106. In addition, the I / O controller 10
Reference numeral 4 denotes a display device 107 such as a CRT display and a pointing device 10 such as a mouse and a touch panel.
8, an input device 109 such as a keyboard, a data transmission / reception communication port 110, and a stereophonic device 111 on the bus 11
Control via 2.

The standard control of such hardware is
This is generally done with software called an operating system. The algorithm of the sound representation method of image data according to the present invention can be realized as software executed on the hardware and operating system.

Software for executing the algorithm of the present method is stored in the ROM 103 or the storage device 105, and after starting the system, the RAM 102 may be used as necessary.
And is executed by the CPU 101. Further, when data stored in advance in the storage device 105, data input through the input device 109 or the communication port 110 is arranged in the display area of the display, a display device 107 such as a liquid crystal display is displayed. Display using. Further, the sound and sound using the stereophonic sound generated by the acoustic representation method of the image data is output through the stereophonic device 111. The stereophonic device can be realized by using a general-purpose board using a high-speed signal processing chip such as DSP and a plurality of speakers.

Since the present hardware configuration is used, for example, in a general personal computer, workstation, portable information terminal or the like having an audio output device, the method of the present invention can be realized by using these devices. Is possible.

The method of the present invention will be described below.

FIG. 2 shows an example in which the constituent elements in the drawing area are represented by sound image localization in the acoustic area. The drawing shows the drawing area 201 and the user 207 in a bird's-eye view from diagonally behind. In the drawing area 201, the component 2
02 and 203 are drawn. The coordinates of the constituent elements on the drawing area are associated with the coordinates of the acoustic area 204, and the sounds expressing each constituent element are localized at the coordinates 205 and 206 on the acoustic area. As a method for locating sound in the acoustic region, for example, the spatial sound image localization technology described in “Sound Image Control Technology” (Journal of the Television Society, Vol.46, No.9, 1076-1079, 1992) is used. It is possible. At this time,
The drawing area 201 may be larger than the display screen size of a display device such as a computer display.

Here, if the distances in the acoustic area and the drawing area distances are arranged to be one-to-one, the positions of the constituent elements in the drawing area and the positions of the sound sources in the acoustic area are completely matched.
Further, when the acoustic region is made larger than the drawing region, the effect of relatively improving the resolution of sound image localization can be obtained.

The correspondence between the drawing area and the acoustic area is
For example, as shown in FIG. 3, the midpoint of both ears of the user is the origin 3
01, and the component 303 on the origin and the drawing area 302
The acoustic region 3 is placed at the geometrical position on the extension connecting the coordinates of
This can be done by assigning the coordinates of the sound image 305 corresponding to the 04 constituent elements. Note that the mapping to the acoustic region may be modified so as to suit the auditory sense according to the characteristics of hearing. In FIG. 3, the two-dimensional area is described as an example,
It goes without saying that this association can be easily extended to the three-dimensional area.

If this method is used, the direction in which the sound is heard differs depending on the position of each component. Therefore, when a plurality of components are selected at the same time, the sounds may be output simultaneously.

FIG. 4 is an example of sound presentation of the window system GUI. Reference numeral 401 represents a personal digital assistant having a stereophonic device. A speaker 401 generates stereophonic sound using a known stereophonic sound generation algorithm.
A stereophonic image is output using 02 and 403. 404 is
It is a display screen such as a liquid crystal display. Further, 405 represents a drawing area in 401, and constituent elements such as windows and icons of the window system are arranged in 405. Here, 406 and 407 are windows, 4
08 to 410 represent icons. However, the display screen 40
Since 4 is smaller than the drawing area 405, the components arranged outside 404 are not displayed. Now, using the method of the present invention, the acoustic region is arranged so as to overlap the drawing region, and, for example, the window system allows the user to select the window 406.
When a command for searching the position of is received, the sound previously assigned to the window is localized and output as the sound image at the center of gravity of the window, for example. This makes it possible to determine the position of the window even when the window is not displayed. At this time, at the same time as outputting the sound, the screen may be scrolled to display the position where the window is arranged.

FIG. 5 shows an example of correspondence between each component of the window system and voice or sound. For example, for a window, the melody associated with the window is always output when the cursor is passing over the window, and when the window is selected, the window name is read at the same time when the melody is output. Regarding the menu, a click sound is output when the cursor is moving on the menu, and when the cursor stops, the command name at the stopped position is read aloud. For watches,
Outputs the wall clock sound when the cursor stops on the clock,
When selected, read out the time. For files, the percussion sound is heard when the cursor passes, and the file name is read when the cursor stops. For folders, the sound is changed when the cursor passes and when selected. Such correspondence makes it possible to determine the contents, positions, and states of the components of the window system.

FIG. 6 shows an example of displaying map information on a portable information terminal. The figure assumes an application that searches the position of a building using a mobile information terminal. In FIG. 6, 601 represents a drawing area and an acoustic area. 6
02 indicates the current position of the user. Also, 603 to 605
Indicates the position of a landmark such as a movie theater, but is not displayed because it is outside the display area 606. Now, when the user gives an instruction to search for a movie theater near the current position, the method of the present invention is used to 603, 604, 60.
At 5 positions, the sound representing the movie theater or the voice reading the name of the movie theater is sound image localized and output in sequence or simultaneously. As a result, the user can know the position of the movie theater without scrolling the entire map even when the movie theater is not displayed.

FIG. 7 shows another example of displaying map information on a portable information terminal. FIG. 7 shows a case where the user searches for a building existing in a specific direction when viewed from the current position 702 on the map in the drawing area and the acoustic area 701. Here, eight icons are arranged around the display screen 703, and when an icon in the direction to be searched is selected from the current position of the user who is arranged at the center of the display screen, a land such as a building in that direction is selected. By reading aloud the name of the mark or outputting a specific sound, it is possible to know the position and type of the landmark that is not currently displayed in the search direction. Now icon 704
Is selected by the pen 705. At this time, the system localizes the voice of reading the name of the landmark such as the building 707 included in the area 706 or the sound pre-assigned to the landmark to the landmark position on the map and outputs them in order.

FIG. 8 shows an example of displaying three-dimensional map information on a mobile information terminal. The figure assumes an application that uses a mobile information terminal to search the position of a room in a multi-storey building on a map. In FIG. 8, (a) represents a plan view including a portable information terminal and a building, in which 801 represents a drawing area and an acoustic area. Here, it is assumed that one room 803 of a building 802 in a map arranged in the drawing area 801 is searched.
(B) represents the three-dimensional arrangement of the building 802, and in this example, the room 803 to be searched is on the second floor of the building 802. Now, when the user inputs a command to select and search for the name of the room 803, the application searches for the location and floor of the room 803 in the building 802 in the map, and if this is not currently displayed, The position is expressed by aloud the name or the sound image localization of a specific sound.
At this time, the planar position is represented by upper and lower, left and right sound image localization, and the floor is represented by front and rear sound image localization with a sense of depth, so that the user can know the position of the searched room.

[0026]

EFFECTS OF THE INVENTION With respect to a constituent element in a drawing area, a voice that reads the name of the constituent element or a specific sound that is associated in advance is sound image localized and output to the acoustic area associated with the drawing area. This makes it possible to acquire the positions and contents of invisible components outside the display area without scrolling the display screen. As a result, it becomes possible to effectively present the image information, especially on a portable information terminal or the like in which it is difficult to enlarge the display screen.

[Brief description of the drawings]

FIG. 1 is a hardware configuration diagram for implementing a method of the present invention.

FIG. 2 is a diagram illustrating an example in which a sound element is localized in a sound area for components in a drawing area.

FIG. 3 is a diagram illustrating an example of association between a drawing area and an acoustic area.

FIG. 4 is a diagram illustrating an example of audio presentation of a window system.

FIG. 5 is a diagram illustrating a voice / sound correspondence example of a window system component.

FIG. 6 is a diagram illustrating an example of acoustic presentation of map information.

FIG. 7 is a diagram illustrating an example of audio presentation of map information.

FIG. 8 is a diagram illustrating an example of audio presentation of three-dimensional map information.

[Explanation of symbols]

106 ... System bus, 112 ... Bus, 201, 302
... drawing area, 202, 203, 303 ... constituent element, 20
4, 304 ... Acoustic region, 205, 206, 305 ... Sound image, 207 ... User, 301 ... Origin, 401 ... Portable information terminal, 402, 403 ... Speaker, 404 ... Display screen, 406, 407 ... Window, 408-410 , 7
04 ... Icon, 405, 601, 701, 801 ... Drawing area and sound area, 602, 702 ... Current position of user, 603 to 605, 707 ... Landmark, 606
703 ... Display area, 705 ... Pen, 706 ... Search area,
802 ... building, 803 ... room.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location G09G 5/14 9377-5H G09G 5/14 Z

Claims (3)

[Claims] 1. A portable information terminal device having means for displaying an image on a display screen, means for generating stereophonic sound, and means for outputting stereophonic sound, including an image area displayed on the display screen. A virtual drawing area and display image data that can be displayed in the drawing area are arranged, acoustic information is assigned to the display image data, and when an access request to the display image data is made, A sound presentation method for image data, wherein a sound pre-assigned to the display image data is sound-localized and output at a predetermined position in the drawing area where the display image data is located. 2. When an access request for the display image data is made and the display image data in the drawing area is outside the image area displayed by the display screen, the display image data is displayed. 2. The sound presentation method for image data according to claim 1, wherein the sound pre-assigned to the data is localized and output at a predetermined position around the display screen. 3. The sound presentation method for image data according to claim 1, wherein the display image data is data expressed in three dimensions.