JP7182997B2 - picture book display system - Google Patents

Description

The present invention relates to a display system, a picture book display system, and a program.

As a conventional technology, for example, there is a picture book display device that displays a page of a picture book on a television when a reader reads text information displayed on the television.

Patent Document 1 discloses text information storage means for storing page numbers, page images, and page numbers of subsequent pages in association with each other; a display image control unit for displaying the text information and the page images on a predetermined display means; An output audio control unit that outputs audio information obtained by the microphone unit to a predetermined output means, and an output audio control unit that analyzes the obtained audio information, determines the characteristics of the audio information, and associates it with the page number to be displayed. In the succeeding page number, a branch processing unit that specifies the page number associated with the feature of the discrimination result of the feature discriminating unit as the succeeding page number, and specifies the specified succeeding page number as the next page number to be displayed. An electronic picture book system with a seismic intensity controller is described.
Further, Patent Document 2 discloses a recording medium in which images and sentences for a picture book are written in advance, a background memory for storing background image data read from the recording medium, and a moving image memory for storing moving image data. , and a synthesizing unit for synthesizing the background data and the moving image data read out from the background memory and the moving image memory into a picture book, wherein the image data synthesized by the synthesizing unit is displayed as a picture book. A device is described.

JP 2009-122498 A JP-A-5-120400

However, conventionally, the story of a picture book is prepared in advance, and the reader cannot freely create it. Also, when a reader reads a story prepared in advance, it is necessary for the reader to use a controller or the like to perform a proceeding operation to switch the display of pictures.
SUMMARY OF THE INVENTION It is an object of the present invention to provide a display system or the like in which a display image such as a picture, text, or the like corresponding to the contents of a reader's utterance is displayed in real time on a listener's portable terminal or the like based on the contents of the reader's speech. and

Thus, according to the present invention, a voice acquisition means for acquiring the voice spoken by the reader, a comprehension means for comprehending the meaning of the voice spoken by the reader, an image acquisition means for acquiring an image corresponding to the meaning comprehended by the comprehension means, arranging means for arranging the acquired images according to their meanings to form a picture book, wherein the image acquiring means determines whether or not a pre-registered word specifying an image is included in the uttered voice of the reader. Searching, when a word is included, an image corresponding to the word is acquired, and the arranging means includes feature information representing the feature of the image registered in advance and acquired by the image acquiring means in the uttered voice of the reader. performs processing to match the features of the image, and the wording and feature information are registered in correspondence with the page numbers indicating the pages of the picture book, and the placement means distinguishes between the background image and the foreground image. A picture book display system characterized in that, when an image acquisition means acquires a new background image, it treats it as a number of pages corresponding to the new background image.
Here, the comprehension means can further comprehend the meaning of the listener's uttered voice, and the image acquisition means can acquire an image corresponding to the meaning of the listener's uttered voice comprehended by the comprehension means. In this case, a picture book can be created based not only on the voice spoken by the reader but also on the voice spoken by the listener.
Further, it is possible to further include separating means for separating the uttered voice of the reader and the uttered voice of the listener. In this case, a picture book can be created based on the uttered voice of the reader .

According to the present invention, it is possible to provide a display system or the like in which a display image such as a picture, text, or the like corresponding to the utterance content of the reader is displayed in real time on the portable terminal of the listener based on the content of the utterance of the reader. .

It is a figure which shows the structural example of the display system in this Embodiment. FIG. 4 is a diagram showing an example of schematic operation of the display system; 2 is a block diagram showing an example functional configuration of a display system according to the first embodiment; FIG. 4 is a flow chart describing an example of the operation of the display system in the first embodiment; It is the figure which showed about the method of isolate|separating a reader's utterance voice and a listener's utterance voice in a separator. 4A and 4B are diagrams showing the data structure of a storage unit used in the first embodiment; FIG. (a) to (c) are diagrams showing a process of arranging an image and making it into a picture book by an arrangement unit. FIG. 11 is a block diagram showing an example of the functional configuration of a display system according to a second embodiment; FIG. 9 is a flow chart describing an example of the operation of the display system of the second embodiment; FIG. 11 is a block diagram showing an example of the functional configuration of a display system according to a third embodiment; FIG. 10 is a flow chart describing an example of the operation of the display system of the third embodiment; FIG. 14 is a flow chart describing an example of the operation of the display system of the fourth embodiment; FIG. It is the figure which showed the data structure of the memory|storage part used by 4th Embodiment. (a) and (b) are diagrams showing an example of a picture book created after the second line of text. FIG. 14 is a block diagram showing an example of the functional configuration of a display system according to a fifth embodiment; FIG. FIG. 15 is a flow chart describing an example of the operation of the display system of the fifth embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Description of Overall Display System 1>
FIG. 1 is a diagram showing a configuration example of a display system 1 according to this embodiment.
As illustrated, the display system 1 of the present embodiment is configured by connecting mobile terminals 20 a and 20 b and a management server 40 via a network 70 , a network 80 and an access point 90 . Hereinafter, when the mobile terminal 20a and the mobile terminal 20b are not distinguished from each other, they may simply be referred to as the "mobile terminal 20".

The mobile terminal 20 is, for example, a mobile terminal such as a mobile computer, mobile phone, smart phone, or tablet. The mobile terminals 20a and 20b connect to the access point 90 for wireless communication. The mobile terminals 20 a and 20 b connect to the network 70 via the access point 90 . Although details will be described later, the mobile terminal 20a is a mobile terminal owned by a reader when reading a story, and the mobile terminal 20b is a mobile terminal owned by a listener.

The management server 40 is a server computer that manages the display system 1 as a whole. Although the details will be described later, for example, the management server 40 acquires an uttered voice when reading a text in a picture book or the like from the portable terminal 20a owned by the reader. Then, an image or the like is acquired based on the content of the uttered voice, and a display image such as a picture book is created. Then, the information of the display image of the picture book or the like is transmitted to the portable terminal 20b of the listener.

The mobile terminal 20 and the management server 40 are provided with a CPU (Central Processing Unit), which is computing means, and a main memory, which is storage means. Here, the CPU executes various types of software such as an OS (basic software) and applications (application software). The main memory is a storage area for storing various software and data used for executing the software. Further, the mobile terminal 20 includes a communication interface (hereinafter referred to as “communication I/F”) for communicating with the outside, a display mechanism including a video memory, a display, etc., an input button, a touch panel, a keyboard, etc. and an input mechanism. The mobile terminal 20 includes a speaker for outputting voice and a microphone for inputting voice. The management server 40 also includes an HDD (Hard Disk Drive) and an SSD (Solid State Drive) as auxiliary storage devices.

The network 70 is communication means used for information communication between the mobile terminal 20 and the management server 40, and is, for example, the Internet.
The network 80 is also a communication means used for information communication between the mobile terminal 20 and the management server 40, like the network 70, and is, for example, a LAN (Local Area Network).

The access point 90 is a device that performs wireless communication using a wireless communication line. The access point 90 mediates transmission and reception of information between the mobile terminal 20 or the management server 40 and the network 70 or 80 .
Available types of wireless communication lines include mobile phone lines, PHS (Personal Handy-phone System) lines, Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), ZigBee, and UWB (Ultra Wideband) lines. is.

<Overview of Operation of Display System 1>
FIG. 2 is a diagram showing an example of the schematic operation of the display system 1. As shown in FIG.
First, a reader who owns the mobile terminal 20a activates a dedicated application used in this system, speaks text from a picture book or the like into the microphone of the mobile terminal 20a, and inputs the uttered voice (1A). A microphone converts speech into an electrical signal corresponding to the sound pressure of the speech. Then, the portable terminal 20a once amplifies the audio signal. Then, sampling is performed at a predetermined sampling frequency, digitization is performed, and information of the uttered voice is created. Note that the listener is on the reader's side and can hear the listener's uttered voice. Also, for example, readers are parents and grandparents, and listeners are children.

Next, the portable terminal 20a transmits the information of the uttered voice to the management server 40 as transmission information (1B). The transmission information is sent to management server 40 via access point 90 , network 70 and network 80 .
The management server 40 grasps the meaning of the uttered voice from the information of the received uttered voice, and acquires display elements representing the grasped meaning from the database (1C). The meaning of the uttered voice can be grasped by voice recognition. Here, the “display element” is an individual element of the display image displayed on the mobile terminal 20b, corresponding to the grasped meaning. Specifically, the "display element" is an image, text, or the like. For example, if the grasped meaning is "bear", the image of the bear is the "display element". It does not matter whether the image is a still image or a moving image. As text, words such as "bear" and "bear" are "display elements". Note that this text may be a word, or may be a sentence composed of a plurality of words.
Furthermore, the management server 40 arranges the images acquired from the database and creates display images such as picture books for listeners to view (1D).
Then, the management server 40 transmits information of the created display image as transmission information to the mobile terminal 20b owned by the listener (1E). Information about the display image is sent to the mobile terminal 20b via the network 80, the network 70, and the access point 90. FIG.

In the portable terminal 20b, the display image is displayed on the display mechanism based on the received display image information (1F). Then, the listener can listen to the uttered voice of the reader and see the displayed display image, thereby reading aloud.

Next, the detailed functional configuration and operation of the display system 1 of this embodiment will be described.

<Description of the functional configuration of the display system 1>
[First Embodiment]
Here, first, the first embodiment of the functional configuration of the display system 1 will be described. In the display system 1 of the first embodiment, the management server 40 acquires an image corresponding to the meaning of the uttered voice of the reader, arranges the acquired images, and creates a picture book as a display image. Then, the listener browses the picture book, which is the display image, on the portable terminal 20b. That is, in this case, the display system 1 functions as a picture book display system.
FIG. 3 is a block diagram showing a functional configuration example of the display system 1 according to the first embodiment.
Here, among the various functions of the display system 1, those related to the present embodiment are selected and illustrated.
In the display system 1, the mobile terminal 20a and the mobile terminal 20b have the same functional configuration, and include a transmission/reception unit 21 for transmitting and receiving transmission information, a display unit 22 for displaying images, and an input unit for inputting information. 23, and a voice acquisition unit 24 that acquires an uttered voice.

The transmitting/receiving unit 21 transmits/receives transmission information such as speech information and picture book information. The transmission/reception unit 21 is, for example, a communication I/F, and transmits/receives information to/from the management server 40 via the access point 90, the network 70, and the network 80. FIG.

The display unit 22 displays images such as picture books. The display unit 22 is, for example, a touch panel. In this case, the display unit 22 includes a display for displaying various information and a position detection sheet for detecting the position touched by a finger, stylus pen, or the like. As a means to detect the contact position, what kind of method is used, such as a resistive film method that detects based on the pressure caused by contact, or a capacitance method that detects based on the static electricity of the contacted object. good too.

The input unit 23 is an operation mechanism for a picture book reader or listener to perform a predetermined operation.
For example, it is the touch panel mentioned above. In this case, the touch panel has the functions of both the display section 22 and the input section 23 . In other words, by displaying an image of a picture book or the like and touching the displayed screen, it is possible to start/end the dedicated application or to operate the dedicated application. Note that the input unit 23 is not limited to this, and may be configured by a keyboard, a mouse, or the like.

The voice acquisition unit 24 acquires uttered voice. The voice acquisition unit 24 is, for example, a microphone. As for the type of microphone, existing various types such as a dynamic type and a condenser type may be used. Further, the microphone is preferably an omnidirectional MEMS (Micro Electro Mechanical Systems) type microphone.

The management server 40 includes a transmission/reception unit 41 that communicates with the outside, a separation unit 42 that separates the uttered voice of the reader and the uttered voice of the listener, a comprehension unit 43 that comprehends the meaning of the uttered voice, and a It has an image acquisition unit 44 for acquiring the image obtained, a storage unit 45 for storing the image, and an arrangement unit 46 for arranging the image to form a picture book.

The transmission/reception unit 41 communicates with the mobile terminal 20 to exchange predetermined information. The transmitting/receiving unit 41 is an example of voice acquisition means, and acquires the uttered voice of the reader sent from the mobile terminal 20a as transmission information. Further, the transmitting/receiving section 41 is an example of output means, and transmits information of a display image such as a picture book to the mobile terminal 20b of the listener.
The separating unit 42 is an example of separating means, and separates the uttered voice of the reader and the uttered voice of the listener. A method for performing these separations will be described later.

The comprehension unit 43 is an example of comprehension means, and comprehends the meaning of the uttered voice of the reader. Although the details will be described later, the grasping unit 43 determines whether or not a pre-registered wording is included in the uttered voice. be one of
The image acquiring unit 44 is an example of an element acquiring unit and an example of an image acquiring unit, and acquires display elements representing the meaning grasped by the grasping unit 43 . Here, an image corresponding to the meaning grasped by the grasping unit 43 is acquired as a display element.

The storage unit 45 stores the image acquired by the image acquiring unit 44 in association with the meaning. In addition, the storage unit 45 stores the content of the processing to be performed on the image in association with the meaning. The details of this processing will be described later.
The arranging unit 46 is an example of arranging means, and arranges the acquired display elements to make a display image that is viewed by the listener. Here, the arranging unit 46 arranges the acquired images according to the comprehended meaning to form a picture book. That is, the arranging unit 46 performs predetermined processing on the image acquired by the image acquiring unit 44 according to the grasped meaning, and arranges the processed image to form a picture book. Although the details will be described later, picture books are drawn by superimposing the foreground on the background.

The transmitter/receiver 41 is, for example, a communication I/F. Also, each function of the separating unit 42, the grasping unit 43, the image acquiring unit 44, and the arranging unit 46 can be realized by, for example, a CPU. Furthermore, the storage unit 45 is a database constructed by using an auxiliary storage device such as an HDD or SSD, for example.

<Description of Operation of Display System 1>
Next, the operation of the display system 1 of the first embodiment will be described in more detail.
FIG. 4 is a flow chart explaining an example of the operation of the display system 1 according to the first embodiment.
First, the reader of the picture book operates the dedicated application using the input unit 23 of the mobile terminal 20a and inputs the uttered voice. This uttered voice is acquired by the voice acquiring unit 24 (step 101).
Next, the transmitting/receiving section 21 of the reader's portable terminal 20a transmits the information of the uttered voice as transmission information to the management server 40 (step 102).

In the management server 40, the transmitting/receiving section 41 receives the information of the uttered voice (step 103). This is once stored in the storage unit 45 .
Next, the separation unit 42 of the management server 40 separates the uttered voice of the reader and the uttered voice of the listener (step 104). In other words, when reading a picture book, it often shows various reactions to the utterances of the reader. For example, the reader may ask a question about the content of the picture book, or utter a voice such as "Amazing!" or "Scary!" as an impression of the content of the picture book. In the present embodiment, the separating unit 42 separates the uttered voice of the reader and the uttered voice of the listener in order to extract only the uttered voice of the reader.

FIG. 5 is a diagram showing how the separator 42 separates the uttered voice of the reader and the uttered voice of the listener.
Here, the vertical axis indicates the sound pressure of the uttered voice of the reader when the sound pressure of the uttered voice of the listener is 1. In this example, the ratio of the sound pressure of the listener's uttered voice to the sound pressure of the reader's uttered voice is four. In other words, the reader speaks toward the mobile terminal 20a, and the distance between the mobile terminal 20a and the mouth, which is the utterance part, is very short. On the other hand, although the listener is near the reader, the listener is relatively far away from the portable terminal 20a. Therefore, due to this distance difference, the sound pressure will differ by a factor of four in this example. Therefore, a threshold is set for the sound pressure, and when the sound pressure greater than this threshold is detected, the separation unit 42 determines that it is the uttered voice of the reader, and when the sound pressure below this threshold is detected, the sound pressure is It can be judged as speech voice. In the illustrated example, 2 is used as the threshold for the sound pressure.

Returning to FIG. 4, next, the comprehension unit 43 comprehends the meaning of the uttered voice of the reader separated by the separation unit 42 (step 105). Specifically, the grasping unit 43 uses known speech recognition technology to convert the speech into a character string. Known speech recognition techniques can use, for example, statistical methods using Hidden Markov Models and dynamic time warping.
Then, the grasping unit 43 checks whether or not the converted character string includes a pre-registered word specifying an image (step 106). This phrase may be a word or a sentence composed of a plurality of words.
As a result, if it is not included (No at step 106 ), the process returns to step 101 .
On the other hand, if it is included (Yes in step 106), the image acquisition unit 44 acquires an image corresponding to this wording from the storage unit 45 (step 107).
Next, the arranging unit 46 checks whether or not feature information representing the feature of the image registered in advance and acquired by the image acquiring unit 44 is included in the uttered voice of the reader (step 108).
As a result, if it is not included (No in step 108), the process proceeds to step 110. FIG.
On the other hand, if it is included (Yes in step 108), the arrangement unit 46 performs processing for matching this feature to this image (step 109).

6A and 6B are diagrams showing the data structure of the storage unit 45 used in the first embodiment.
As shown in the figure, there are two types of data structures for data stored in the storage unit 45: the data structure for image information shown in FIG. 6(a) and the data structure for feature information shown in FIG. 6(b). . Here, the "feature information" is information that characterizes the things appearing in the picture book.
The data structure for the image information shown in FIG. 6(a) is used in steps 106-107 of FIG. This data structure is No. It consists of four elements: text, attributes, and images. Among these, "No." is a number assigned to each wording. "Text" is the pre-registered text mentioned in step 106 . Further, "attribute" is the attribute of the image, indicating either background or foreground. "Image" indicates the file name of the file in which image data is stored.

Also, the data structure for the feature information shown in FIG. 6(b) is used in steps 108 and 109 of FIG. This data structure is No. It consists of three parts: wording and processing. Among these, "No." is a number assigned to each wording. "Text" is the pre-registered text mentioned in step 106 . Furthermore, "processing" indicates the content of the processing to be performed on the image.

At step 106 in FIG. 4, the grasping unit 43 checks whether or not the wording shown in FIG. 6(a) exists in the converted character string. Then, if it exists, the image acquisition unit 44 acquires an image corresponding to this wording in step 107 . In the example shown in FIG. 6A, for example, "forest", "sky", etc. are registered as words. Then, image data with a file name of "□×◇.jpg" is prepared as an image representing the forest, and image data with a file name of "○×△.jpg" is prepared as an image representing the sky. be done.

Also, at step 108 in FIG. 4, the arrangement unit 46 checks whether the wordings listed in FIG. 6B exist in the converted character string. Then, if there is, the placement unit 46 performs a process of matching this feature to this image in step 109 . In the example shown in FIG. 6B, for example, "large", "small", etc. are registered as words. Then, when the word "large" exists, the corresponding image is enlarged. On the other hand, when the word "small" exists, the corresponding image is reduced. Further, for example, in the case of a combination of "number + animal" such as "3 animals", the number of images of animals or the like is increased or decreased to three animals. Furthermore, when there is, for example, "run" as a word, the corresponding image is moved quickly to represent running. Furthermore, when there is, for example, "blue" as a word, the corresponding image is colored blue.

Returning to FIG. 4 again, the placement unit 46 then places the images to form a picture book (step 110).
FIGS. 7A to 7C are diagrams showing the process of arranging images to form a picture book by the arranging unit 46. FIG. Here, the case where the reader utters the following sentence will be explained as an example.

――――――――――――――――――――――――――――――――――――――――
Reader: "There was a bear in a deep, deep forest. There were three bears."
――――――――――――――――――――――――――――――――――――――――

Among them, FIG. 7A shows a state in which the arrangement unit 46 arranges the image Gm corresponding to the forest when the word "forest" is present in the uttered voice of the reader. The word "forest" is No. shown in FIG. 6(a). Since the word "forest" in 1001 is matched, the image acquisition unit 44 acquires the forest image Gm, and the arrangement unit 46 arranges the forest image Gm. This corresponds to the case where the reader utters "in a deep, deep forest". Also, the layout unit 46 considers the attributes shown in FIG. 6A when arranging the images. That is, the attributes include background and foreground, and the placement unit 46 places the background image and the foreground image separately. In this case, since the attribute of the words “forest” is the background, the arrangement unit 46 arranges the forest image Gm as the background.

Next, FIG. 7B shows a state in which the arrangement unit 46 arranges an image Gk corresponding to a bear when the word "bear" is present in the uttered voice of the reader. The wording of "bear" is No. shown in FIG. 6(a). Since the word "bear" in 1006 is matched, the image acquisition unit 44 acquires the bear image Gk, and the arrangement unit 46 arranges the bear image Gk. This corresponds to the case where the reader utters "There was a bear." after "In the deep, deep forest." At this time, similarly to the case described above, the arranging unit 46 considers the attributes shown in FIG. 6A when arranging the images. In this case, since the attribute of the words “bear” is foreground, the placement unit 46 places the bear image Gk as the foreground. Specifically, the background forest image Gm is hidden, and the bear image Gk is superimposed.

Furthermore, FIG. 7(c) shows a state in which the arrangement unit 46 arranges an image corresponding to the word "three" in the uttered voice of the reader. In this case, the wording of "three" is No. shown in FIG. 6(b). 5004, the arrangement unit 46 increases the number of bear images Gk to three, and arranges the three bear images Gk. This corresponds to, for example, the case where the reader utters "There were three bears." after "There was a bear in a deep, deep forest."

Returning to FIG. 4, the transmitting/receiving section 41 transmits the picture book information created by the arranging section 46 to the portable terminal 20b of the listener (step 111). In this embodiment, the picture book images shown in FIGS. 7A to 7C are sequentially sent.
The picture book information is received by the transmitter/receiver 21 of the portable terminal 20b (step 112), and the picture book is displayed on the display unit 22 (step 113). In this case, the listener listens to the reader's voice, "There was a bear in a deep forest. There were three bears." The images in (c) are sequentially viewed, and the picture book can be read aloud.

When the image acquiring unit 44 acquires a new background image, it can be treated as a new page in the picture book. In other words, when the background is new, it can be regarded as a different scene from the previous page and a new page. Therefore, the placement unit 46 prepares a new page and places the foreground image on the new background. That is, this enables page turning.

[Second embodiment]
Next, a second embodiment will be described. In the second embodiment, a picture book is created by adding not only the uttered voice of the reader but also the uttered voice of the listener.
FIG. 8 is a block diagram showing a functional configuration example of the display system 1 according to the second embodiment.
In this display system 1, the mobile terminal 20a and the mobile terminal 20b have the same functional configuration as the display system 1 in the first embodiment shown in FIG. It includes a display unit 22 for displaying images, an input unit 23 for inputting information, and a voice acquisition unit 24 for acquiring uttered voices.
On the other hand, the management server 40 has the same functional configuration as that of the first embodiment except that the separation unit 42 is not present. That is, it has a transmitting/receiving section 41 , a grasping section 43 , an image obtaining section 44 , a storage section 45 and an arrangement section 46 .
Each functional unit of the mobile terminal 20a, the mobile terminal 20b, and the management server 40 operates in the same manner as in the first embodiment.

FIG. 9 is a flow chart explaining an example of the operation of the display system 1 of the second embodiment.
Steps 201 to 212 of the flowchart shown in FIG. 9 are the same as steps 101 to 103 and steps 105 to 113 of the flowchart shown in FIG. That is, it is the same except that step 104 is omitted. Therefore, in the second embodiment, the uttered voice of the reader and the uttered voice of the listener are not separated. Then, since the processing from step 204 onwards is performed for both uttered voices, the meaning of not only the reader's uttered voice but also the listener's uttered voice is grasped and reflected in the creation of the picture book. As shown in FIG. 5, since the sound pressure of the listener's uttered voice is low, the listener's uttered voice may be acquired by the mobile terminal 20b instead of the mobile terminal 20a. In this case, since the mobile terminal 20b is owned by the listener and is located closer than the mobile terminal 20a, the utterance can be obtained with a higher sound pressure.

In the second embodiment, the comprehending unit 43 comprehends not only the uttered voice of the reader but also the uttered voice of the listener, and the image acquiring unit 44 comprehends the meaning of the uttered voice of the listener comprehended by the comprehending unit 43. Get the corresponding image. Furthermore, the placement unit 46 creates a picture book by placing the images acquired by the image acquisition unit 44 .

This specific example will be described again with reference to FIGS. 7(a) to 7(c). In this case, the following conversation occurs between the reader and the listener.

――――――――――――――――――――――――――――――――――――――――
Reader: "There was a bear in a deep, deep forest."
Interviewer: "How many are there?"
Reader: "Three."
――――――――――――――――――――――――――――――――――――――――

In this case, the arranging unit 46 arranges the images shown in FIGS. is similar to the embodiment of
On the other hand, in the second embodiment, the comprehension unit 43 comprehends that the listener is asking about the number of bears based on the listener's "How many bears are there?" Then, assuming that the next reader's answer is "three." As a result, an image as shown in FIG. 7(c) is obtained.
In the case of the second embodiment, a picture book can be created by reflecting not only the uttered voice of the reader but also the uttered voice of the listener.

[Third Embodiment]
Next, a third embodiment will be described. In the third embodiment, when a character appearing in a picture book speaks, the spoken voice is output as an actual voice by the portable terminal 20b of the listener.
FIG. 10 is a block diagram showing a functional configuration example of the display system 1 according to the third embodiment.
In this display system 1, the management server 40 has the same functional configuration as in the first embodiment. That is, it has a transmitting/receiving section 41 , a separation section 42 , a comprehension section 43 , an image acquisition section 44 , a storage section 45 and an arrangement section 46 .
On the other hand, the mobile terminals 20a and 20b differ from the first embodiment in that an audio output unit 25 is added.

The voice output unit 25 outputs the uttered voice of the character sent from the management server 40 and appearing in the display image. When the displayed image is a picture book, the "character" is a picture that appears in the picture book. The character is not particularly limited, and may be a character that exists in reality or a character that does not exist in reality. In addition, even if the character exists in reality, it is not limited to the case where the character has the ability to actually utter a voice or the like, such as a person or an animal. Examples include insects, plants such as trees and flowers, celestial bodies such as the sun and moon, and toys. Characters that do not exist in reality include, for example, fairies, dinosaurs, monsters, gods, and ghosts.

FIG. 11 is a flow chart explaining an example of the operation of the display system 1 of the third embodiment.
Steps 301 to 313 of the flowchart shown in FIG. 11 are the same as steps 101 to 113 of the flowchart shown in FIG. Then, step 314 is newly added. At step 314, as described above, the uttered voices of the characters appearing in the picture book are output.

In this case, the reader utters, for example, the following sentences.

――――――――――――――――――――――――――――――――――――――――
Reader: “While running, Mr. Bear said, wait!”
――――――――――――――――――――――――――――――――――――――――

In this case, the comprehension unit 43 comprehends that the uttered voice of the reader is the utterance of a character appearing in the picture book, based on "Kuma-san said this." In this case the character is a bear. Then, the transmitting/receiving unit 41 further transmits the uttered voice of the bear as picture book information to the portable terminal 20b of the listener. As a result, as described in step 314, the portable terminal 20b outputs the uttered voice of the bear appearing in the picture book. Specifically, the listener hears the following speech.

――――――――――――――――――――――――――――――――――――――――
Reader: “While running, Mr. Bear said, wait!”
Portable terminal 20b: "Wait!"
――――――――――――――――――――――――――――――――――――――――

At this time, the utterance voice output from the mobile terminal 20b may be prepared in advance or may be synthesized voice. Moreover, what processed the utterance voice of a reader may be used. It is preferable to output this uttered voice with a voice quality that matches the image of the character. For example, in the case of a bear, a low-pitched voice is output. In the case of fairies, high-pitched voices are output. In this case, if the uttered voice output by the portable terminal 20b is the uttered voice of the reader, it can be realized by performing frequency conversion or the like on the original uttered voice.
In the case of the third embodiment, the listener can hear the character's uttered voice from the portable terminal 20b, which further improves the sense of presence.

[Fourth embodiment]
Next, a fourth embodiment will be described. In the fourth embodiment, the reader speaks by reading a predetermined voice. In other words, the story of the picture book may be created by the reader or read from the actual book, but if the story is displayed on the mobile terminal 20a and read and spoken, the burden on the reader can be reduced. In this case, the picture book story is prepared in the management server 40 .

A block diagram showing an example of the functional configuration of the display system 1 according to the fourth embodiment is the same as that according to the first embodiment shown in FIG. 3, so description thereof will be omitted here.

FIG. 12 is a flow chart explaining an example of the operation of the display system 1 of the fourth embodiment.
First, a picture book reader operates a dedicated application using the input unit 23 of the portable terminal 20a to select a picture book to be read aloud (step 401). This can be done by the dedicated application displaying a list of picture books stored in the management server 40 and selecting from this list.
Next, the transmitting/receiving unit 41 of the management server 40 outputs the text of the picture book, which is prepared in advance and read by the reader, to the portable terminal 20a of the reader (step 402). As a result, the text is displayed on the display unit 22 of the mobile terminal 20a, and the reader reads it as the text of a picture book.

Steps 403 to 415 below are the same as steps 101 to 113 in FIG. 4 described in the first embodiment.

FIG. 13 is a diagram showing the data structure of the storage unit 45 used in the fourth embodiment.
The data structure shown is No. It consists of six items: page, text, attribute, image, and process. Among these, "No.", "text", "attribute", "image", and "processing" are the same as those described with reference to FIG. 6 in the first embodiment. "Page" represents the number of pages of the picture book.
A picture book story for which this data structure is prepared is, for example, the case shown below.

――――――――――――――――――――――――――――――――――――――――
Reader: "There was a bear in a deep, deep forest. There were three bears."
Reader: "There, a girl appeared. She was frightened and ran away."
Reader: "And Mr. Bear also started running after the girl."
Reader: “While running, Mr. Bear said, wait!”
Reader: "A girl who ran back to town asked for help."
――――――――――――――――――――――――――――――――――――――――

In this case, the utterance content of the reader is fixed, and the comprehension unit 43 checks whether the wording shown in FIG. 13 appears in the uttered voice as the wording specifying the image. Then, when the wording appears, the image acquiring section 44 acquires an image corresponding to this wording in step 409 . Also, the placement unit 46 checks whether the wording shown in FIG. 13 appears in the converted character string. Then, when the wording appears, the arrangement unit 46 performs a process of matching this feature to this image in step 411 .

In the case of the sentences described above, in FIG. 7001-No. It corresponds to the location indicated by 7009 .
Here, first, the picture books shown in FIGS. 7(a) to 7(c) are created. Specifically, as explained in FIGS. 7(a) to 7(c), the first line of the sentence "There was a bear in a deep, deep forest. There were three bears." , No. 7001-No. 7003 is referred to and the picture book of FIGS. 7A to 7C is created. That is, the arranging unit 46 arranges the image of the forest as the background using the words "forest" (FIG. 7(a)). Further, the arrangement unit 46 arranges an image of a bear as the foreground using the words "bear" (FIG. 7(b)). Then, the arranging unit 46 increases the number of bears to three with the words "three" (FIG. 7(c)).

FIGS. 14A and 14B are diagrams showing examples of picture books created after the second line of text.
In this case, according to the second line of the sentence, "A girl appeared there. The girl was surprised and ran away." 7004-No. 7005 is referred to and the picture book of FIG. 14(a) is created. Then, the girl moves quickly in the direction indicated by the arrow Ys. That is, the arranging unit 46 arranges the image of the girl as the foreground using the words "girl". Then, the girl is moved quickly in the direction indicated by the arrow Ys so as to express the appearance of running away from the word "I ran away".
Next, on the third line of the sentence, "And Mr. Bear started running after the girl." 7006 is referred to, and the bear moves quickly in the direction indicated by the arrow Yk. That is, the arrangement unit 46 causes the bear to move quickly in the direction indicated by the arrow Yk so as to express the state of chasing the girl by the words "I started running".

Furthermore, according to the 4th line of the sentence, "While running, Mr. Bear said this. Wait!" 7007 is referred to, and as described in the third embodiment, the bear's uttered voice "Wait!" is output to the mobile terminal 20b of the listener.
Then, on the fifth line of the sentence, "The girl who escaped back to town asked for help." 7008-No. 7009 is referred to and the picture book of FIG. 14(b) is created. That is, the arranging unit 46 arranges the image of the town as the background using the word "town". In this case, since the number of pages changes from 1 to 2, the pages are turned. Then, the arrangement unit 46 arranges the image of the girl as the foreground using the words "girl".
In the case of the fourth embodiment, the reader can read aloud only by reading the text displayed on the mobile terminal 20a, and the burden on the reader can be reduced.

[Fifth Embodiment]
Next, a fifth embodiment will be described. In the display system 1 of the fifth embodiment, the management server 40 acquires text corresponding to the meaning of the uttered voice of the reader, arranges the acquired text, and forms a sentence as a display image. Then, the listener views the text as a display image on the portable terminal 20b.
FIG. 15 is a block diagram showing a functional configuration example of the display system 1 according to the fifth embodiment.
In this display system 1, the mobile terminals 20a and 20b have the same functional configuration as the display system 1 in the first embodiment shown in FIG. That is, the mobile terminal 20a and the mobile terminal 20b include a transmission/reception section 21 for transmitting and receiving transmission information, a display section 22 for displaying an image, an input section 23 for inputting information, and a voice acquisition section 24 for acquiring uttered voice. and Each of these functional units operates in the same manner as in the first embodiment.
On the other hand, the management server 40 includes a text acquisition section 47 instead of the image acquisition section 44 as compared with the first embodiment. Also, the operations of the grasping unit 43, the image acquisition unit 44, and the storage unit 45 are different from those of the first embodiment. Therefore, the following description will focus on this matter.

The comprehension unit 43 comprehends the meaning of the uttered voice of the reader. In this case, it is preferable to grasp all the meanings of the speech uttered by the reader by means of speech recognition or the like.
The text acquisition unit 47 is an example of element acquisition means, and acquires a display element representing the meaning grasped by the grasping unit 43 . Here, as a display element, a text corresponding to the meaning grasped by the grasping unit 43 is acquired.

The storage unit 45 stores the text acquired by the image acquisition unit 44 in association with the meaning.
The arranging unit 46 arranges the acquired text according to the comprehended meaning and uses it as display information. That is, the arranging unit 46 arranges the text acquired by the image acquiring unit 44 to compose a sentence. The arrangement unit 46 preferably arranges the text in consideration of vertical writing, horizontal writing, line breaks, etc., and forms sentences.

FIG. 16 is a flow chart explaining an example of the operation of the display system 1 of the fifth embodiment.
Steps 501 to 505 of the flowchart shown in FIG. 16 are the same as steps 101 to 105 of the flowchart shown in FIG.
After step 506, the arrangement unit 46 arranges the text to form a sentence (step 506).
Then, the transmitting/receiving section 41 transmits the information of the sentence created by the arranging section 46 to the portable terminal 20b of the listener (step 507).
The text information is received by the transmitting/receiving section 21 of the portable terminal 20b (step 508), and the text is displayed on the display section 22 (step 509). In this case also, the text can be read aloud.

According to the display system 1 detailed above, in the first to fourth embodiments, the comprehension unit 43 comprehends the meaning of the uttered voice of the reader, and the placement unit 46 displays a picture corresponding to this. Arrange and create a picture book. As a result, a picture book can be created even when the reader freely speaks and creates a story. In addition, the reader does not need to press the buttons of the controller, etc., and the change of picture book, page turning, etc. are automatically performed, so to speak, and the story progresses, so that the burden on the reader is reduced. As a result, based on the contents of the reader's story, the picture book is displayed in real time on the listener's mobile terminal, etc., so that the listener can feel the realism on the mobile terminal while listening to the reader's utterance. You can enjoy the overflowing picture books. Further, in the fifth embodiment, the comprehension unit 43 comprehends the meaning of the uttered voice of the reader, and the arrangement unit 46 arranges the text accordingly to create sentences. This enables reading aloud based on the text.
In addition, by using the portable terminal 20, the restrictions on the place where the reading is performed are relaxed, and the reader and the listener can read the story at their favorite place. Furthermore, having a mobile terminal 20 that can be carried by each person improves convenience.

Further, in the embodiment described in detail above, the display system 1 is configured by connecting the mobile terminal 20 and the management server 40 via the network 70, the network 80, and the access point 90, but only the management server 40 is configured. However, it can be regarded as a display system. Also, the processing performed by the management server 40 can be performed similarly on the mobile terminal 20 . Therefore, in this case, the portable terminal 20 can also be regarded as a display system.

Furthermore, in the example described above, an example using the mobile terminal 20 was shown, but the present invention is not limited to this. For example, instead of the mobile terminal 20, a desktop computer or television can be used.
Furthermore, in the above example, the reader and listener are close to each other, but the present invention is not limited to this, and the reader and listener may be separated. In this case, the speech voice of the reader does not reach the listener directly. Therefore, the reader's uttered voice is sent from the mobile terminal 20a to the listener's mobile terminal 20b via the management server 40, and is output from a speaker or the like provided in the mobile terminal 20b. As a result, the listener can hear the uttered voice of the reader. At this time, the mobile terminal 20b may display not only the picture book and text but also the image of the reader captured by the mobile terminal 20a. Furthermore, the mobile terminal 20a may output the image of the listener acquired by the mobile terminal 20b and the voice of the listener. In this case, since the uttered voices of the reader and the listener are acquired only by the respective mobile terminals 20a and 20b, the separating unit 42 for separating the uttered voice of the reader and the uttered voice of the listener may be unnecessary. .

<Explanation of the program>
Here, the processing performed by the management server 40 according to the present embodiment described above is prepared as a program such as application software, for example. This processing is realized through the cooperation of software and hardware resources. That is, a CPU (not shown) inside a computer provided in the management server 40 executes a program for realizing each function described above to realize each function.

Therefore, in the present embodiment, the processing performed by the management server 40 includes, in the computer, a voice acquisition function for acquiring the uttered voice of the reader, a comprehension function for comprehending the meaning of the uttered voice by the reader, and a meaning comprehended by the comprehension function. It can also be regarded as a program for realizing an element acquisition function that acquires display elements that represent , and an arrangement function that arranges the acquired display elements and uses them as display information to be viewed by listeners.
Further, in the present embodiment, the processing performed by the management server 40 includes, in the computer, a voice acquisition function for acquiring the voice uttered by the reader, a comprehension function for comprehending the meaning of the voice uttered by the reader, and a meaning comprehended by the comprehension function. It can also be regarded as a program for realizing an image acquisition function of acquiring an image according to the meaning and an arrangement function of arranging the acquired images according to the meaning to make a picture book.

It should be noted that the program that implements the present embodiment can be provided not only by communication means but also by being stored in a recording medium such as a CD-ROM.

Although the present embodiment has been described above, the technical scope of the present invention is not limited to the range described in the above embodiment. It is clear from the scope of claims that various modifications and improvements to the above embodiment are also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Display system 20, 20a, 20b... Portable terminal, 40... Management server, 41... Transmission-and-reception part, 42... Separation part, 43... Grasping part, 44... Image acquisition part, 45... Storage part, 46... Arrangement part, 47 text acquisition unit

Claims (3)

voice acquisition means for acquiring voice spoken by a reader; grasping means for grasping the meaning of the voice spoken by the reader;
an image obtaining means for obtaining an image corresponding to the meaning grasped by the grasping means;
arranging means for arranging the images acquired according to the meaning to form a picture book;
has
The image acquiring means checks whether or not a pre-registered word specifying the image is included in the uttered voice of the reader, and acquires an image corresponding to the word when the word is included. ,
When the uttered voice of the reader includes feature information representing a feature of the image registered in advance and acquired by the image acquisition means, the arrangement means performs processing to match the image with the feature,
The wording and the feature information are registered in association with the number of pages indicating the pages of the picture book,
The arranging means arranges the background image and the foreground image separately,
A picture book display system, wherein when the image acquisition means acquires a new background image, it treats it as a page number corresponding to the new background image. The grasping means further grasps the meaning of the listener's uttered voice,
2. The picture book display system according to claim 1 , wherein said image obtaining means obtains an image corresponding to the meaning of said listener's uttered voice grasped by said grasping means. 2. The picture book display system according to claim 1 , further comprising separating means for separating said reader's uttered voice from said listener's uttered voice.

Images