JP6621595B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to a technique for performing processing according to an image and sound.

  Patent Document 1 discloses an information processing apparatus that searches for information on an object in a displayed image. This information processing apparatus extracts a region including an object in an image obtained by a camera, and displays a rectangular frame representing the extracted region together with the obtained image. When the user designates an area for the displayed image and performs a search operation with a soft key, the information processing apparatus performs a search process of related information of the object in the rectangular frame and displays the search result. According to the information processing apparatus of Patent Document 1, it is possible to perform a search process on an object in an image.

JP 2010-205121 A

  By the way, when obtaining information on an object appearing in an image, there are cases where it is desired to search even for the same object, but in the information processing apparatus of Patent Document 1, the search is limited to what the object is, Various information cannot be obtained.

  An object of the present invention is to provide a technique for specifying a user instruction for an image and performing processing according to the user's intention.

The present invention provides an image data acquisition unit that acquires an image data that represents an image being displayed and has a tag, an audio data acquisition unit that acquires audio data representing a voice spoken during display of the image, and the audio Corresponding to the specifying means for specifying the audio content represented by the audio data acquired by the data acquiring means, the tag included in the image data representing the image being displayed acquired by the image data acquiring means, and the content specified by the specifying means There is provided an information processing apparatus including processing means for executing processing to be performed.

  In the present invention, the image includes a plurality of tags, and includes a selection unit that selects one or a plurality of tags from the plurality of tags, and the processing unit includes one or a plurality of tags selected by the selection unit. And a process corresponding to the contents specified by the specifying means may be executed.

In the present invention, the selection unit may select one or a plurality of tags from the plurality of tags according to the content specified by the specifying unit.
In the present invention, there is a database storing a plurality of associations between tags and words corresponding to the tags, the specifying unit specifies a word represented by the voice, and the selecting unit includes: The tag associated with the word specified by the specifying unit may be selected.

  In the present invention, the image data acquisition unit acquires coordinates representing a position in the image, the tag is associated with a position in the image, and the selection unit includes the image data It is good also as a structure which selects the tag matched with the position of the coordinate which the acquisition means acquired.

Further, the present invention represents an image being displayed, an image data acquisition step for acquiring image data having a tag, an audio data acquisition step for acquiring audio data representing an audio spoken during display of the image , A specifying step for specifying the content of the sound represented by the sound data acquired in the sound data acquiring step, a tag included in the image data representing the image being displayed acquired in the image data acquiring step, and a content specified in the specifying step An information processing method comprising a processing step for executing processing corresponding to the above.

The present invention also provides an image data acquisition means for acquiring image data representing a displayed image and having a tag, and audio data acquisition for acquiring audio data representing a voice spoken during the display of the image. Specifying means for specifying the content of the sound represented by the sound data acquired by the sound data acquiring means, the tag included in the image data indicating the image being displayed acquired by the image data acquiring means, and the specifying means A program for functioning as processing means for executing processing corresponding to the contents is provided.

  According to the present invention, it is possible to specify a user instruction for an image and perform processing according to the user's intention.

The figure which showed the apparatus which concerns on one Embodiment of this invention. The block diagram which showed the hardware constitutions of the terminal device 10. The functional block diagram of the terminal device 10. FIG. The block diagram which showed the hardware constitutions of the server apparatus 20. FIG. The functional block diagram of the server apparatus 20. FIG. The flowchart which showed the flow of the process which the control part 101 performs. The figure which showed an example of the screen of the terminal device.

[Embodiment]
<Overall configuration>
FIG. 1 is a diagram showing devices included in a voice agent system 1 according to an embodiment of the present invention. The communication network 2 includes the Internet, a fixed telephone network, a mobile communication network that provides communication services such as voice communication and data communication. The terminal device 10 is connected to the communication network 2 by wireless communication. The terminal device 10 according to the present embodiment is a smartphone. The terminal device 10 is not limited to a smartphone, and may be a wearable device such as a tablet terminal or feature phone capable of data communication, a wristwatch type terminal, a computer device specialized for data communication, or the like. Although there are many terminal devices connected to the communication network 2, only one terminal device 10 is shown in FIG. 1 in order to prevent the drawing from becoming complicated. The server device 20 is a device that provides a voice agent service to the terminal device 10. The server device 20 is connected to the communication network 2.

  The voice agent service in the present embodiment is a service that executes various processes in accordance with the utterance of the user of the terminal device 10. The voice agent service according to the present embodiment has a feature that, when an utterance is made with respect to an image displayed on the terminal device 10, processing according to the content of the image and the utterance content is executed. With this service, for example, the user can search for information by speaking to the terminal device 10 without manually inputting characters.

(Configuration of terminal device 10)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the terminal device 10. The communication unit 105 functions as a communication interface that performs wireless communication with the wireless base station of the communication network 2. The audio processing unit 107 has a microphone and a speaker. When the terminal device performs a voice call between the terminal devices, when the digital signal related to the voice of the other party is supplied from the communication unit 105, the voice processing unit 107 converts the supplied digital signal into an analog signal. This analog signal is supplied to a speaker, and the voice of the other party is emitted from the speaker. In addition, when the microphone collects sound, the sound processing unit 107 converts the collected sound into a digital signal. The voice processing unit 107 supplies a digital signal obtained by converting the user's voice to the communication unit 105 when performing a voice call on the terminal device. This digital signal is transmitted from the communication unit 105 to the communication network 2 and transmitted to the terminal device of the other party.

  The touch panel 103 is a device that combines a display device (for example, a liquid crystal display) and a sensor that detects contact of a finger on the display surface of the display device, and is an example of an operation unit operated by a user. The touch panel 103 displays characters, a GUI (Graphical User Interface), a menu screen for operating the terminal device 10, and the like on the display device. Further, the control unit 101 detects a position where the user touches with a finger using a sensor. The control unit 101 specifies a user operation according to the position detected by the touch panel 103 and the screen displayed on the touch panel, and executes control of each unit and various processes according to the specified operation.

  The positioning unit 108 includes an antenna that receives radio waves from the satellite navigation system. The positioning unit 108 measures the latitude and longitude of the position of the terminal device 10 based on the received radio waves, and sends position information indicating the measured position to the control unit 101. Output. The imaging unit 109 includes an imaging device (such as CMOS or CCD) and an optical system that forms an image on the imaging device. The imaging unit 109 generates image data representing the captured image and outputs the generated image data.

  The storage unit 102 includes a nonvolatile memory, and stores operating system programs and application programs, image data generated by the imaging unit 109, and the like. In the present embodiment, the storage unit 102 stores an application program that accesses the server device 20 and receives the voice agent service as an application program.

  The control unit 101 includes a CPU (Central Processing Unit) and a RAM (Random Access Memory). When the CPU executes an operating system program stored in the storage unit 102, a smartphone operating system is realized. Further, when the operating system is realized, an application program can be executed.

(Functional configuration of terminal device 10)
FIG. 3 is a block diagram showing a configuration of functions according to the present invention among functions realized in the terminal device 10. The image acquisition unit 1001 acquires image data generated by the imaging unit 109. The voice data generated by the voice acquisition unit 1002 and the voice processing unit 107 is acquired. The sound data is data obtained by converting a digital signal generated by the sound processing unit 107 from sound collected by a microphone. The transmission unit 1003 transmits the image data acquired by the image acquisition unit 1001 and the audio data acquired by the audio acquisition unit 1002 to the server device 20 by controlling the communication unit 105. The receiving unit 1004 receives content transmitted from the server device 20 as a response to the transmitted image data and audio data. The presentation unit 1005 controls the touch panel 103 so that the content received by the reception unit 1004 is displayed.

(Configuration of server device 20)
FIG. 4 is a diagram illustrating a hardware configuration of the server device 20. The display unit 203 includes a liquid crystal display, and displays a menu screen for operating the server device 20. The operation unit 204 has an input device such as a keyboard and a mouse. The server device 20 operates in accordance with operations performed on the keyboard and mouse. The communication unit 205 functions as a communication interface that performs data communication via the communication network 2.

  The storage unit 202 includes a hard disk device, and stores operating system programs, application programs, and the like. In the present embodiment, the storage unit 202 is a program that implements a voice agent service, acquires voice data and image data from the terminal device 10, executes a process corresponding to the acquired data, and sends an execution result to the terminal. An application program to be transmitted to the device 10 is stored.

  The control unit 201 includes a CPU and a ROM and RAM that store a boot loader. When the CPU executes the operating system program, the application program can be executed.

(Functional configuration of server device 20)
FIG. 5 is a block diagram showing a configuration of functions according to the present invention among functions realized in the terminal device 10. The data receiving unit 2001 controls the communication unit 205 to receive the image data and audio data transmitted from the terminal device 10. The data receiving unit 2001 functions as an audio data acquisition unit that acquires audio data. The image analysis unit 2002 analyzes the image data acquired by the data reception unit 2001. The image analysis unit 2002 identifies the content of the image represented by the image data by image analysis, and adds a tag corresponding to the identified content to the image data. The voice analysis unit 2003 performs voice recognition processing on the voice data received by the data reception unit 2001, and converts the voice data into text data. The voice analysis unit 2003 analyzes the text data and specifies the content of the voice represented by the voice data. The voice analysis unit 2003 functions as a specifying unit that specifies the content of the voice represented by the voice data. The data processing unit 2004 acquires image data having a tag from the image analysis unit 2002 and functions as an image data acquisition unit. The data processing unit 2004 identifies the intention of the speech utterance content based on the tag added to the acquired image data and the content of the text data identified by the speech analysis unit 2003, and performs processing according to the identified intention. Execute. In the present embodiment, the data processing unit 2004 accesses an external search engine connected to the communication network 2 and executes a search process according to the specified intention. The providing unit 2005 transmits the processing result of the data processing unit 2004 to the terminal device 10 that has transmitted the image data and the audio data.

(Operation example of embodiment)
Next, an operation example of this embodiment will be described. When the user points the imaging unit 109 of the terminal device 10 executing the application program toward the subject and performs an operation to instruct the terminal device 10 to perform shooting, the control unit 101 controls the imaging unit 109 to control the subject. Take a photo of The imaging unit 109 generates image data of the photographed subject, and the control unit 101 acquires the image data generated by the imaging unit 109. The control unit 101 controls the touch panel 103 so that an image represented by the acquired image data is displayed.

  If the user has something to look at about the subject, he tells them that he wants to look at what is in the displayed image. The voice when the user speaks is converted into a voice signal by the microphone of the voice processing unit 107, and the voice signal is converted into voice data digitized by the voice processing unit 107. The audio data is supplied from the audio processing unit 107 to the control unit 101. When acquiring the audio data supplied from the audio processing unit 107, the control unit 101 controls the communication unit 105 to transmit the acquired image data and audio data to the server device 20.

  The communication unit 205 acquires image data and audio data transmitted by the terminal device 10. The control unit 201 acquires image data and audio data acquired by the communication unit 205. When acquiring the image data and the sound data, the control unit 201 executes the process shown in FIG.

  First, the control unit 201 analyzes the image represented by the acquired image data, and adds a tag corresponding to the object shown in the image to the image data (step SA1). A tag is added to image data as metadata. Examples of techniques for analyzing images and adding tags include http://j-net21.smrj.go.jp/develop/digital/entry/001-20110223-01.html and http: //www.rbbtoday A known technique described in .com / article / 2011/05/13 / 76887.html, a known technique disclosed in JP 2010-252236 A, or the like can be used. As illustrated in FIG. 7, when there is an automobile with a model name of “Mt. Fuji” and “AAA” in the image, for example, a tag “Mt. Fuji” and a tag “AAA” are added to the image data. Next, the control unit 201 performs voice recognition processing on the voice data, and converts the user's voice into text data (step SA2).

  When the process of step SA2 ends, the control unit 201 performs a process of interpreting the intention of the user's speech content based on the content of the tag added to the image data and the user's speech content represented by the text data ( Step SA3). Specifically, the control unit 201 performs morphological analysis on the text data to identify a word included in the text data, and utters the user from a combination of the identified word and tag data added to the image data. Interpret the intent of

  This interpretation can be performed by, for example, building a database in the storage unit 202 that associates a word to be a tag with a word that may be used for the word to be a tag. For example, in this embodiment, a word such as “I want to go” or “Height” is associated with the word “Mt. Fuji” as a tag, and each vehicle type name of a car as a tag Words such as “price” and “store” are associated in the database.

  Here, it is assumed that the user's utterance with respect to the image of FIG. 7 is “I want to go”, for example. In the database, “I want to go” is associated with the word “Mt. Fuji” in the tag, and “I want to go” is not associated with the word “AAA” in the tag. In this case, the control unit 201 specifies that the user's utterance is for “Mt. Fuji” associated with the utterance content, and interprets that the user's intention is “I want to go to Mt. Fuji”.

  On the other hand, when the user's utterance is, for example, “What is the price?”, The word “AAA” is associated with “price” in the database, and the tag “Mt. Fuji” is Are not associated with "price". In this case, the control unit 201 specifies that the user's utterance is for “AAA” associated with the utterance content, and interprets that the user's intention is “AAA's price?”.

  When the control unit 201 interprets the intention of the utterance content of the user, the control unit 201 executes a process according to the interpreted utterance content (step SA4). For example, when the user's intention is “I want to go to Mt. Fuji”, an external search engine connected to the communication network 2 is used, for example, a search is performed with keywords “Mt. Fuji” and “I want to go”. When receiving a search result such as traffic access to Mt. Fuji from an external search engine, the control unit 201 controls the communication unit 205 to transmit the received search result to the terminal device 10 (step SA5). When the terminal device 10 is configured to transmit the position information generated by the positioning unit 108 together with the audio data and the image data, a route from the current location of the user to Mt. Fuji may be searched.

  When the user's intention is “What is the price of AAA?”, An external search engine connected to the communication network 2 is used to perform a search using, for example, the keywords “AAA” and “price”. When receiving the search result of the price of the model “AAA” from an external search engine, the control unit 201 controls the communication unit 205 to transmit the received search result to the terminal device 10.

  When the communication unit 105 receives the search result transmitted by the server device 20, the control unit 101 acquires the search result received by the communication unit 105. When the control unit 101 acquires the search result, the control unit 101 controls the touch panel 103 so that the acquired search result is displayed. For example, when the user utters “I want to go” with respect to the image in FIG. 7, a search result of traffic access to Mt. Fuji is displayed on the touch panel 103. Further, when the user utters “What is the price?” On the image of FIG. 7, the price of the photographed automobile is displayed on the touch panel 103.

  As described above, according to the present embodiment, the intention of the user's utterance content can be accurately identified and the process corresponding to the utterance content can be executed by the combination of the image analysis result and the sound analysis result. .

[Modification]
As mentioned above, although embodiment of this invention was described, this invention is not limited to embodiment mentioned above, It can implement with another various form. For example, the present invention may be implemented by modifying the above-described embodiment as follows. In addition, you may combine each of embodiment mentioned above and the following modifications.

  In the embodiment described above, an image obtained by shooting is displayed, and when the user utters the displayed image, processing corresponding to the displayed image and utterance content is executed. The image displayed by the terminal device 10 is not limited to the image immediately after being captured. For example, the image represented by the image data selected by the user from the image data that has been shot and stored in the storage unit 102 is displayed, and the user can display the image data of the displayed image and the displayed image. The voice data of the spoken voice may be transmitted to the server device 20. Further, the terminal device 10 displays an image of image data received from another computer device via the communication network 2 or an image of image data acquired from a recording medium removable from the terminal device 10 on the terminal device 10, and displays the image. The image data of the displayed image and the sound data of the voice spoken by the user with respect to the displayed image may be transmitted to the server device 20.

  In the present invention, the tag attached to the image data is not limited to the tag described in the above embodiment. For example, the control unit 201 adds a dish name as a tag when a dish appears in the image, and adds a person name as a tag when a person appears in the image. In addition, when the photographed image is a landscape image, the control unit 201 adds the place name of the photographed place as a tag. When the landmark is reflected in the photographed image, the control unit 201 sets the name of the landmark being reflected. Add as a tag.

  The server device 20 shows a dish in the image. For example, when the user's utterance is “eat”, the server apparatus 20 performs a search process using the tag dish name and the keyword “eat”. In addition, the server device 20 shows a dish in the image. For example, when the user's utterance is “calorie”, the server device 20 performs a search process using the tag dish name and the keyword “calorie”. When a plurality of types of dishes are shown in the image, the calories of each dish may be searched, and the calories of the plurality of dishes may be summed and presented to the user. When the professional sports player is shown in the image and the user's utterance is “What is the score?”, The server device 20 performs a search process using the name of the sports player and the keyword “score”. When the landmark appears in the image and the user's utterance is “access”, the server device 20 performs a search process using the landmark name and the keyword “access”.

  In the embodiment described above, the terminal device 10 displays a still image, but may display a moving image. When the user utters during the display of the moving image, the terminal device 10 extracts the image of the frame of the moving image when the utterance is performed, the image data of the extracted image, the voice data of the user's utterance, May be transmitted to the server device 20.

  In the embodiment described above, the server device 20 performs a search process according to the tag and the content of the utterance, but the process performed by the server device 20 is not limited to the search process. For example, when a place name of a shooting location is attached as a tag in a landscape image, and the user's utterance is “correction”, the control unit 201 sets brightness, contrast, saturation, and the like for the landscape image data. Automatic correction may be performed, and the corrected image data may be transmitted to the terminal device 10.

  In the present invention, in the terminal device 10, the user specifies an object shown in the image, and the server device 20 executes processing according to the tag attached to the specified object and the text data. You may do it. For example, when a captured image is displayed, the user makes a speech by bringing a finger into contact with the position of an object displayed in the image. The control unit 101 specifies the position where the finger is in contact with the touch panel 103, and specifies the coordinates of the position corresponding to the position of the finger in the displayed image. When the user speaks, the control unit 101 transmits image data, audio data, and coordinates of a position touched by a finger in the image to the server device 20. After adding a tag to the image data, the server device 20 identifies a tag corresponding to the object at the position touched by the user's finger in the image, and performs processing according to the identified tag and utterance content. For example, in the image shown in FIG. 7, when the user touches the position of the car with a finger and utters “What is the price?”, The control unit 201 does not refer to the database, and does not refer to the database. The image is analyzed to determine the model name of the car as a tag, and a search process is performed using the determined tag and the keyword “price”. According to this configuration, the intention of the user's utterance can be specified with higher accuracy.

  Note that, in the terminal device 10 including an imaging unit capable of photographing a user's face on the display surface side of the touch panel 103, the user's face is seen by looking at the touch panel 103, the user's line of sight is detected, and the touch panel 103 displays The coordinates of the position where the user is gazing in the displayed image may be specified. In the case of this configuration, when the user utters, the control unit 101 transmits image data, audio data, and coordinates of the position where the user gazes in the image to the server device 20. For example, in the image shown in FIG. 7, when the user gazes at a car and utters “What is the price?”, The control unit 201 does not refer to the database and analyzes the car image at the coordinates of the gaze position. Then, the model name of the car is determined as a tag, and search processing is performed using the determined tag and the keyword “price”. Even with this configuration, the intention of the user's utterance can be specified with higher accuracy.

  When the terminal device 10 is a glasses-type terminal, the range of the user's field of view is captured by the imaging unit to generate image data. When the terminal device 10 is a glasses type, the user's line of sight may be detected, and the coordinates of the position where the user is gazing in the captured image may be specified. In the case of this configuration, when the user utters, the control unit 101 transmits image data, audio data, and coordinates of the position where the user gazes in the image to the server device 20. When receiving the search result transmitted from the server device 20, the terminal device 10 displays the search result. According to this configuration, it is possible to easily search for an object that enters the user's field of view simply by speaking while shooting.

  In the present invention, a tag corresponding to an utterance may be selected according to the size of an object in an image and the focus state of each object. For example, when there is an object in which an area shown in the image is less than a predetermined threshold, an intention of the user's utterance may be interpreted by excluding a tag corresponding to the object. Moreover, it is good also as a structure which performs the search process according to the tag corresponding to the identified object and the content of utterance, specifying the object which has the best focus among the objects reflected in the image.

  When the image obtained by the image sensor of the imaging unit 109 is displayed on the touch panel 103 and the user utters while the image is displayed, the image when the utterance ends is extracted as a still image, and the extracted image The image data and the voice data of the user's utterance may be transmitted to the server device 20. According to this configuration, for example, the intention of the utterance content of the user is specified and the utterance content is determined without performing an operation for shooting such as tapping the touch panel 103 or operating a button of the terminal device 10. A corresponding process can be executed. Also in this modified example, an object shown in the image may be specified with a finger or line of sight as in the above modified example.

  In the embodiment described above, the server device 20 performs voice recognition on the voice data, but the terminal device 10 may perform voice recognition. In the case of this configuration, the terminal device 10 transmits the text data obtained by speech recognition to the server device 20 instead of the speech data. According to this configuration, the load on the server device 20 can be suppressed.

  In the embodiment described above, the server device 20 adds a tag to the image data. However, the terminal device 10 may add a tag to the image data. In the terminal device 10, when adding a tag to the image data, the user may operate the touch panel 103 to add the tag to the image data. When the server device 20 acquires the image data to which the tag is added, the server device 20 omits the process of step SA1 and executes the processes after step SA2. According to this configuration, the load on the server device 20 can be suppressed.

  In the present invention, when the user's voice is recorded and an image corresponding to the recorded voice is obtained by the imaging unit 109, processing according to the image and the content of the utterance may be performed. For example, when an image of Mt. Fuji is obtained by the imaging unit 109 after recording the voice “What is the height of Mt. Fuji,” the height of Mt. You may perform the process to search.

  The program for realizing the functions according to the present invention includes a magnetic recording medium (magnetic tape, magnetic disk (HDD (Hard Disk Drive), FD (Flexible Disk)), etc.), optical recording medium (optical disk, etc.), magneto-optical recording medium, The program may be provided in a state stored in a computer-readable recording medium such as a semiconductor memory and installed in each device. Further, the program may be downloaded via a communication network and installed in each device.

DESCRIPTION OF SYMBOLS 1 ... Voice agent system, 2 ... Communication network, 10 ... Terminal device, 20 ... Server apparatus, 101 ... Control part, 102 ... Memory | storage part, 103 ... Touch panel, 105 ... Communication part, 107 ... Voice processing part, 108 ... Positioning part 109: Imaging unit, 201: Control unit, 202 ... Storage unit, 203 ... Display unit, 204 ... Operation unit, 205 ... Communication unit, 1001 ... Image acquisition unit, 1002 ... Audio acquisition unit, 1003 ... Transmission unit, 1004 ... Receiving unit, 1005 ... presenting unit, 2001 ... data receiving unit, 2002 ... image analyzing unit, 2003 ... voice analyzing unit, 2004 ... data processing unit, 2005 ... providing unit

Claims (7)

An image data acquisition unit that represents an image being displayed and acquires image data having a tag;
Voice data acquisition means for acquiring voice data representing voice spoken during display of the image ;
Specifying means for specifying the content of the sound represented by the sound data acquired by the sound data acquiring means;
An information processing apparatus comprising: a processing unit that executes processing corresponding to a tag included in image data representing an image being displayed acquired by the image data acquisition unit and the content specified by the specifying unit. The image has a plurality of tags,
A selection means for selecting one or more tags from the plurality of tags,
The information processing apparatus according to claim 1, wherein the processing unit executes processing corresponding to one or more tags selected by the selection unit and the content specified by the specifying unit. The information processing apparatus according to claim 2, wherein the selection unit selects one or a plurality of tags from the plurality of tags according to the content specified by the specifying unit. It has a database that stores a plurality of correspondences between tags and words corresponding to the tags,
The specifying means specifies a word represented by the voice,
The selecting means selects a tag associated with the word specified by the specifying means in the database.
The information processing apparatus according to claim 3. The image data acquisition means acquires coordinates representing a position in the image;
The tag is associated with a position in the image,
The information processing apparatus according to claim 2, wherein the selection unit selects a tag associated with a coordinate position acquired by the image data acquisition unit. An image data obtaining step for representing an image being displayed and obtaining image data having a tag;
A voice data acquisition step of acquiring voice data representing voice spoken during display of the image ;
A specifying step for specifying the content of the voice represented by the voice data acquired in the voice data acquisition step;
An information processing method comprising: a processing step of executing processing corresponding to a tag included in image data representing an image being displayed acquired in the image data acquisition step and the content specified in the specifying step. Computer
An image data acquisition unit that represents an image being displayed and acquires image data having a tag;
Voice data acquisition means for acquiring voice data representing voice spoken during display of the image ;
Specifying means for specifying the content of the sound represented by the sound data acquired by the sound data acquiring means;
A program for functioning as processing means for executing processing corresponding to a tag included in image data representing an image being displayed acquired by the image data acquisition means and the content specified by the specifying means.

Images