JP7459020B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Outlier information, which is information indicating deviant behavior that deviates from the user's normal behavior, is extracted from historical information regarding user behavior, and based on the extracted deviant information, user characteristics that are manifested by deviant behavior are extracted. A technique has been disclosed for estimating user characteristics, which are characteristics associated with a given user.

JP 2019-49946 A

However, with the above-mentioned conventional technology, it is possible to estimate the characteristics of domestic users who are usually active in Japan and whose actions can be tracked from historical information, but it is possible to estimate the characteristics of domestic users who are usually active in Japan and whose actions can be tracked from historical information. Since there is no historical information and it is not possible to track the user's behavior, it is not possible to estimate the user characteristics of a foreign visitor (visitor) visiting Japan for the first time.

This application was made in view of the above, and aims to appropriately acquire data on inbound visitors.

The information processing device according to the present application includes an acquisition unit that acquires the user's voice when interacting with the user through one of the smart speakers installed at a plurality of locations , and a an identifying unit that identifies that the users who interacted with the user at different locations or at different times are the same person through voice recognition; a management unit, an estimation unit that estimates the language used by the user through voice recognition using the user's voice, and estimates that the user is a foreigner visiting Japan based on the language used by the user; The present invention is characterized by comprising a provision unit that provides information according to the language used by the user via a smart speaker or display device installed at the acquired location .

According to one aspect of the embodiment, data on inbound visitors can be appropriately acquired.

FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of an information providing device according to an embodiment. FIG. 4 is a diagram showing an example of a visitor information database. FIG. 5 is a diagram showing an example of the provided information database. FIG. 6 is a flowchart showing the processing procedure according to the embodiment. FIG. 7 is a diagram illustrating an example of a hardware configuration.

Below, the information processing device, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. In addition, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[1. Overview of information processing method]
First, with reference to FIG. 1, an overview of an information processing method performed by an information processing apparatus according to an embodiment will be described. FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. In addition, in FIG. 1, a case will be described as an example in which information is provided to and behavior tracking is performed for visitors to Japan from overseas.

As shown in FIG. 1, the information processing system 1 includes a terminal device 10, an information providing device 100, a smart speaker 200, and a camera 300. The information providing device 100, the smart speaker 200, and the camera 300 are connected to each other via a network N (see FIG. 2) so that they can communicate with each other by wire or wirelessly. In this embodiment, the information providing device 100 cooperates with a smart speaker 200 and a camera 300. The smart speaker 200 and the camera 300 are each installed at multiple locations.

The terminal device 10 is a smart device such as a smartphone or a tablet used by a user U (user), such as a visitor to Japan from overseas, and supports 4G (Generation), LTE (Long Term Evolution), etc. This is a mobile terminal device that can communicate with any server device via a wireless communication network. Further, the terminal device 10 has a screen such as a liquid crystal display, which has a touch panel function, and displays data such as content through a tap operation, a slide operation, a scroll operation, etc. from the user U using a finger or a stylus. Accepts various operations on. Note that an operation performed on an area of the screen where content is displayed may be an operation on the content. Further, the terminal device 10 may be not only a smart device but also a portable information processing device such as a notebook PC (Personal Computer).

In this embodiment, the terminal device 10 is not essential. The user U may be a person (for example, a child) who does not own the terminal device 10.

The information providing device 100 is an information processing device that collects information regarding a user U who is a visitor to Japan from overseas, and is realized by a server device, a cloud system, or the like. In this embodiment, the information providing device 100 cooperates with the smart speaker 200 and the camera 300 to collect information about the user U from the smart speaker 200 and the camera 300.

The smart speaker 200 is a speaker with an AI assistant function that supports interactive voice operations, and recognizes voice using a built-in microphone. Furthermore, the smart speaker 200 identifies the user U through voice recognition from the voice data obtained from the interaction with the user U. Furthermore, the smart speaker 200 provides information to the user U through dialogue with the user U.

The smart speaker 200 also estimates that user U is a foreign visitor to Japan based on the language used by user U during the conversation. For example, the smart speaker 200 estimates the language used by user U from user U's voice data, and estimates that user U is a foreign visitor to Japan who has come from a country or region where that language is an official language. However, there is not necessarily a one-to-one relationship between the language used and the country (for example, English, Spanish, Arabic, etc. are official languages in many countries and regions), so it is not essential to estimate user U's place of origin (mother country, etc.) from the language used.

The smart speaker 200 may also find out information related to the user U's place of origin (home country, etc.) from the user U during a dialogue (conversation) with the user U. For example, instead of asking a direct question about the name of the place of origin (home country, etc.), the smart speaker 200 may touch on topics related to the place of origin (home country, etc.) (trends, famous people, local products/famous products, etc.). Alternatively, the smart speaker 200 may infer the place of origin of the user U from the content of the conversation with the user U.

Furthermore, even if the smart speakers 200 use the same language, they may be further subdivided due to differences in dialects, accents, etc. That is, the language used is not limited to standard language, but may be a regional language (local language). For example, even if the smart speaker 200 uses the same language in a broad sense, if there is a difference in dialect or accent depending on the place of birth of the user U, the smart speaker 200 may treat the languages as different languages. Furthermore, the smart speaker 200 may estimate the user U's country of origin (America, China, etc.) and region (which region in the United States, etc.) from the dialect, accent, etc. included in the language used.

Further, the smart speaker 200 may allow the user U to specify (select) the language to use or the place of birth before interacting with the user U. In general, it is assumed that the user U desires dialogue and guidance in his or her own language, which he or she is familiar with.

Furthermore, the smart speaker 200 may estimate whether the purpose of the user U's visit to Japan is business or sightseeing based on the content of the dialogue (conversation) with the user U. The content of information provided to user U may be changed depending on the purpose of user U's visit to Japan. It is assumed that the information that user U seeks differs depending on the purpose of visiting Japan.

Note that in reality, the smart speaker 200 provides the information providing device 100 with the voice data of the user U with whom it interacted, and the information providing device 100 identifies the user U from the user U's voice data through voice recognition. , the language used by user U may be estimated, and it may be estimated that user U is a foreigner visiting Japan.

The camera 300 is an imaging device mounted on the smart speaker 200 or installed near the smart speaker 200, and photographs the user U who is interacting with the smart speaker 200. Furthermore, the camera 300 identifies the user U through image recognition from the photographed image data of the user U. However, image recognition is just one example. In reality, the method is not limited to image recognition, but may also be biometric authentication such as fingerprint authentication, face authentication, iris authentication, vein authentication, etc.

For example, there are some people (bilingual, trilingual, etc.) who are the same person but have the ability to use multiple languages. There are also people who try to communicate using English even if they are not from an English-speaking country. Even if the same person uses different languages, if it is estimated that they are the same person through image recognition, they may be treated as the same person. For example, even if the user U uses different languages depending on the location, if the camera 300 identifies the user U through image recognition, the user is treated as the user U regardless of the language used.

Furthermore, the camera 300 determines the appearance, clothes, and belongings of the user U through image recognition using the image data of the user U, and from the appearance, clothes, and belongings of the user U, the user U It may be assumed that the person is a foreigner. Furthermore, the camera 300 may estimate the preferences (objects of interest) of the user U from the user's U's appearance, clothes, belongings, or behavior. The smart speaker 200 may cooperate with the camera 300 to change the content of information provided to the user U according to the estimated preferences of the user U. For example, the smart speaker 200 may suggest products that match the estimated preferences of the user U.

Furthermore, the camera 300 may estimate the user U's country of origin (America, China, etc.) or region (which region in the U.S., etc.) from the user U's appearance, costume, belongings, etc. For example, if the user U has sunglasses from a certain manufacturer, the camera 300 infers that the user U comes from the region where the sunglasses were sold. Further, if the user U is carrying an item that is mainly owned only by residents of a certain area, the camera 300 estimates that the user has come from that area.

Note that in reality, the camera 300 provides the photographed image data of the user U to the information providing device 100, and the information providing device 100 identifies the user U from the image data of the user U through image recognition. , it may be assumed that user U is a foreigner visiting Japan.

In the example shown in FIG. 1, for convenience, the smart speaker 200 and camera 300 installed at location A are designated as smart speaker 200A and camera 300A, and the smart speaker 200 and camera 300 installed at location B are designated as smart speaker 200B and The camera 300B, smart speaker 200 and camera 300 installed at location C are referred to as smart speaker 200C and camera 300C.

Here, in the example shown in FIG. 1, pairs of smart speakers 200 and cameras 300 are shown at all locations, but in reality, smart speakers 200 and cameras 300 are located at locations ( It is sufficient to have both at location A), and at least one of the smart speaker 200 and the camera 300 is sufficient at the locations where the user U comes into contact with the user U for the second time and thereafter (location B, location C). In a place where only the smart speaker 200 is installed, the user U is identified only by voice recognition through dialogue with the user U. In a place where only the camera 300 is installed, the user U is identified only by image recognition taken by the user U.

The camera 300 may also capture a picture of the user U approaching the smart speaker 200, identify the user U using only image recognition, and provide the smart speaker 200 with information about the identified user U (such as identification information). This allows the smart speaker 200 to speak to the approaching user U in the language used by the user U.

Note that the smart speaker 200 and camera 300 described above may be portable and easily installed at each location. For example, the smart speaker 200 and the camera 300 may be mounted on or connected to a smart device, a notebook PC, or the like. Additionally, smart speaker 200 and camera 300 may be wireless devices.

Furthermore, the above smart speaker 200 and camera 300 may be installed in the same in-store terminal or digital signage (electronic signboard). Further, the smart speaker 200 and the camera 300 may be installed in a robot (such as an autonomous conversation robot) equipped with AI (Artificial Intelligence).

Further, the terminal device 10 of the user U may play the role of the smart speaker 200 and camera 300 described above. For example, when the terminal device 10 of the user U reads a two-dimensional code such as a QR code (registered trademark) placed at a store or a specific location (or displayed on a terminal installed in the store), the terminal device 10 can read the above-mentioned An application program (app) for exhibiting the functions of the smart speaker 200 and the camera 300 may be introduced (installed).

The locations (location A, location B, location C) where the smart speaker 200 and the camera 300 are installed are, for example, individual stores such as convenience stores, supermarkets, drug stores, home improvement stores, discount stores, and other retail stores, cafes, restaurants, bars, and other eating and drinking establishments. The locations may be, for example, large commercial facilities (shopping centers, outlet malls, underground shopping malls), entertainment facilities (theme parks, amusement parks, amusement parks, movie theaters, zoos, aquariums, swimming pools, bathing facilities), cultural facilities (hall, theater, library, art museum, museum), complex facilities, sports facilities, parking lots, service areas (SA) and parking areas (PA), or train stations, roadside stations, airports, ports (boating areas), and the like. The locations may be, for example, tourist information centers, tourist spots, landmarks, castles and castle ruins, temples and shrines, parks, gardens, famous places, historical sites, scenic spots, hot springs, campsites, cottages, lodges, bungalows, mountain, coast, lakeside, and riverbed surrounding facilities, tourist farms, tourist ranches, and the like. In particular, it would be preferable if the location is a popular place or one that is likely to attract visitors from overseas.

In addition to the above, places where the smart speaker 200 and camera 300 are installed include, for example, apartment complexes such as condominiums and apartments, detached houses, office buildings such as companies, commercial facilities such as stores, and accommodation facilities such as hotels. , educational institutions such as schools, medical institutions such as hospitals, research institutions such as research institutes, industrial plants such as factories, distribution bases such as distribution centers, etc.

[1-1. Providing information to and tracking the behavior of overseas visitors to Japan]
Now, with reference to FIG. 1, the provision of information to and behavior tracking of visitors from overseas to Japan will be described.

As shown in FIG. 1, the smart speaker 200 installed at location A acquires the voice data of the user U with whom the user U interacted, and the image data of the user U captured by the camera 300 installed at the location A is acquired ( Step S1). At this time, the smart speaker 200 and the camera 300 may transmit the acquired audio data and image data of the user U to the information providing device 100, respectively.

Next, the smart speaker 200 identifies the user U through voice recognition using the user U's voice data, and the camera 300 identifies the user U through image recognition using the user U's image data (step S2). Note that, in reality, the information providing apparatus 100 may identify the user U through voice recognition and image recognition using the user's U voice data and image data. Further, identification information for identifying the specified user U is set.

Next, the information providing device 100 provides identification information (position information is also acceptable) of the location A where the smart speaker 200 and camera 300 acquired the audio data and image data of the user U, and information regarding the date and time when the data was acquired. is associated with the identified identification information of the user U and accumulated as the action history of the user U (step S3).

Note that, in reality, the information providing device 100 obtains location information (or installed location information) and the identification information of the terminal device 10 of the user U, identify the user U from the identification information of the terminal device 10 of the user U, and obtain the location information of the smart speaker 200 and the identification information of the user U's terminal device 10. The date and time when the identification information of the device 10 was acquired may be linked and accumulated as the user U's action history.

Next, the smart speaker 200 estimates the language used by the user U through voice recognition using the voice data of the user U, and estimates that the user U is a foreigner visiting Japan from the language used (step S4). . In reality, the information providing device 100 estimates the language used by the user U through voice recognition using the voice data of the user U transmitted from the smart speaker 200, and from the language used, the information providing device 100 estimates the language used by the user U when visiting Japan. It may be assumed that the person is a foreigner. Furthermore, the camera 300 determines the appearance, clothes, and belongings of the user U through image recognition using the image data of the user U, and from the appearance, clothes, and belongings of the user U, the user U It may be assumed that the person is a foreigner.

Next, the smart speaker 200 cooperates with the information providing device 100 to provide information according to the language used by the user U (step S5). That is, the smart speaker 200 provides different information depending on the language used by the user U (changes the content of the information provided depending on the language used). The provision of information to the user U according to the language used by the user U includes, for example, information on recommended sightseeing spots, products, and services for visitors to Japan who use the language concerned. Alternatively, it may be information about stores, facilities, etc. that corresponds to the wishes (requirements) of the user U mentioned during the dialogue. In this embodiment, the smart speaker 200 provides information targeted at visitors to Japan who use the relevant language.

For example, the provision of information to user U may include guidance to other locations (location B, location C, etc.), presentation of advertisements for products and services, or issuance of coupons. Alternatively, information may be provided regarding activities (plays/experiences) that are popular among visitors to Japan who use the language concerned. Alternatively, the information may be provided regarding places (such as hidden spots) where there are not many visitors to Japan who use the language concerned. It is assumed that some visitors to Japan want to avoid places where people from the same country gather. Further, the information provision may be an emergency alert or a warning for visitors to Japan who use the language concerned.

Here, the smart speaker 200 provides information to the user U through voice guidance in the language used by the user U. Note that, in reality, the information providing device 100 may provide information to the user U via the smart speaker 200 with voice guidance in the language used by the user U. Furthermore, if a display device is installed or connected to the smart speaker 200, the smart speaker 200 may provide information by displaying a guide on the display device. That is, the guidance in the language used by the user U may be voice guidance, or may be guidance display using text and images (both may be used).

At this time, the information providing apparatus 100 may perform machine learning on the behavior history of other users who use the same language as the user U, and estimate information suitable for the user U. For example, a learning model is constructed using a machine learning method using a neural network, etc., using a dataset of languages used and places visited by users who use those languages. Then, the information providing device 100 inputs the language used by the user U into the constructed learning model, and obtains information regarding the place to be guided as an output from the learning model. Here, the information providing device 100 inputs the language used by the user U into a guide destination estimation model that is generated through machine learning using, for example, RNN (Recurrent Neural Network) or LSTM (Long short-term memory). , information regarding the place to be guided is obtained as an output from the guide destination estimation model.

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Attention can handle data such as sentences where the order in which they come before and after is important. Further, the information providing device 100 may use a similar natural language processing model. By using such a model to estimate the location to be guided from various information including the language used by user U, it is possible to estimate the location to be guided with more emphasis on the order of information. .

Further, the information providing device 100 may perform machine learning in order to improve the accuracy of identifying the user U from the user U's voice data and image data. Audio data and image data are compatible with machine learning methods. By repeating learning using a set of user U's identification information and user U's audio data and image data as correct data, it can be expected that the accuracy of identifying user U will improve. The same applies to the case where the language used by the user U is estimated from the user U's voice data.

This will resolve the language communication gap and lack of information faced by overseas visitors to Japan, and will also enable companies providing services domestically to obtain information on the flow of people and their needs.

Furthermore, in this embodiment, personal information such as attributes of the user U is not collected. Therefore, even if identification of the user U or estimation of the language used fails, there is no need to worry about leakage or leak of personal information of the user U. If identification of the user U or estimation of the language used fails, the accuracy of information provided to the user U will simply decrease.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing apparatus 100 according to the embodiment will be described using FIG. 2. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes an information providing device 100, a smart speaker 200, and a camera 300. These various devices are connected via a network N so that they can communicate by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Furthermore, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to what is illustrated. For example, in FIG. 2, only three smart speakers 200 and three cameras 300 are shown to simplify the illustration, but this is just an example and is not limited to four or more. good.

Furthermore, the smart speaker 200 and the camera 300 are connected to wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), Bluetooth (registered trademark), and wireless communication networks. It is also possible to connect to the network N via short-range wireless communication such as a LAN (Local Area Network) and communicate with the information providing device 100. Furthermore, the smart speaker 200 can also communicate with the terminal device 10 via short-range wireless communication.

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, etc. The information providing device 100 may be realized by cloud computing.

[3. Configuration example of information providing device]
Next, the configuration of the information providing apparatus 100 according to the embodiment will be described using FIG. 3. FIG. 3 is a diagram showing a configuration example of the information providing apparatus 100 according to the embodiment. As shown in FIG. 3, the information providing device 100 includes a communication section 110, a storage section 120, and a control section 130.

(Communication Department 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). Further, the communication unit 110 is connected to the network N (see FIG. 2) by wire or wirelessly.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. Ru. As shown in FIG. 3, the storage unit 120 includes a visitor information database 121 and a provided information database 122.

(Visitor information database 121)
The visitor information database 121 stores various information related to users U who are visitors (visitors) to Japan from overseas. Fig. 4 is a diagram showing an example of the visitor information database 121. In the example shown in Fig. 4, the visitor information database 121 has items such as "user ID (identifier)", "audio", "image", "acquisition location", "acquisition date and time", and "language used".

“User ID” indicates identification information for identifying user U. Note that the "user ID" may be an identification number automatically assigned by the information providing device 100 or the smart speaker 200 and camera 300.

Moreover, "audio" indicates the user U's voice data acquired by the smart speaker 200 that interacted with the user U. Note that actually, the information may be information indicating the location of the audio data (storage location: directory and file name, URL (Uniform Resource Locator), etc.).

In addition, "image" refers to image data of user U acquired by the camera 300 that photographed user U. Note that, in reality, it may also be information indicating the location of the image data (storage location: directory and file name, URL, etc.).

Further, "acquisition location" indicates information regarding the location where the smart speaker 200 acquired the audio data and the location where the camera 300 acquired the image data. That is, information regarding the installation locations of the smart speaker 200 and camera 300 is shown. Note that the information regarding the location may be location information, address, etc., or may be the name of the installation location, identification information, etc.

Furthermore, the “acquisition date and time” indicates information regarding the date and time when the smart speaker 200 acquired the audio data and the date and time when the camera 300 acquired the image data. Here, information regarding the acquisition location and acquisition date and time are linked.

Furthermore, "language used" indicates the type of language used by user U that is estimated from user U's voice data, etc. Examples of languages used include English, French, Spanish, Chinese, Russian, Arabic, and other languages used in each country or region. Languages used may also be further subdivided based on differences in dialects, accents, and the like. Here, the language used is estimated each time for each location where voice data is obtained, but in reality it may be estimated only the first time. Furthermore, if user U uses multiple languages, the number of languages used may be multiple.

For example, in the example shown in FIG. 4, the location where the audio data "audio #1A" and the image data "image #1A" of the user U identified by the user ID "U1" are acquired is "location A". , the acquired date and time is "2021/6/5", indicating that the language used by user U is "Chinese". Note that Chinese may be further subdivided into Mandarin and Cantonese.

Here, in the example shown in FIG. 4, abstract values such as "U1", "audio #1A", "image #1A", and "location A" are used for illustration, but "U1", "audio #1A" ", "Image #1A" and "Location A" store information such as specific character strings and numerical values. Below, abstract values may be illustrated in diagrams related to other information as well.

Note that the visitor information database 121 is not limited to the above, and may store various information depending on the purpose. For example, the visitor information database 121 may store various information regarding the terminal device 10 of the user U. Furthermore, the visitor information database 121 may store information regarding the place of birth (home country, etc.) of the user U, which is estimated from the language used by the user U, his appearance, and the like.

(Provided information database 122)
The provided information database 122 stores information to be provided to visitors to Japan who speak the same language. FIG. 5 is a diagram showing an example of the provided information database 122. In the example shown in FIG. 5, the provided information database 122 has items such as "language used", "place of stay", "destination visited", and "information".

“Language used” indicates the language used by foreign visitors to Japan (visitors).

Furthermore, the “place of stay” indicates a place where a visitor to Japan who uses the relevant language has interacted with the smart speaker 200 in the past. That is, the place of stay indicates a place where information has been provided to other visitors to Japan who use the language concerned in the past. Note that the "place of stay" is checked against the location of the smart speaker 200 that is currently communicating with the visitor to Japan using the language in question (the current location of the visitor in Japan with whom the visitor is in conversation in the language in question).

Furthermore, the "visited destination" indicates the next place (destination, etc.) that a visitor to Japan who uses the language used in the past went from the place of stay. That is, the visited destination indicates the next place where other visitors to Japan who used the language used in the past went from their place of stay. Note that the visited destinations are extracted from the behavior history of a group of visitors to Japan who used the language in question in the past. At this time, each destination may be ranked based on the number of visitors to Japan who have visited each destination in the past, and arranged in descending order (destination ranking).

Further, "information" indicates information regarding the place (visit destination) where the visitor to Japan with whom the visitor is currently interacting (the visitor who uses the language concerned) is expected to go next from the place of stay. That is, the information indicates information to be provided to the visitor to Japan with whom the visitor is currently interacting. At this time, the information may also be arranged in the order of the ranking of the visited destinations, so that the information is provided to the visitors to Japan in the order of the ranking of the visited destinations. Note that the information is not limited to information about the place where the user is expected to go next (destination to visit), but may also be current information, recommended information, or the like. Further, the information may be an emergency alert or a warning for visitors to Japan who use the language concerned.

For example, in the example shown in Figure 5, if a visitor to Japan whose language is "Chinese" stays at "Location A," he/she tends to head to "Location B" as his/her next destination. This indicates that it is preferable to provide the latest information on location B, route guidance from location A to location B, and "information B" regarding surrounding stores and facilities. Further, the information B may be information of a category related to the place B without being directly related to the place B.

Here, in the example shown in FIG. 5, abstract values such as "location A", "location B", and "information B" are used for illustration, but "location A", "location B", and "information B" It is assumed that information such as specific character strings and numerical values is stored in . Below, abstract values may be illustrated in diagrams related to other information as well.

The provided information database 122 may store various information according to the purpose, not limited to the above. For example, the provided information database 122 may store information on the popularity (number of visitors) of the destination for which information is provided. The provided information database 122 may also store information on the number and percentage of visitors to Japan who use the language in question who actually visited the destination for which information is provided. The provided information database 122 may also store information on services and activities (play/experiences) that are popular among visitors to Japan who use the language in question, in addition to the destination. The provided information database 122 may also store information on places (hidden spots, etc.) where there are not many visitors to Japan who use the language in question. The provided information database 122 may also store information on visitors to Japan who use the language in question, classified according to the purpose of their visit to Japan (business/tourism).

(Control unit 130)
Returning to FIG. 3, the explanation will be continued. The control unit 130 is a controller, and controls the information providing apparatus 100 using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). This is realized by executing various programs (corresponding to an example of an information processing program) stored in an internal storage device using a storage area such as a RAM as a work area. In the example shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a specification unit 132, a management unit 133, an estimation unit 134, and a provision unit 135.

(Acquisition unit 131)
The acquisition unit 131 acquires audio data and image data of the user U from the smart speaker 200 and the camera 300 via the communication unit 110 when the smart speaker 200 interacts with the user U (user). For example, the acquisition unit 131 acquires voice data and image data of the user U from a terminal installed at a store with which the user U interacted. Alternatively, when the acquisition unit 131 interacts with the user U via the user U's terminal device 10 having functions as the smart speaker 200 and the camera 300, the acquisition unit 131 may receive the user U's voice from the user U's terminal device 10 Acquire data and image data.

In addition, the acquisition unit 131 obtains the location information (position information, identification information) of the in-store terminal and the identification of the terminal device 10 of the user U from the in-store terminal that has performed near field wireless communication with the terminal device 10 of the user U. You may also obtain information.

(Specific unit 132)
The identification unit 132 identifies the user U through voice recognition based on the user U's voice data and image recognition based on the user U's image data. Further, the identifying unit 132 may identify the user U from the identification information of the user U's terminal device 10.

(Management Department 133)
The management unit 133 associates the location where the user U was identified with the date and time and stores the location as the user U's action history. That is, the management unit 133 associates the location and date and time at which the user U's audio data and image data were acquired, and stores them as the user's U action history. At this time, the management unit 133 associates the location where the user U is identified with the date and time and stores it as the action history of the user U. The history information indicating that the user has visited a place guided by the information provided by 135 is accumulated.

Further, the management unit 133 may associate the location information of the in-store terminal with the date and time when the identification information of the terminal device 10 of the user U is acquired, and store the information as the user U's action history.

(Estimation unit 134)
The estimation unit 134 estimates the language used by the user U from the voice data of the user U. At this time, the estimation unit 134 may estimate the language used by the user U from the content of the dialogue (conversation) with the user U. The estimation unit 134 may also estimate the language used by the user U from image data of characters input or specified (selected) by the user U. Alternatively, the estimation unit 134 may estimate the language used by the user U from the language written in a guidebook, tourist pamphlet, or the like held by the user U. The estimation unit 134 may also estimate the language used by the user U from the language used or information related to the place of origin input or specified (selected) by the user U.

Furthermore, the estimation unit 134 estimates that the user U is a foreigner visiting Japan based on the language used by the user U. At this time, the estimation unit 134 may estimate that the user U is a foreigner visiting Japan based on the language used by the user U and the user U's appearance. Note that the appearance includes the user U's age group (generation), gender, race, and other physical characteristics of the user U. Furthermore, the estimation unit 134 may estimate that the user U is a foreigner visiting Japan based on the language used by the user U and the clothes and belongings of the user U. Note that costumes include all things worn by the person, such as clothes, headgear (hats, etc.), footwear (shoes, etc.). In addition, belongings include bags, accessories, accessories, and personal belongings. For example, the estimating unit 134 may estimate that the user U is a foreigner visiting Japan from the marks and symbols attached to the belongings, the shape and pattern of the belongings, and the like.

(Providing Unit 135)
The providing unit 135 provides information to the user U by guidance in the language used by the user U. At this time, the providing unit 135 may provide information to the user U by at least one of voice guidance and displayed guidance in the language used by the user U. In addition, the providing unit 135 changes the content of the information provided to the user U depending on the language used by the user U and at least one of the appearance, clothing, and belongings of the user U.

[4. Processing procedure]
Next, a processing procedure by the information providing device 100, smart speaker 200, and camera 300 according to the embodiment will be described using FIG. FIG. 6 is a flowchart showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the information providing device 100, the smart speaker 200, and the camera 300.

As shown in FIG. 6, the smart speaker 200 interacts with the user U, who is a visitor from overseas, etc. (step S101). At this time, the smart speaker 200 may cooperate with the information providing device 100 to notify the information providing device 100 of the content of the interaction with the user U. For example, the information providing device 100 may interact with the user U via the smart speaker 200.

Next, the smart speaker 200 acquires user U's voice data by interacting with the user U (step S102). At this time, the acquisition unit 131 of the information providing device 100 acquires user U's audio data from the smart speaker 200 via the communication unit 110.

Subsequently, the camera 300 acquires image data of the user U by photographing the user U who is interacting with the smart speaker 200 (step S103). At this time, the acquisition unit 131 of the information providing device 100 acquires image data of the user U from the camera 300 via the communication unit 110.

Next, the smart speaker 200 identifies the user U through voice recognition based on the user U's voice data, and the camera 300 identifies the user U through image recognition based on the user U's image data (step S104). At this time, the identification unit 132 of the information providing device 100 may identify the user U through voice recognition based on the user U's voice data and image recognition based on the user U's image data. Further, the acquisition unit 131 may acquire the identification information of the terminal device 10 of the user U with which the user U interacted with the smart speaker 200 and/or the camera 300 that performed short-range wireless communication with the terminal device 10. . The identification unit 132 may identify the user U from the identification information of the user U's terminal device 10. Alternatively, the acquisition unit 131 may acquire information regarding the identified user U from the smart speaker 200 and/or camera 300 that identified the user U.

Subsequently, the management unit 133 of the information providing device 100 associates the location and date and time at which the user U was identified and accumulates this as the user U's action history (step S105). At this time, the management unit 133 stores the user U's behavior history by linking the date and time for each location where the user U is identified, so that the user U can provide information. Accumulates historical information indicating that you have visited the places you were guided to.

Subsequently, the estimation unit 134 of the information providing device 100 estimates the language used by the user U from the voice data of the user U (step S106). At this time, the estimating unit 134 may estimate the language used by the user U based on the content of the interaction (conversation) with the user U. Furthermore, the estimation unit 134 may estimate the language used by the user U from the image data of characters input or specified (selected) by the user U.

Subsequently, the estimation unit 134 of the information providing device 100 estimates that the user U is a foreigner visiting Japan based on the language used by the user U (step S107). At this time, the estimation unit 134 may estimate that the user U is a foreigner visiting Japan based on the language used by the user U and the user U's appearance, costume, and belongings.

Then, the providing unit 135 of the information providing device 100 provides information according to the language used by the user U (step S108). In this embodiment, the providing unit 135 provides information targeted at visitors to Japan who use that language. For example, the providing unit 135 provides information to the user U by voice guidance in the language used by the user U via the smart speaker 200. At this time, the providing unit 135 provides the smart speaker 200 with data used to provide information to the user U. Furthermore, if a display device is mounted on or connected to the smart speaker 200, the smart speaker 200 may provide information by displaying guidance on the display device. That is, the guidance in the language used by the user U may be voice guidance or may be a guidance display using text and images (or both).

[5. Modified example]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. Therefore, a modification of the embodiment will be described below.

In the above embodiment, some or all of the processing performed by the information providing device 100 may actually be performed by the smart speaker 200. For example, the processing may be completed in a stand-alone manner (by the smart speaker 200 alone). In this case, the smart speaker 200 is assumed to have the functions of the information providing device 100 in the above embodiment. In addition, the smart speaker 200 is linked to other smart speakers 200 via the network N. Note that in the above embodiment, since the smart speaker 200 is linked to the information providing device 100, from the perspective of the user U, the processing of the information providing device 100 also appears to be performed by the smart speaker 200. In other words, from another perspective, it can be said that the smart speaker 200 is equipped with the information providing device 100.

Furthermore, in the above embodiment, the smart speaker 200 and the information providing device 100 estimate the language used by the user U from the voice data of the user U during the conversation, but in reality, the camera 300 and the information providing device 100 , the language used by the user U may be estimated from data regarding characters written or selected by the user U in written communication or the like. For example, when a touch panel type in-store terminal or the like receives character input (or character selection) from user U using a finger, stylus, etc., the language used by user U may be estimated from the characters.

In addition, in the above embodiment, the camera 300 and the information providing device 100 may estimate the language used by the user U, who is a visitor to Japan, from the language written in a guidebook, tourist brochure, etc. held by the user U. It is assumed that visitors to Japan will use guidebooks, tourist brochures, etc. that correspond to the language they use.

[6. effect〕
As described above, the information processing device (information providing device 100) according to the present application includes the acquisition unit 131 that acquires the voice of the user U (user) when interacting with the user, and the acquisition unit 131 that acquires the voice of the user U when interacting with the user an identification unit 132 that identifies the user U; a management unit 133 that associates the location and date and time at which the user U was identified and accumulates the action history of the user U; , and a providing section 135 that provides information according to the language used by the user U.

Further, the acquisition unit 131 acquires an image of the user U when interacting with the user U. The identification unit 132 identifies the user U through image recognition.

For example, the estimation unit 134 estimates that the user U is a foreigner visiting Japan based on the language used by the user U and the user U's appearance.

Furthermore, the estimating unit 134 estimates that the user U is a foreigner visiting Japan based on the language used by the user U and the costume and belongings of the user U.

In addition, the providing unit 135 changes the content of the information provided to the user U depending on the language used by the user U and at least one of the user U's appearance, clothing, and belongings.

Further, the providing unit 135 provides information to the user U at least through audio guidance and guidance display in the language used by the user U.

In addition, the management unit 133 stores the location where the user U was identified as the user U's behavior history by linking the location where the user U was identified with the date and time for each location where the user U was identified, thereby storing the history information indicating that the user U visited the location guided by the information provided by the provision unit 135.

Furthermore, the acquisition unit 131 acquires the voice of the user U from the in-store terminal with which the user U interacted.

The acquisition unit 131 also acquires the voice of the user U from the user U's terminal device when the user U interacts with the user U via the user's U terminal device.

Further, the acquisition unit 131 acquires the location information of the in-store terminal and the identification information of the user U's terminal device from the in-store terminal that has performed short-range wireless communication with the user U's terminal device. The identification unit 132 identifies the user U from the identification information of the user U's terminal device. The management unit 133 associates the location information of the in-store terminal with the date and time when the identification information of the user U's terminal device was acquired, and stores the information as the user U's action history. The estimation unit 134 estimates that the user U is a foreigner visiting Japan based on the language specified by the user U. The providing unit 135 provides information to the user U with guidance in the language specified by the user U.

Through any one or a combination of the above-described processes, the information processing device according to the present application can appropriately acquire data on inbound visitors.

[7. Hardware configuration]
Further, the information providing apparatus 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 7, for example. The information providing apparatus 100 will be described below as an example. FIG. 7 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected to a bus. 1090.

The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and performs various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or a SD (Secure Digital) memory card. The secondary storage device 1050 may be a cloud storage device (online storage device), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface) and HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by, for example, a USB or the like.

Further, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, output device 1010 and input device 1020 may be wireless devices.

Moreover, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends it to computing device 1030, and also sends data generated by computing device 1030 to other devices via network N.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Further, the arithmetic device 1030 of the computer 1000 may load a program obtained from another device via the network I/F 1080 onto the primary storage device 1040, and execute the loaded program. Furthermore, the arithmetic unit 1030 of the computer 1000 may cooperate with other devices via the network I/F 1080, and may call and use program functions, data, etc. from other programs of other devices.

[8. others〕
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are in a so-called equivalent range. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the constituent elements can be made without departing from the gist of the embodiments described above.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the information providing apparatus 100 described above may be realized by a plurality of server computers, and depending on the function, it may be realized by calling an external platform etc. using an API (Application Programming Interface), network computing, etc. can be changed flexibly.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Information processing system 10 Terminal device 100 Information providing device 110 Communication unit 120 Storage unit 121 Visitor information database 122 Provided information database 130 Control unit 131 Acquisition unit 132 Identification unit 133 Management unit 134 Estimation unit 135 Provision unit

Claims (12)

an acquisition unit that acquires the user's voice when interacting with the user through one of the smart speakers installed at a plurality of locations ;
an identification unit that identifies that the users who interacted with the user at different locations or at different times are the same person through voice recognition using the user's voice ;
a management unit that associates the location and date and time at which the user 's voice was acquired and stores it as the user's action history;
an estimation unit that estimates the language used by the user through voice recognition using the user's voice, and estimates that the user is a foreigner visiting Japan based on the language used by the user;
a providing unit that provides information according to the language used by the user via a smart speaker or display device installed at a location where the user's voice is acquired ;
An information processing device comprising: The acquisition unit is further configured to capture an image of the user through a camera installed together with the smart speaker when the user interacts with the smart speaker through one of the smart speakers installed at the plurality of locations. Acquired,
The identification unit further identifies that the users who interacted with the user at different locations or at different times are the same person through image recognition using the image of the user ;
The estimating unit further determines the user's appearance, costume, and belongings through image recognition using the user's image, and determines that the user is a foreigner visiting Japan from the user's appearance, costume, and belongings. The information processing device according to claim 1, wherein the information processing device estimates . The information processing apparatus according to claim 2 , wherein the identification unit treats the same person as if they use different languages, when the identification unit determines that the person is the same person through image recognition . The estimation unit estimates the object of interest of the user from the user's appearance , costume, belongings, or behavior ,
The information processing apparatus according to claim 2 or 3 , wherein the providing unit changes the content of information provision to the user according to the estimated object of interest of the user . The provision unit may provide the user with information on places, products or services recommended for visitors to Japan who speak the language used, issue coupons, and provide information on places and products that meet the user's requests raised during dialogue. Or providing information about services, providing information on popular activities for visitors to Japan who use the language mentioned above, providing information about hidden places where there are not many visitors who use the language mentioned above, or providing emergency information to visitors who use the language mentioned above. Provide breaking news and alerts
The information processing device according to any one of claims 1 to 4. The estimation unit performs machine learning on the behavior history of other users who use the same language as the user, and estimates information suitable for the user,
The information processing device according to any one of claims 1 to 5, wherein the providing unit provides information suitable for the user. The management unit associates the location and date and time of the user 's voice for each location where the user 's voice was acquired, and stores it as the user's action history, so that the user receives the information. The location and the location visited by the user after receiving the information are accumulated, indicating that the user visited the location as a result of the user being guided to the recommended location by the information provision by the providing unit. The information processing device according to any one of claims 1 to 6, wherein the information processing device stores history information. The acquisition unit acquires the user's voice from an in-store terminal that has interacted with the user,
Claims 1 to 7, wherein the management unit associates the installation location of the in-store terminal that acquired the user's voice with the date and time when the user's voice was acquired, and stores the result as the user's action history. The information processing device according to any one of the above. The acquisition unit acquires the user's voice from the user's terminal device when interacting with the user via the user's terminal device ,
According to any one of claims 1 to 8 , the management unit associates a date and time with a place where the user's terminal device acquired the user's voice and stores the same as the user's action history. The information processing device described in . If the provision unit estimates that the user is a foreigner visiting Japan at the first location, and the user moves from the first location to a second location where a smart speaker is installed, The information processing device according to any one of claims 1 to 9, wherein the information processing device provides information via a smart speaker or a display device installed at the second location . An information processing method executed by an information processing device,
An acquisition step of acquiring a voice of a user when the user interacts with the user through any one of smart speakers installed in a plurality of locations ;
a step of identifying the users who have had conversations at different locations or dates and times as the same person through speech recognition using the voices of the users ;
A management process of linking the location and date and time of acquiring the user's voice to accumulate the user's behavior history;
an estimation step of estimating a language used by the user through speech recognition using the user's voice and estimating that the user is a foreign visitor to Japan based on the language used by the user;
A providing step of providing information according to the language used by the user via a smart speaker or a display device installed at a location where the user's voice is acquired ;
13. An information processing method comprising: An acquisition step of acquiring a voice of a user when the user interacts with the user through one of smart speakers installed in a plurality of locations ;
a step of identifying that the users who have had conversations at different locations or dates and times are the same person by using voice recognition using the voice of the users ;
A management procedure for linking the location and date and time of acquisition of the user's voice with each other and accumulating the user's behavior history;
an estimation step of estimating a language used by the user through speech recognition using the user's voice, and estimating that the user is a foreign visitor to Japan based on the language used by the user;
A provision step of providing information according to the language used by the user via a smart speaker or a display device installed at a location where the user's voice is acquired ;
An information processing program for causing a computer to execute the above.

Images