WO2020003695A1

WO2020003695A1 - Information processing device, information processing method, and information processing system

Info

Publication number: WO2020003695A1
Application number: PCT/JP2019/015868
Authority: WO
Inventors: 俊之関矢
Original assignee: ソニー株式会社
Priority date: 2018-06-25
Filing date: 2019-04-11
Publication date: 2020-01-02

Abstract

Provided are an information processing device, information processing method, and information processing system, for assisting with use of an agent for conversing with a user. This information processing device comprises: a receiving part for receiving first information from a companion device, said first information including a photographic image of a room wherein an agent device is to be installed; a processing part for generating second information including an image whereon the agent device is superimposed in an installation position determined by comparing a result of an object recognition on the photographic image with a location suitable for the agent device and a device or location to be avoided by the agent device; and a transmission part for transmitting the second information to the companion device.

Description

Information processing apparatus, information processing method, and information processing system

The technology disclosed in this specification relates to an information processing apparatus, an information processing method, and an information processing system that support the use of an agent interacting with a user.

Recently, agents that present various information to users according to applications and situations while interacting with users using voices and the like have begun to spread. For example, besides turning on / off and adjusting operations of home appliances such as lighting and air conditioners, responding to voices when asked about weather forecasts, stock and exchange information, news, accepting product orders, and the contents of purchased books Agents that read aloud are known.

The agent function is generally provided by cooperation between an agent device installed around a user in a home or the like and an agent service built on the cloud. For example, the agent device mainly provides a user interface such as a voice input for receiving a voice spoken by the user and a voice output for responding to an inquiry from the user by voice. On the other hand, the agent service performs high-load processing such as recognition and semantic analysis of voice input by the agent device, information retrieval in response to a user inquiry, and voice synthesis based on the processing result. Some agents have a display function such as a projector in addition to voice input / output (for example, refer to Patent Document 1).

WO2016 / 158792

The purpose of the technology disclosed in this specification is to provide an information processing apparatus, an information processing method, and an information processing system that support the use of an agent that interacts with a user.

A first aspect of the technology disclosed in the present specification is as follows.
A receiving unit that receives first information including information on an environment in which the agent device is installed;
A processing unit configured to generate, based on the first information, second information that suggests an installation position of the agent device in the environment;
A transmitting unit that returns the second information;
It is an information processing apparatus including:

The first information includes a captured image of a room where the agent device is installed, and the second information includes an image in which the agent device is superimposed on a proposed installation position in the captured image. Then, the processing unit proposes an installation position of the agent device by comparing the result of object recognition of the captured image with a place where the agent device is suitable and a device or a place to be avoided.

A second aspect of the technology disclosed in the present specification is as follows.
A transmitting unit that transmits first information including information on an environment in which the agent device is installed;
A receiving unit that receives second information regarding the installation position of the agent device proposed based on the first information;
A presentation unit for presenting the second information;
It is an information processing apparatus including:

情報処理 The information processing device according to the second aspect further includes a photographing unit. Then, the transmitting unit transmits the first information including an image of the room where the agent device is installed by the imaging unit, and the receiving unit transmits the first information to a suggested installation position in the captured image. The second information including the image on which the agent device is superimposed is received, and the presentation unit displays the image on which the agent device is superimposed at a suggested installation position.

A third aspect of the technology disclosed in the present specification is as follows.
Transmitting first information including information on an environment in which the agent device is installed;
A receiving step of receiving second information on an installation position of the agent device proposed based on the first information;
A presentation step of presenting the second information;
An information processing method having the following.

A fourth aspect of the technology disclosed in the present specification is as follows.
Transmitting first information including information on an environment in which the agent device is installed, and receiving second information on an installation position of the agent device proposed based on the first information, A first device for presenting information;
A second device that receives the first information from the first device and returns the second information;
An information processing system including:

However, the term “system” as used herein refers to a logical collection of a plurality of devices (or functional modules that realize specific functions), and each device or functional module is in a single housing. It does not matter in particular.

According to the technology disclosed in this specification, it is possible to provide an information processing apparatus, an information processing method, and an information processing system that support installation of an agent interacting with a user in a room.

The effects described in the present specification are merely examples, and the effects of the present invention are not limited to these. In addition, the present invention may exhibit additional effects other than the above effects.

{Other objects, features, and advantages of the technology disclosed in this specification will become apparent from the following embodiments and the more detailed description based on the accompanying drawings.

FIG. 1 is a diagram illustrating an example of an application environment of an agent. FIG. 2 is a diagram showing a configuration example of the placement place recommendation system 200. FIG. 3 is a diagram for explaining the operation procedure of the place recommendation system 200. FIG. 4 is a diagram showing a state where candidate positions for installing agent devices in a room are determined. FIG. 5 is a diagram for explaining a method of photographing a candidate position for installing an agent device. FIG. 6 is a diagram for explaining a method of photographing a candidate position for installing an agent device. FIG. 7 is a diagram illustrating a configuration example of the agent device table. FIG. 8 is a diagram illustrating an example of an image obtained by photographing a corner where an agent device is to be placed. FIG. 9 is a diagram showing a state where candidate positions for installing agent devices are superimposed on a captured image. FIG. 10 is a diagram showing a state where candidate positions for installing agent devices are superimposed on a captured image. FIG. 11 is a diagram illustrating a state in which candidate positions for installing agent devices are superimposed on a captured image. FIG. 12 is a diagram illustrating an example of an image of the room captured by the companion device 220. FIG. 13 is a diagram illustrating an example in which the captured image illustrated in FIG. 12 is encrypted by blurring. FIG. 14 is a diagram illustrating an example in which an agent device is superimposed on a candidate position in the encrypted captured image illustrated in FIG. FIG. 15 is a diagram illustrating an example in which the image illustrated in FIG. 14 is decoded. FIG. 16 is a diagram illustrating an example of an image of a room captured by the companion device 220. FIG. 17 is a diagram illustrating an example in which the captured image illustrated in FIG. 16 is encrypted by generalization or generalization. FIG. 18 is a diagram illustrating an example in which an agent device is superimposed on a candidate position in the encrypted captured image illustrated in FIG. FIG. 19 is a diagram illustrating an example in which the image illustrated in FIG. 18 is decoded. FIG. 20 is a diagram illustrating a configuration example of the furniture information table. FIG. 21 is a diagram exemplifying a layout of a room where other furniture is placed at an optimal installation position of the agent device. FIG. 22 is a diagram exemplifying a layout of a room in which other furniture is placed at an optimal installation position of the agent device. FIG. 23 is a diagram exemplifying a layout of a room in which other furniture is placed at an optimal installation position of the agent device. FIG. 24 is a diagram exemplifying a composite image in which furniture is moved and an agent device is installed. FIG. 25 is a diagram showing an installation cost table of the agent device. FIG. 26 is a diagram illustrating an operation example of an agent device according to a change in the layout of a room. FIG. 27 is a diagram illustrating an example of changing the layout of furniture in a room. FIG. 28 is a diagram schematically illustrating a functional configuration of an agent device 210 that performs fine adjustment of a position. FIG. 29 is a diagram showing a state in which a message urging a task of improving the installation position of the agent device is displayed on the projector screen. FIG. 30 is a diagram illustrating a state in which a message for prompting a task for improving the installation position of the agent device is displayed on the projector screen. FIG. 31 is a diagram illustrating a state in which a message urging a task of improving the installation position of the agent device is displayed on the projector screen. FIG. 32 is a diagram showing a configuration example of a UI screen for drawing an illustration of a room layout. FIG. 33 is a diagram illustrating a configuration example of a UI screen in which an agent device icon is superimposed on a candidate position. FIG. 34 is a flowchart showing a basic processing procedure for recommending an optimal installation location of an agent device. FIG. 35 is a flowchart showing a processing procedure for recommending an optimal installation location of an agent device in consideration of a moving cost of furniture and an installation cost according to a distance from an outlet. FIG. 36 is a flowchart showing a processing procedure for recommending an optimal installation location of an agent device in consideration of a user's desired installation location. FIG. 37 is a flowchart showing a processing procedure for recommending an optimal installation location of an agent device in consideration of feedback from a user.

Hereinafter, embodiments of the technology disclosed in this specification will be described in detail with reference to the drawings.

エージェント An agent having a voice interaction function with a user is also called a “smart speaker”, an “AI speaker”, an “AI assistant”, or the like, but in this specification, is simply called an “agent”.

FIG. 1 shows a living room 1 as an example of an application environment of an agent. On the sideboard 13, a television receiver 11 and an agent device 12 are installed. In the living room 1, a sofa 14 is provided so as to face the television receiver 11, and a sofa table 15 is provided in front of the sofa 14.

In the example shown in FIG. 1, three parents and children are sitting on the sofa 14. The three are the viewers of the television program displayed on the television receiver 11 and the users of the agent device 12, inquire about the agent device 12, and wait for a reply. Of course, a scene in which the agent device 12 speaks autonomously to any user is also assumed. The agent device 12 mainly uses a voice UI (User @ Interface) to perform on / off and adjustment operations of home appliances such as lighting and air conditioners, and, when asked about weather forecasts, stock / exchange information, and news, sounds. Various services are provided to the user, such as answering, receiving an order for a product, and reading out the content of a purchased book. When the agent device 12 has a display function such as a projector, a service based on information display can also be provided.

(4) The sound wave of the sound emitted from the television receiver 11 reaches not only three persons sitting on the sofa 14 as direct waves, but also as reflected waves from a wall surface, a ceiling, and a floor surface. Similarly, the sound emitted from the agent device 12 reaches not only three persons sitting on the sofa 14 as direct waves, but also as reflected waves from a wall surface, a ceiling, and a floor surface.

適用 Note that the application environment of the technology disclosed in this specification is not limited to a living room shared by family members in a general home as shown in FIG. The technology disclosed in the present specification can be applied to various rooms in which an agent device is installed, such as a private room used by a specific user such as a study or a bedroom. Further, the technology disclosed in this specification can be applied not only to homes but also to corporate offices.

In order for the user to enjoy the service provided by the agent device 12, it is necessary to sufficiently consider the positional relationship between the television receiver 11 and other audio devices installed in the same room.

When the agent device 12 is installed on the sideboard 13 alongside the television receiver 11 as in the example shown in FIG. 1, the sounds output from each other overlap and reach the user's ear, so that the user Difficult questions and a problem that the agent device 12 mistakes the voice output from the television receiver 11 for a voice command from the user easily occur. In some cases, it is difficult for the user to understand the position where the voice input and voice output of the agent device 12 can be effectively performed.

In addition, when the agent device 12 has a projector display function, if the agent device 12 is installed away from a wall as in the example shown in FIG. 1, the screen may not be projected properly. In addition, if wallpaper with a pattern is stuck on the wall facing the agent device 12 or if there is unevenness, the projected image and the pattern are mixed, or the projected image becomes uneven, making it difficult to see. There is.

In short, in the room, there are an installation position suitable for sufficiently exhibiting the performance of the agent device 12 and an inappropriate installation position. However, it is not always possible for the user to accurately determine an appropriate installation position, and the agent device 12 is installed in a place not intended by the designer, and the user cannot fully enjoy the service provided by the agent device 12. Is concerned. Technical expertise may be required to install the agent device 12 at an appropriate location, and a general user may not be able to determine.

Therefore, in the present specification, the following technology for supporting a user to install the agent device 12 in a room is proposed. If this technology is used, the user can easily install the agent device 12 in a room at a position where the performance of the agent device 12 can be sufficiently exhibited. The agent device 12 can fully enjoy the service, and the user can enjoy the service provided by the agent device 12.

A. System Configuration FIG. 2 shows a configuration example of a storage location recommendation system 200 that recommends a storage location of an agent device in a room to a user. The illustrated place recommendation system 200 performs an operation of searching for an agent device 210 to be installed in a room, a companion device 220 to take the user, and a candidate for a place to install the agent device 210 in the room. It is composed of a device 230.

The agent device 210 is assumed to be equipped with almost the same hardware configuration as a general voice agent, also called “smart speaker”, “AI speaker”, “AI assistant”, and the like. However, the agent device 210 is not a dedicated device, but may be an application for an agent resident in a device such as an information home appliance or an IoT (Internet of Things) device.

The agent device 210 illustrated in FIG. 2 includes a control unit 211 including a CPU (Central Processing Unit) and a memory, a display unit 212 that externally displays information processed by the control unit 211, and an agent device 210. A sensor unit 213 for detecting information around the place and a communication unit 214 for communicating with an external device.

Expression unit 212 basically includes a speaker. The speaker may be a stereo speaker or a multi-channel speaker. Further, some speakers may be externally connected to the main body of the agent device 210. In addition, the display unit 212 may include a projector.

The configuration of the sensor unit 213 (that is, what kind of sensor element is included) is arbitrary. For example, the sensor unit 213 may include a microphone, a camera, an object detection sensor, and a depth sensor. The camera may be, for example, a camera having an angle of view of 90 degrees, an all-around camera having an angle of view of 360 degrees, or a stereo camera or a multi-lens camera. The layout of the furniture installed in the room can be detected based on the detection results of the camera, the object detection sensor, and the depth sensor. Further, the sensor unit 213 may include an environment sensor that detects an environment in the room, such as an illuminance sensor, a temperature sensor, and a humidity sensor. Further, the sensor unit 213 may include an infrared sensor or a human sensor. The sensor unit 213 may include a biological sensor that detects a user's pulse, sweating, brain waves, myoelectric potential, exhalation, and the like. Some or all of the sensor units constituting the sensor unit 213 may be externally connected to the agent device 210 or may be wirelessly connected.

The communication unit 214 interconnects with an external device of the agent device 210 using wired communication such as Ethernet (registered trademark) or wireless communication such as Wi-Fi (registered trademark). In the present embodiment, a companion device 220 is assumed as an external device of the agent device 210. For example, environmental information sensed by the sensor unit 213 and information such as the device name of the agent device 210 are transmitted to the companion device via the communication unit 214. 220.

Note that FIG. 2 mainly illustrates components necessary for the location recommendation, but it is assumed that the agent device 210 includes components other than those illustrated. The communication unit 214 may be connected to an external network such as the Internet via an access point or a router, but is not shown.

Also, although the agent device 210 may be battery-powered, it is assumed that power is supplied from a commercial power supply via a power cable (not shown) because of the need to respond to inquiries from the user for 24 hours.

The companion device 220 is a device that takes the user when recommending the location of the agent device in the room to the user. The companion device 220 is realized by executing a predetermined program (in this specification, also temporarily referred to as a “companion” application) on an information terminal carried by the user such as a smartphone or a tablet.

The companion device 220 illustrated in FIG. 2 includes a control unit 221 including a CPU and a memory, a display unit 222 that externally displays information processed by the control unit 221, an imaging unit 223, a communication unit 224, An encryption unit 225 and an information decryption unit 226 are provided.

The control unit 221 executes the companion application, and executes a process for recommending the location of the agent device 210 to the user in the room in cooperation with the arithmetic device 230.

The display unit 222 is, for example, a smartphone screen or a speaker. The photographing unit 223 is a camera mounted on an information terminal such as a smartphone, and is used mainly for photographing a scene of a room where the agent device 210 is placed.

The communication unit 224 interconnects with an external device of the companion device 220 using wired communication such as Ethernet (registered trademark) or wireless communication such as Wi-Fi (registered trademark). Of course, the communication unit 224 may correspond to a wireless communication method of a mobile communication system such as LTE (Long Term Evolution) or LTE-Advanced.

In the present embodiment, the agent device 210 and the arithmetic device 230 are assumed as external devices to which the companion device 220 is connected using the communication unit 224. For example, environment information sensed by the sensor unit 213 and information such as the device name of the agent device 210 are received from the agent device 210 via the communication unit 224. In addition, the information received from the agent device 210 is transmitted to the computing device 230 via the communication unit 224, and the information regarding the candidate position of the agent device 210 in the room is transmitted from the computing device 230 via the communication unit 224. To receive. Information on the candidate for the location of the agent device 210 is presented to the user using the display unit 222, for example.

The information transmitted / received to / from the external device via the communication unit 224 includes information related to privacy and which the user does not want to disclose to the outside. Therefore, when transmitting the information to the external device, the companion device 220 performs an encryption process for converting a part of the information that the user does not want to disclose included in the transmitted information into a format that cannot be decrypted by the external device. It is carried out in. When the information subjected to the above-described encryption processing is returned from the external device, the companion device 220 performs the decryption processing for restoring the encrypted information to the original state by the information decryption unit 226. I do. Since the companion device 220 and the agent device 210 are used at home or by the same user, the agent device 210 is an external device that does not require information encryption. On the other hand, the entity of the arithmetic device 230 is assumed to be a cloud (described later), and corresponds to an external device that requires information encryption. The details of the encryption and decryption processing will be described later.

The computing device 230 is a device that provides a “place recommendation service” that recommends the place of the agent device 210 in the room to the user in cooperation with the companion device 220. In the present embodiment, the entity of the computing device 230 is, for example, a software module that is executed on a cloud, and is assumed to provide a service that recommends a location of a large number of agent devices installed in each home or the like. . In this specification, the term “cloud” generally refers to cloud computing (Cloud Computing). The cloud provides computing services via a network such as the Internet.

The arithmetic device 230 shown in FIG. 2 includes a control unit 231 including a CPU and a memory, a communication unit 232, an agent device table 233, and a furniture information table 234.

The control unit 231 cooperates with the companion device 220 to execute an application program for recommending the location of the agent device 210 in the room to the user. The control unit 231 acquires information necessary for executing this application from the agent device table 233 and the furniture information table 234 as appropriate.

The communication unit 232 interconnects with an external device of the companion device 220 using wired communication such as Ethernet (registered trademark) or wireless communication such as Wi-Fi (registered trademark). In the present embodiment, a companion device 220 is assumed as an external device of the arithmetic device 230. From the companion device 220, information such as environment information of the agent device 210 (such as an image of the inside of the room) and a device name is received via the communication unit 232. Then, the control unit 231 calculates information on the candidate location of the agent device 210 based on the received information, and transmits the calculation result to the companion device 220 via the communication unit 232.

The agent device table 233 is a table in which Capabilities (functions), device specifications, locations suitable for installation, and devices and locations to be avoided for each device name of the agent device will be described later. Further, the furniture information table 234 is a table in which moving costs for each furniture name are collected, and details will be described later. The moving cost is obtained by quantifying the difficulty of moving (or the burden on the user for moving) based on the weight and volume of each piece of furniture, installation conditions, and the like.

Although only one companion device 220 is illustrated in the system configuration example illustrated in FIG. 2, in actuality, the computing device 230 that is a cloud server simultaneously transmits the agent device 210 to each of a large number of companion devices 220. It is assumed that a service for recommending the place of the storage is provided.

The computing device 230 is not a cloud, but a PC (Personal Computer) installed in the same room as the agent device 210 and the companion device 220, or another application that is executed in parallel on an information terminal that executes a companion application. (In the latter case, the companion device 220 and the computing device 230 will be constructed simultaneously on the same hardware device).

B. Basic Operation Next, an operation procedure of the place recommendation system 200 will be described with reference to FIG.

The process for recommending the location of the agent device 210 to the user is performed, for example, during the initial setup of the agent device 210. Alternatively, the processing may be executed when the user wants to change the location of the agent device 210 or when changing the layout of the room.

First, the user starts a companion application on the companion device 220 (step 1). Then, the user uses the photographing unit 223 to photograph a corner where the agent device 210 is to be placed (step 2).

For example, in a room 400 laid out as shown in FIG. 4, it is assumed that three places on a sideboard, a counter, and a desk indicated by reference numerals 401 to 403 are candidate positions where the user places the agent device 210. . In such a case, the user shoots each of the candidate positions 401 to 403 from inside the room (see FIG. 5), and shoots each of the candidate positions 401 to 403 with his / her back (see FIG. 5). See FIG. 6). As shown in FIG. 5, the reason why the photographing is performed toward the candidate position is to confirm the suitability of the projection plane of the projector (however, when the agent device has a projector display function). In addition, as shown in FIG. 6, the photographing is performed with the candidate position turned to the back in order to confirm the suitability of the input device such as a microphone or a camera or the acoustic environment of the speaker.

FIGS. 5 and 6 are only examples of the method of photographing a candidate position for installing an agent device, and the method is not limited thereto. For example, shooting methods such as shooting the entire room as a zoomed out subtracted image, shooting a panorama at the center of the room, shooting the front, right, rear, and left directions at every 90 degrees (at the center of the room) Are also mentioned. In addition, a shooting method in which the candidate position is overlooked, a shooting method in which the candidate position is viewed upward, and the like can also be mentioned.

The companion device 220 transmits an image of the candidate position where the agent device 210 is to be installed and data including information on the device name of the agent device 210 to the computing device 230 (step 3). The data transmission to the arithmetic device 230 is performed, for example, in the form of uploading data from a smartphone to the cloud.

Since the information transmitted from the companion device 220 to the computing device 230 includes information that the user does not want to disclose to the outside, the information is encrypted by the information encryption unit 225 and then transmitted to the computing device 230. You. The details of the encryption process will be described later.

The calculation device 230 evaluates a candidate position for installing the agent device 210 (step 4). Specifically, the arithmetic device 230 acquires information about a function or a location that conforms to the specification of the inquired agent device 210 with reference to the agent device table 233. Further, the computing device 230 performs object recognition in the room based on the transmitted captured image. At the time of object recognition, a location where a device such as a television receiver which competes with the voice input / output function of the agent device 210 is detected. When the target agent device 210 has a projector display function, it detects a white wall and determines the unevenness of the wall, determines a window that becomes an obstacle to a projected image, detects brightness, and the like. Also go. Then, the computing device 230 determines whether each candidate position in the room is a place that conforms to the functions and specifications of the agent device, or whether a device to be avoided is close to or is not a place to be avoided. Further, the computing device 230 quantifies the degree to which each candidate position matches the agent device 210 and assigns a priority.

In addition, if it is determined that the candidate position sent from the companion device 220 does not include a place suitable for the installation of the agent device 210, the arithmetic device 230 refers to the furniture information table 234. Then, a plan for moving furniture that is an obstacle at each candidate position is planned, or another candidate position is planned in the room (step 5). For example, if the vase placed on the center table hinders the user from seeing the image projected by the projector, the movement of the vase is planned.

Then, the computing device 230 prioritizes each of the candidate position, the furniture movement plan, and the candidate position planned by itself from the companion device 220, and creates proposal data of the installation position of the agent device 210. (Step 6), and transmits this to the companion device 220 (Step 7).

The companion device 220 outputs the proposal data of the installation position of the agent device 210 received from the arithmetic device 230 to the display unit 222 and presents it to the user (step 8).

Then, the user changes the installation position of the agent device 210 in the room based on the proposal data presented by the companion device 220, and moves furniture as necessary (step 9).

After installing the agent device 210 at a desired position, the user repeats photographing of the agent device 210 and uploading of the photographed image to the computing device 230, and requests the computing device 210 to verify the actual installation location. You may do so. Further, at the time of shooting, the image projected by the agent device 210 on the wall by the projector display function may be shot together. Then, the computing device 230 verifies whether the current installation position of the agent device 210 is appropriate. For example, the computing device 230 re-creates proposal data for performing a slight position adjustment so that the projection size of the projector is desired (or corrects trapezoidal distortion), and returns the proposal data to the companion device 220. I do. The companion device 220 presents the received proposal data to the user again, and the user looks at it and adjusts the position of the agent device 210.

C. Specific Operation Example Next , a specific operation example of the place recommendation system 200 will be described.

The calculation device 230 evaluates each candidate position in light of functions and device specifications of the agent device. For example, an agent device without a projector display function can evaluate candidate positions without considering the presence of a wall that projects images, but an agent device with a projector display function gives priority to the presence of a wall. And evaluate the candidate position.

Therefore, the computing device 230 summarizes Capability (function), device specifications, locations suitable for installation, and devices and locations to be avoided for each device name of the agent device, in order to evaluate the candidate position for installing the agent device. The agent device table 233 is used. FIG. 7 shows a configuration example of the agent device table 233.

(2) The companion device 220 sends an image of a corner where the agent device is to be placed and the device name of the agent device to be placed to the computing device 230. Here, an example of the operation when the captured image shown in FIG. 8 and the agent device name “XXXX” are transmitted from the companion device 220 to the arithmetic device 230 will be described.

The arithmetic device 230 recognizes the captured image sent from the companion device 220 as an object and grasps the environment such as the layout of furniture and the color of the wall in the room where the agent device is to be installed. In the case of the image shown in FIG. 8, a dining table is placed in front of a room covered with a white wall, and in the living room, a sideboard and a television receiver are installed near the wall, and a sofa is provided facing the television screen. It is possible to grasp through object recognition that a rug is laid in front of the sofa, a center table is placed on the rug, and a side table is placed on the right side of the sofa.

The computing device 230 refers to the agent device table 233 for the agent device name “XXXX”, has two microphones and one speaker, has voice operation and speaker playback functions, and Know that the center is a suitable place, and that the TV receiver, speakers, and kitchen are the equipment to be avoided or the place to be avoided.

Then, the arithmetic device 230 compares the object recognition result of the captured image with information on a place suitable for the agent device and information on a device to be avoided or a place to be avoided, which is obtained from the agent device table 233. The candidate positions are calculated with priorities. Specifically, the computing device 230 calculates a score that quantifies the degree to which each candidate position is suitable for the agent device, based on information on a location suitable for the agent device and equipment to be avoided or a location to be avoided. Then, priorities are assigned in descending order of the score. However, details of the method of calculating the score for each candidate position will be described later.

For example, it is assumed that the arithmetic device 230 has a higher score on the side table on the right side of the sofa and then on the dining table. The computing device 230 creates proposal data indicating a candidate position with a high score, and returns the proposal data to the companion device 220.

The companion device 220 outputs the proposal data of the installation position of the agent device 210 received from the computing device 230 to the display unit 222 and presents it to the user. In the present embodiment, the companion device 220 superimposes an agent device icon on a candidate position having a high score on an image of a corner where the agent device is to be placed and displays the icon on the screen. When the score is higher on the side table on the right side of the sofa and on the dining table in order, as shown in FIG. 9, the icon 901 of the agent device is displayed on the side table on the right side of the sofa and on the dining table, respectively. And 902 are displayed in a superimposed manner. The user can easily grasp where in the room it is recommended to place the agent device through a display screen as shown in FIG. Further, the user can easily grasp the situation in the room where the agent device is installed at the candidate position. A numerical value indicating the priority order may be displayed near

icons

901 and 902 of each agent device.

The process of superimposing the icon of the agent device on the captured image may be performed by the computing device 230, or may be performed by the companion device 220 based on information from the computing device 230. When the image of the icon to be superimposed corresponds to the actual device of the agent device to be placed (that is, the appearance of the agent device having the device name “XXXX”), the user can install the agent device in the room. It is easy to imagine how you are.

Next, an operation example when the captured image shown in FIG. 8 and the agent device name “YYYY” are transmitted from the companion device 220 to the computing device 230 will be described.

The computing device 230 recognizes the captured image sent from the companion device 220 as an object and can grasp the environment such as the layout of furniture and the color of the wall in the room where the agent device is to be installed (same as above). .

The computing device 230 refers to the agent device table 233 for the agent device name “YYYY”, and is provided with three microphones, a camera with a 360-degree angle of view, and one speaker. It is understood that the camera has the function and that the center of the room is a suitable place for 360-degree (all-around) shooting, and that the television receiver, the speaker, the kitchen, and the window are devices or places to avoid.

Then, the computing device 230 calculates a score that quantifies the degree to which each candidate position is suitable for the agent device, based on the information on the location suitable for the agent device and the equipment to be avoided or the location to be avoided, Assign priorities in descending order of score. However, details of the score calculation method will be described later. Locations near windows that are prone to oblique light when the user approaches the camera are avoided. For example, it is assumed that the arithmetic device 230 has a high score near the center of the room on the dining table. The computing device 230 creates proposal data indicating a candidate position with a high score, and returns the proposal data to the companion device 220.

The companion device 220 outputs the proposal data of the installation position of the agent device 210 received from the computing device 230 to the display unit 222 and presents it to the user. FIG. 10 shows a display example of a screen in which the icon 1001 of the agent device is superimposed on a candidate position near the center of the room on the dining table having a high score. It is preferable that the icon image corresponds to the actual device of the agent device to be placed (that is, the appearance of the agent device having the device name “YYYY”). The user can easily grasp where in the room it is recommended to place the agent device through a display screen as shown in FIG. Further, the user can easily grasp the situation in the room where the agent device is installed at the candidate position.

Next, an operation example when the captured image shown in FIG. 8 and the agent device name “ZZZZ” are transmitted from the companion device 220 to the computing device 230 will be described.

Also, the arithmetic device 230 refers to the agent device table 233 for the agent device name “ZZZZ”, includes three microphones, one speaker, and a projector, and has voice operation, projector display, and speaker playback functions. In addition, it is understood that a place near a white flat wall is suitable for projecting an image, and that a television receiver, a speaker, a kitchen, and a window are devices to be avoided or a place to be avoided.

Then, the computing device 230 calculates a score that quantifies the degree to which each candidate position is suitable for the agent device, based on the information on the location suitable for the agent device and the equipment to be avoided or the location to be avoided, Assign priorities in descending order of score. However, details of the score calculation method will be described later. Windows where images cannot be projected and places where obstacles covering the wall such as television receivers are installed are avoided. In addition, a wall that is lit by another light source and is bright, such as near a window or near a lighting fixture, is avoided. For example, it is assumed that the computing device 230 has a high score on a dining table near a wall of a room. The computing device 230 creates proposal data indicating a candidate position with a high score, and returns the proposal data to the companion device 220.

The companion device 220 outputs the proposal data of the installation position of the agent device 210 received from the computing device 230 to the display unit 222 and presents it to the user. FIG. 11 shows a display example of a screen in which the icon 1101 of the agent device is superimposed on a candidate position near the center of the room on the dining table having a high score. It is preferable that the icon image corresponds to the actual device of the agent device to be placed (that is, the appearance of the agent device having the device name “ZZZZ”). The user can easily grasp where in the room it is recommended to place the agent device through a display screen as shown in FIG. Further, the user can easily grasp the situation in the room where the agent device is installed at the candidate position.

D. Operation Example Considering Privacy Protection Next, privacy protection in the location recommendation system 200 will be described.

As described above, the companion device 220 transmits an image of a corner where the agent device is to be placed to the outside together with the agent device name in order to enjoy the information service relating to the candidate position for installing the agent device from the computing device 230. Must be sent. However, in some cases, the photographed image includes privacy-related information that the user does not want to disclose to the outside.

Regarding the shooting method, shooting toward the candidate position, shooting with the candidate position as a back, shooting of the entire room with a pull image, shooting with a panorama at the center of the room (or front, right, and back every 90 degrees)・ Take a picture in the left direction) (described above). Increasing the camera's field of view increases the risk of privacy-related information being reflected.

Therefore, when the companion device 220 transmits a captured image to the arithmetic device 230, the information encryption unit 225 performs an encryption process of converting a part of the information that the user does not want to disclose included in the image into a format that cannot be decrypted. When the image subjected to the above-described encryption processing is returned from the arithmetic device 230, the companion device 220 performs a decryption process of restoring the encrypted information portion to the original state by the information decryption unit. 226.

{Suppose, for example, that an image of a corner where an agent device is to be placed is an image in which several families are shown as shown in FIG. In such a case, as shown in FIG. 13, the companion device 220 blurs the face area of the family, encrypts the information by the information encryption unit 225 so that the individual cannot be identified, and then transmits the information to the arithmetic device 230. I do. Note that the information encryption unit 225 may use image processing such as mosaicing in addition to blurring.

On the computing device 230 side, even in the case of an image in which a human face is blurred, the layout of furniture in the room, the position of windows, and the like can be grasped by object recognition. The position can be evaluated correctly. The arithmetic device 230 does not need to be aware of whether the captured image sent from the companion device 220 has been encrypted by blurring or the like.

As shown in FIG. 14, an image in which the icon of the agent device is superimposed on the candidate position in the photographed image in which the face is blurred is returned from the computing device 230 to the companion device 220. In such a case, the companion device 220 may directly present the user with an image in which the icons of the

agent devices

1401 and 1402 are superimposed as shown in FIG. As illustrated in FIG. 15, the information decoding unit 226 creates an image in which the icons of the

agent devices

1501 and 1502 are re-mapped to the captured image obtained by restoring the blurring processing to the original state, and presents the image to the user. Good.

Furthermore, as shown in FIG. 16, the image of a corner where the agent device is to be placed is an image in which no family members are reflected, but there are cases where the user does not want to disclose furniture or the layout of the room. In such a case, as shown in FIG. 17, the companion device 220 performs an encryption process by replacing the image with a similar image that has no problem even if it is made public, and then transmits the encrypted image to the arithmetic device 230. The similar image referred to here is, for example, a generalized image or a generalized image obtained by replacing the entire room or individual unique furniture with a general image while maintaining the same arrangement of furniture and the structure of the room. is there.

The arithmetic device 230 does not need to be aware of whether the captured image sent from the companion device 220 has been encrypted by generalization or generalization. On the computing device 230 side, even if it is a generalized image or a generalized image, if the layout of furniture in the room, the position of the window, and the like are almost the same, the same object as the original captured image can be recognized. Therefore, each candidate position for installing an agent device can be correctly evaluated.

As shown in FIG. 18, the arithmetic device 230 returns the generalized image or the image in which the icon of the agent device is superimposed on the candidate position in the generalized image to the companion device 220. In such a case, the companion device 220 may present the user with an image in which the icon of the agent device 1801 is superimposed on the image in the generalized or generalized state as shown in FIG. As shown in FIG. 19, the information decoding unit 226 may create an image in which the icon of the agent device 1901 is re-mapped to the captured image restored as before, and present the image to the user.

プライバシー Also, by replacing the captured image with an illustration instead of a generalized or generalized image, privacy-related information can be kept from being disclosed. An application that creates a room layout may be executed on the companion device 220, and the user himself may draw the room layout.

FIG. 32 shows a configuration example of a UI screen 3200 for drawing an illustration of a room layout. A room floor plan 3201 is displayed in the upper half of the illustrated UI screen, and a plurality of furniture icons 3202 are displayed in the lower half.

The floor plan 3201 can be automatically generated by, for example, taking a three-dimensional image of an actual room, converting the sides of the three-dimensional object into solid lines, and converting invisible sides into dashed lines as necessary. . Alternatively, if you do not want to disclose the room floor plan itself, the floor plan creation application may prepare a standard floor plan such as a square in advance, or the user may select from among several floor plan samples. You may make it selectable.

家具 Furniture icons 3202 represent main furniture such as a refrigerator, a television receiver, a sink, a sofa, a speaker, an outlet, a table, a sideboard, and a window in standard patterns. The user can illustrate the floor plan of the room by selecting the icon corresponding to the furniture that is actually placed in the room and arranging it at the corresponding location on the floor plan 3201 in the upper half of the screen. .

Alternatively, instead of illustration of the room layout by the user's manual operation as described above, the information encryption unit 225 may automatically convert a captured image into a room layout illustration. In this case, the information decoding unit 226 may automatically restore the floor plan illustration to the original captured image.

When the companion device 220 is, for example, a smartphone, a UI screen for drawing an illustration as shown in FIG. 32 is displayed on the screen of the smartphone. Then, the user can draw a floor plan illustration by touching the UI screen. Alternatively, a floor plan creation application may be started on an information terminal such as a tablet or a personal computer that allows easier editing than the companion device 220, and the created illustration may be transmitted from the information terminal to the companion device 220.

(5) Then, the illustration of the room layout is transmitted from the companion device 220 to the computing device 230 together with the device name of the agent device. In the illustration of the room layout, each piece of furniture is iconified, so that the computing device 230 can identify the piece of furniture more accurately than recognizing the captured image as an object. Then, the arithmetic device 230 evaluates a candidate position for installing the agent device in the illustration of the room layout. Then, the icon of the agent device is superimposed on the recommended place on the illustration and returned to the companion device 220. As shown in FIG. 33, the companion device 220 displays an illustration on which the icon 3301 of the agent device is superimposed and displayed on the UI screen 3200, and suggests the installation location of the agent to the user.

Alternatively, the companion device 220 does not send the room layout illustration itself to the arithmetic device 230, but sends geometric information indicating the shape of the room layout and numerical information such as position information of each furniture icon. You may do so. On the computing device 230 side, an illustration of the room layout can be reproduced based on the received numerical information. The computing device 230 may return the position information of the recommended installation location instead of the illustration in which the icon of the agent device is superimposed on the recommended installation location. Then, the companion device 220 may perform a process of superimposing the agent device icon 3301 on the floor plan 3200 of the room.

物体 Also, by replacing the object included in the captured image with the object name and the position information on the image, it is possible to encrypt the image in a format that cannot be decrypted more than when using the captured image or illustration. For example, assuming that an image of a corner where the agent device is to be placed is as shown in FIG. 16, the companion device 220 replaces the sofa or center table included in the image with the object name and the image shown below. And transmits it to the computing device 230.

Softa: (x0, y0, h0, w0)
Table: (x1, y1, h1, w1)
…

On the computing device 230 side, even from the object name and the positional information on the image, the layout of the furniture in the room, the position of the window, and the like can be grasped spatially or three-dimensionally in the same manner as the original captured image. Yes, and therefore, each candidate position for installing an agent device can be correctly evaluated. In this case, the computing device 230 calculates the following candidate agent positions from the object name and the position information on the image.

Agent1: (x2, y2, h2, w2)

Since the image is not sent from the companion device 220, the computing device 230 does not superimpose the icon of the agent device on the image, and returns the information on the candidate position of the agent to the companion device 220 as it is. Then, on the companion device 220 side, based on the received information on the candidate positions of the agents, the icon of the agent device may be superimposed on the corresponding position on the captured image and presented to the user. Note that only position information of an object such as furniture or an agent device is transmitted between the computing device 230 and the companion device 220, and image information having a large data amount is not transmitted, which leads to a reduction in communication bandwidth.

E. FIG. Example of Operation Considering Movement of Furniture Subsequently, a method of proposing a candidate position for installing an agent device in the place recommendation system 200 including movement of an object in a room such as furniture will be described.

In step 5 of the operation procedure of the storage location recommendation system 200 shown in FIG. 3, with reference to the furniture information table 234, a plan for moving furniture that is an obstacle at each candidate position is planned, or another plan is moved in the room. Or a candidate position for

The furniture information table 234 is a table that describes furniture that can be moved by the user and the movement cost thereof. The moving cost is obtained by quantifying the difficulty of moving (or the burden on the user for moving) based on the weight and volume of each piece of furniture, installation conditions, and the like. FIG. 20 shows a configuration example of the furniture information table 234. In the illustrated furniture information table 234, low moving costs are given to lightweight and easy-to-move furniture such as speakers, photo frames and vases, and high moving costs are given to heavy objects such as closets and refrigerators.

(4) Some proposed operation examples of candidate positions for installing agent devices including movement of objects in a room such as furniture will be described below.

For example, assume that the captured image 2100 shown in FIG. 21 and the agent device name “ZZZZ” are transmitted from the companion device 220 to the computing device 230. The computing device 230 refers to the agent device table “233” (see FIG. 7) for the agent device name “ZZZZ”, and recognizes that the target agent device has the projector display function. In addition, the computing device 230 recognizes the captured image 2100 as an object, and has no irregularities on the white wall 2102 facing the sofa 2101. Therefore, the arithmetic device 230 can find above the cabinet 2103 as a candidate position suitable for projection.

However, since the photo stand 2104 has already been placed on the cabinet 2103, it is an obstacle to installing the agent device. Therefore, the arithmetic device 230 refers to the furniture information table 234 for “photo stand” and confirms that the moving cost is low, and moves the photo stand 2104 to another location while setting the top of the cabinet 2103 as a candidate position. Is created and sent back to the companion device 220.

The arithmetic device 230, for example, removes the picture frame 2104 from the top of the cabinet 2103 and synthesizes an image (not shown) in which an agent device with the device name “ZZZZ” is installed on the captured image shown in FIG. Then, it may be transmitted to the companion device 220. The computing device 230 may transmit a message notifying that the photo frame 2104 should be moved to the companion device 220 together. The companion device 220 presents the image received from the computing device 230 to the user, and when a message accompanying the image is received, also presents it to the user, moves the photo stand 2104 and places the agent device on the cabinet 2103. The user can be prompted to install.

{Suppose that the captured image 2200 and the agent device name “ZZZZ” shown in FIG. 22 are transmitted from the companion device 220 to the arithmetic device 230. The computing device 230 refers to the agent device table “233” (see FIG. 7) for the agent device name “ZZZZ”, and recognizes that the target agent device has the projector display function. In addition, the computing device 230 recognizes the captured image 2200 as an object, and since the white wall 2202 facing the sofa 2201 has no irregularities, the computing device 230 can find the position above the sideboard 2203 as a candidate position suitable for projection. .

However, the television receiver 2204 and the vase 2205 are already placed on the sideboard 2203. The television receiver 2204 and the vase 2205 obstruct the placement of the agent device on the sideboard 2203. Furthermore, since the sound of the television receiver 2204 interferes with the sound input / output of the agent device, the device to be avoided is specified in the agent device table 233 (see FIG. 7). Therefore, the arithmetic device 230 refers to the “furniture information table 234 television receiver” and “vase” and confirms that the moving cost is not high, and sets the candidate position 2206 on the sideboard 2203 and sets the television reception. Proposal data for moving the machine 2204 away from the candidate position 2206 on the sideboard 2203 and moving the vase 2205 to another location is created and returned to the companion device 220.

The arithmetic device 230 moves the television receiver 2204 away from the candidate position 2206 on the sideboard 2203 and removes the vase 2205 from the sideboard 2203, for example, with respect to the captured image shown in FIG. Alternatively, an image (not shown) in which an agent device having the device name “ZZZZ” is installed on the sideboard 2203 may be combined and transmitted to the companion device 220. The computing device 230 may transmit a message notifying that the television receiver 2204 and the vase 2205 should be moved to the companion device 220 together. The companion device 220 presents the image received from the computing device 230 to the user, and if a message accompanying the image is received, also presents the message to the user, moves the television receiver 2204 and the vase 2205 to move the sideboard 2203 The user can be prompted to place an agent device on top.

Also, assume that the captured image 2300 shown in FIG. 23 and the agent device name “WWWW” have been sent from the companion device 220 to the computing device 230. The computing device 230 refers to the agent device table “233” (see FIG. 7) for the agent device name “WWW”, and determines that the target agent device is equipped with a camera photographing function and is a suitable place near a wall. Figure out. Also, the computing device 230 can recognize the photographed image 2200 as an object and find out the telephone stand 2301 placed on the wall as a candidate position suitable for camera photographing.

However, since the sofa 2302 on which the user sits is placed with his / her back facing the telephone stand 2301, placing the agent device on the telephone stand 2301 poses a problem that the user cannot be photographed from the front. Then, the arithmetic device 230 refers to the “sofa” in the furniture information table 234 and confirms that the moving cost is not high, and sets a candidate position on the telephone stand 2301 and sets the sofa 2302 to the telephone stand 2301. Is created, and the proposal data is moved to the companion device 220.

The arithmetic device 230 combines, for example, an image 2302 ′ in which the sofa 2302 has been turned and an image in which an agent device with the device name “WWWW” is installed on the telephone stand 2301 with the captured image shown in FIG. Then, it may be transmitted to the companion device 220. The arithmetic device 230 may transmit a message notifying that the direction of the sofa 2302 should be changed to the companion device 220 together. FIG. 24 illustrates a composite image created based on FIG. The companion device 220 presents the image received from the computing device 230 to the user, and when a message accompanying the image is received, also presents it to the user, changes the orientation of the sofa 2302, and places the message on the telephone stand 2301. The user can be prompted to install the agent device 2401.

Note that even in the same furniture category, the difficulty of movement may differ depending on the individual circumstances of each home, so that the user may be able to customize the furniture information table 234. For example, a speaker is lightweight and easy to carry, but if the user does not want to move the speaker from the viewpoint of the sound effect, the moving cost is increased, or if the vase is fixed from the viewpoint of the beauty of the room, the moving cost is increased. . In addition, if the closet is empty and it is difficult to move, the move cost may be reduced to make it easier to propose a candidate position of the agent device.

F. Operation Example Considering Wiring of Power Cable Next, a description will be given of a method of proposing a candidate position for installing an agent device in the storage location recommendation system 200, including routing of the power cable.

So far, the embodiment has been described in which the installation location of the agent device is proposed, focusing mainly on the functions of the agent device. However, even if it is an excellent location for displaying projectors and taking pictures with a camera, the distance from the outlet increases the length of the power cable, which may interfere with walking and impair the aesthetics of the room. Or In addition, there may be a problem that an extension cord is required, and the battery cannot be reached by a commercial power source, so that the battery must be driven (charging work is troublesome).

Therefore, as shown in FIG. 25, an installation cost table that defines the installation cost according to the distance from the outlet is prepared, and the arithmetic device 230 is placed in a suitable place (or a place to be avoided) according to the function of the agent device. (Device), the installation location of the agent device may be proposed in consideration of the installation cost in addition to the moving cost of the furniture.

When the arithmetic device 230 receives an image of a corner where the agent device is to be placed from the companion device 220, it recognizes the captured image as an object and searches for the position of the outlet in the room. Further, when the outlet cannot be recognized because the outlet is hidden in the captured image, the position of the outlet in the room may be estimated based on an empirical rule or learning data on the architectural design of the house.

{Circle around (5)} Then, the arithmetic device 230 searches for a place suitable for the target agent device to perform its function in consideration of the installation cost together with the furniture moving cost. For example, when two or more candidate positions having the same score are found, by considering the installation cost of each candidate position, the priority of the candidate position closest to the outlet is higher.

G. FIG. Operation Examples According to Room Layout Changes Although various operation examples have been described above, they are common in that an agent device searches for a place suitable for performing a function according to a room layout. Therefore, when the layout of the room changes, such as when moving furniture, replacement of the installation location of the agent device may be proposed.

For example, after the user has changed the room, the companion device 220 may be used to request the computing device 230 for a proposal regarding the location of the agent device 210 according to the procedure described with reference to FIG. Good.

Alternatively, when the agent device 210 detects, based on sensor information or the like acquired by the sensor unit 213, that an event, such as a change in the layout of a room, that changes the installation location suitable for the agent device 210 itself has occurred, The replacement of the device installation location may be notified to the user using the display unit 212 or may be notified to the companion device 220 via the communication unit 214.

The operation of the agent device 210 in this case will be described with reference to FIG.

The sensor unit 213 senses the environment in the room where the agent device 210 is installed (S2601), and transmits environment data to the CPU in the control unit 211 at regular intervals (S2602). The CPU should temporarily store the received environment data in the memory (S2603), read the environment data stored in the past, compare the current environment data (S2604), and change the installation location of the agent device 210 itself. It is determined whether or not the room environment has changed.

When the sensor unit 213 includes a camera, the sensor unit 213 sends an image of the room taken by the camera to the CPU in the control unit 211. Then, the CPU sequentially and temporarily stores the photographed images in the room in the memory, calculates the difference between the photographed images in the room at regular intervals, and determines the installation location of the agent device 210 itself such as movement of furniture. Detect changes in the room environment to be changed. For example, as shown in FIG. 27, by taking the image difference between the photographed image 2701 before the change and the photographed image 2702 after the change, it is possible to detect that the position and the direction of the sofa 2710 have changed.

If the sensor unit 213 includes a human sensor, the sensor unit 213 also sends human sensor data to the CPU in the control unit 211 (S2605).

Then, upon detecting a change in the room environment in which the installation location of the agent device 210 should be changed, the CPU in the control unit 211 notifies the user using the display unit 212 (S2606). For example, when the display unit 212 has a speaker reproduction function, the user is notified of the occurrence of an environmental change in which the installation location of the agent device 210 itself is to be changed in the room or an audio message for announcing the changed content. (S2607). If the display unit 212 has a projector display function, a screen indicating that the environment in the room has changed or the changed content is projected on a wall to notify the user (S2608).

However, even if voice output or screen display is performed when the user is not present, the user cannot be notified. Therefore, the CPU in the control unit 211 may execute the expression operation by the expression unit 212 at the timing when the presence of the user can be confirmed by the human sensor of the sensor unit 213.

When the CPU in the control unit 211 detects a change in the environment in the room, it also notifies the companion device 220 via the communication unit 214 (S2609). The companion device 220 also uses the display unit 222 to notify the user of a change in the room environment in which the installation location of the agent device 210 needs to be changed.

As described above, the user can know from at least one of the agent device 210 and the companion device 220 that there has been a change in the room environment in which the installation location of the agent device 210 needs to be changed. Then, similarly to the initial setup of the agent device 210, the user can photograph the room and enjoy the service of proposing the installation position of the agent device 210 according to the procedure already described with reference to FIG.

H. Proposal of fine adjustment by agent device itself In the basic operation of the place recommendation system 200 shown in FIG. 3, after the user installs the agent device 210 in the room based on the proposal data presented by the companion device 220, Further, the photographing of the agent device 210 and the uploading of the photographed image to the arithmetic device 230 are repeatedly performed to verify the appropriateness of the installation location and adjust the position of the agent device 210.

However, the arithmetic device 230 may propose a rough installation position, and may perform a fine-tuning of the actual position based on the sensing of the agent device 210 itself. There is a limit to the granularity of information that the arithmetic device 230 can acquire through object recognition of the captured image sent from the companion device 220. On the other hand, the agent device 210 estimates the surrounding environment of the installation position based on the sensing result of the sensor unit 213, and can finely adjust the position of the agent device 210 with finer granularity.

FIG. 28 schematically shows the functional configuration of the agent device 210 for performing fine adjustment of the position. However, the illustrated agent device 210 is provided with a projector display function as the display unit 212, and it is assumed that an installation position near a wall is proposed.

The agent device 210 includes a six-axis sensor 2801, a distance sensor 2802, a human sensor 2803, and the like as the sensor unit 213. The 6-axis sensor 2801 is, for example, a sensor unit that includes an inertial measurement device (Inertia Measurement Unit) (IMU) and measures the position and orientation. Further, the distance sensor 2802 may be a distance sensor using reflection of laser, ultrasonic waves, infrared rays, or the like.

The control unit 211 includes a self-orientation detection unit 2811, a wall distance estimation unit 2812, and an installation position evaluation unit 2813. These functional modules 2811 to 2813 may be, for example, software modules executed by a CPU in the control unit 211.

The self-orientation detecting unit 2811 detects the posture of the main body of the agent device 210 based on the detection signal of the six-axis sensor 2801. The wall distance estimating unit 2812 estimates the distance from the main body of the agent device 210 to the wall serving as the projection plane based on the detection signal of the distance sensor 2802. The installation position evaluation unit 2813 evaluates whether the installation position of the agent device 210 is appropriate based on the detected posture of the agent device 210 and the distance to the wall.

As a result of the evaluation, the installation position evaluation unit 2813 shows that the installation position of the main body of the agent device 210 is too far from the wall, the wall is inclined with respect to the emission direction of the projector, and the installation surface is inclined and unstable. When it is determined that the installation position is inappropriate, the user is notified by voice reproduction or the projector screen using the display unit 212. For example, "It looks like you are away from the wall. Please move it closer to the wall.", "It's a little diagonal. Isn't it hard to read my letters?", "Please put it on a straight table, slip. May be displayed on the projector screen prompting the user to improve the installation position of the agent device 210 (see FIGS. 29 to 31). Alternatively, the agent device 210 may output a voice message by voice instead of or together with the screen display.

However, even if voice output or screen display is performed when the user is not present, the user cannot be notified. Therefore, the control unit 211 may execute the expression operation by the expression unit 212 at the timing when the presence of the user can be confirmed by the motion sensor 2803.

I. Processing action for installation location recommendation
I-1. When recognizing an object such as furniture or a window installed in the room from a captured image of the room (or an illustration of the room layout) from the captured image of the room, the basic operation calculation device 230 avoids a place suitable for the target agent device or avoids it. A score score (x, y) at each position (x, y) in the room is calculated based on information on a device or a place to be set, and an optimal installation position (x _opt , y _opt ) at which the score becomes maximum is determined. And presents it to the companion device 220.

Here, when the position (x, y) of the agent device approaches a suitable place, the score rises, but decreases when the distance increases, and also increases when approaching a device and place to be avoided, but increases when moving away. Using F, the score score (x, y) at each position (x, y) in the room is defined as in the following equation (1).

{In the above equation (1),} is a parameter indicating what kind of object is where (location). When the two-dimensional position of the object is represented by (x, y) and the size of the object in the vertical and horizontal directions is represented by (h, w), the location of each object can be represented by the following expression (2). . Then, the parameter 表す can be expressed as in the following equation (3).

Therefore, the optimal installation position (x _opt , y _opt ) of the agent device can be treated as an optimization problem of searching for a position where the score score (x, y) = F (x, y | Θ) is maximum. And can be expressed as in the following equation (4).

FIG. 34 shows, in the form of a flowchart, a basic processing procedure in which the computing device 230 recommends the companion device 220 an optimal installation location of the agent device.

The arithmetic device 230 receives, from the companion device 220, the captured image of the room where the agent device is installed and the device name of the target agent device (step S3401).

The arithmetic device 230 recognizes the received captured image as an object and estimates the attribute (category), position, and size of each object in the room (step S3402). Based on this estimation result, the parameter の of the above equation (3) can be obtained.

Next, the computing device 230 refers to the agent device table 233 for the device name received in step S3401, and obtains information on a location suitable for the agent device and devices and locations to be avoided. Then, the arithmetic device 230 uses the function F, which increases the score as it approaches a device and a place to be avoided but increases the score as it moves away, and calculates the score score (x, y) when the agent device is placed at the position (x, y). ) Is calculated according to the above equation (1) (step S3403).

Next, the arithmetic device 230 solves the optimization problem of searching for a position where the score score (x, y) = F (x, y | Θ) shown in the above equation (4) is the maximum, and solves the agent device. The optimal installation position (x _opt , y _opt ) is determined (step S3404). If there are a plurality of candidate positions, the calculation is also performed for the second and third positions. In addition, prioritization of each candidate position is performed based on the score.

Then, the computing device 230 superimposes the image of the agent device on the installation position (x _opt , y _opt ) of the agent device determined in step S3404 on the captured image received in step S3401, and transmits the image to the request source. Is transmitted to the companion device 220 (step S3405), and this processing ends.

I-2. When defining the recommendation function F of the installation location in consideration of the moving cost of the furniture and the distance from the outlet , in addition to the parameter に関する regarding the location suitable for the target agent device and the equipment and location to be avoided, The parameter に関する relating to the moving cost of each scattered piece of furniture and the installation cost based on the distance from the outlet can be considered. By considering the movement of the furniture, it becomes possible to determine the optimum installation position of the agent device from a wider range. In addition, by considering the distance from the outlet, it is possible to determine an installation position where the wiring length of the power cable is short. The parameter Ψ can be expressed as in the following equation (5). However, the parameter 示す shown in the following equation (5) includes the moving cost of each object recognized from the captured image and the installation cost according to the distance from the outlet. g _TV is the moving cost of the television receiver, g _Table is the moving cost of the table, and g _c is the installation cost according to the distance from the outlet.

In this case, the score score (x, y) at each position (x, y) in the room can be defined as in the following expression (6). Then, the optimal installation position (x _opt , y _opt ) of the agent device is treated as an optimization problem of searching for a position at which the score score (x, y) = F (x, y | Θ, Ψ) is maximum. And can be expressed as in the following equation (7).

FIG. 35 is a flowchart showing a processing procedure for the arithmetic device 230 to recommend an optimal installation location of the agent device to the companion device 220 in consideration of the moving cost of furniture and the installation cost according to the distance from the outlet. In the format

The arithmetic device 230 receives, from the companion device 220, the captured image of the room where the agent device is installed and the device name of the target agent device (step S3501).

The arithmetic device 230 recognizes the received captured image as an object and estimates the attribute (category), position, and size of each object in the room (step S3502). Based on this estimation result, the parameter Θ shown in the above equation (3) can be obtained.

In addition, the computing device 230 refers to the furniture information table 234 (see FIG. 20) and the installation cost table (see FIG. 25), and obtains parameters relating to furniture moving costs and installation costs according to outlet lengths. Ψ is acquired (step S3503).

Next, the computing device 230 refers to the agent device table 233 for the device name received in step S3501, and acquires information on a place suitable for the agent device and a device and a place to be avoided. Then, the arithmetic device 230 uses the function F, which increases the score as it approaches a device and a place to be avoided but increases the score as it moves away, and calculates the score score (x, y) when the agent device is placed at the position (x, y). ) Is calculated according to the above equation (6) (step S3504).

Next, the computing device 230 solves the optimization problem of searching for a position where the score score (x, y) = F (x, y | Θ, Ψ), which is shown in the above equation (7), is maximum, An optimal installation position (x _opt , y _opt ) of the agent device is determined in consideration of the movement of the furniture and the distance from the outlet (step S3505). If there are a plurality of candidate positions, the calculation is also performed for the second and third positions. In addition, prioritization of each candidate position is performed based on the score.

Then, the arithmetic device 230 superimposes the image of the agent device on the installation position (x _opt , y _opt ) of the agent device determined in step S3505 on the captured image received in step S3501, and the image of the furniture after movement. Are combined, and the image is transmitted to the companion device 220 of the request source (step S3506), and this processing ends.

I-3. There is a case where a recommended user of an installation location in consideration of a desired installation location of the user desires an installation location of the agent device. For example, if you consider the agent device as furniture or furniture and are aesthetically observant about the installation position in the room, or if you plan to place other items and want to limit the installation location of the agent device, the reasons are: Various.

When the position where the user desires to install the agent device is (x _u , y _u ), the score score (x, y) shown in the above equation (6) is calculated, and the score shown in the above equation (7) is given. The solution of the optimization problem is implemented under the condition shown in the following equation (8).

FIG. 36 shows, in the form of a flowchart, a processing procedure in which the computing device 230 recommends the companion device 220 an optimal installation location of the agent device in consideration of a user's desired installation location.

The arithmetic device 230 receives, from the companion device 220, the captured image of the room where the agent device is installed and the device name of the target agent device (step S3601).

The arithmetic device 230 recognizes the received captured image as an object and estimates the attribute (category), position, and size of each object in the room (step S3602). Based on this estimation result, the parameter Θ shown in the above equation (3) can be obtained.

In addition, the computing device 230 refers to the furniture information table 234 (see FIG. 20) and the installation cost table (see FIG. 25), and obtains parameters relating to furniture moving costs and installation costs according to outlet lengths. Ψ is acquired (step S3603).

Furthermore, computing device 230, the installation location of the agent device installed desired position the user agent device desires (x _u, y _u) acquires (step S3604). The method by which the computing device 230 acquires the desired installation position (x _u , _yu ) is arbitrary. For example, the user touches or clicks on the captured image display screen to indicate a desired installation position, and the companion device 220 transmits the input desired installation position (x _u , _yu ) to the arithmetic device 230. Is also good.

Next, the computing device 230 refers to the agent device table 233 for the device name received in step S3601, and acquires information on a place suitable for the agent device and a device and a place to be avoided. Then, the arithmetic device 230 uses the function F, which increases the score as it approaches a device and a place to be avoided but increases the score as it moves away, and calculates the score score (x, y) when the agent device is placed at the position (x, y). ) Is calculated according to the above equation (6) under the condition of the installation position shown in the above equation (8) (step S3605).

Next, the arithmetic device 230 calculates the score score (x, y) = F (x, y | Θ, Ψ) shown in the above equation (7) under the condition of the installation position shown in the above equation (8). Solving the optimization problem of searching for the maximum position, allowing the furniture to move, and considering the distance from the outlet, determines the optimal installation position (x _opt , y _opt ) of the agent device ( Step S3606). If there are a plurality of candidate positions, the calculation is also performed for the second and third positions. In addition, prioritization of each candidate position is performed based on the score.

Next, the arithmetic device 230 calculates the score (x _opt , y _opt ) at the optimal installation position (x _opt , y _opt ) determined in step S3606 according to the following equation (9), and the score is determined by the predetermined threshold th. It is checked whether it is the above (step S3607).

If the score (x _opt , y _opt ) at the optimum installation position (x _opt , y _opt ) is equal to or greater than the predetermined threshold th (Yes in step S3607), the arithmetic device 230 returns to the captured image received in step S3601. Above, the image of the agent device is superimposed on the installation position (x _opt , y _opt ) of the agent device determined in step S3606, and the image of the furniture after movement is synthesized, and the image is transmitted to the companion device 220 of the request source. (Step S3608), and the process ends.

On the other hand, when the score (x _opt , y _opt ) at the optimal installation position (x _opt , y _opt ) is less than the predetermined threshold th (No in step S3607), the arithmetic device 230 acquires in step S3604. installation desired position of the user (x _u, y _u) is notified that no suitable location of the agent device in companion device 220 (step S3609), the process ends.

I-4. According to the processing procedure shown in any one of FIGS. 34 to 36, the installation operation recommendation calculation device 230 in consideration of the feedback from the user converts the image in which the agent device is superimposed on the optimal installation position to the requesting companion device 220. Reply to On the companion device 220 side, it is recommended that the image on which the agent device is superimposed be presented to the user, and that the agent device be placed at an optimal installation position. The user looks at the image on which the agent device is superimposed, or actually puts the agent device at that location, and determines whether to permit the recommended location. Further, the companion device 220 may feed back to the arithmetic device 230 whether or not the user has approved the recommended installation position.

FIG. 37 shows, in the form of a flowchart, a processing procedure for the arithmetic device 230 to recommend an optimal installation location of an agent device in consideration of feedback from a user.

The arithmetic device 230 receives, from the companion device 220, the captured image of the room where the agent device is installed and the device name of the target agent device (step S3701). Then, the arithmetic device 230 performs the same processing as in steps S3602 to S3608 of the flowchart shown in FIG. 36, for example, to determine the optimal installation position of the agent device, and superimposes the image of the agent device on that location. The image is transmitted to the companion device 220 (step S3702).

(4) Thereafter, upon receiving the user's response regarding the above-described optimal installation position from the companion device 220 (step S3703), the arithmetic device 230 checks whether the user has approved the recommended installation position (step S3704).

Here, if the user permits the recommended installation position (Yes in step S3704), the process ends.

On the other hand, when the recommendation was installed position the user is not licensed (No in step S 3704), the computing device 230, obtains the installation position of the agent device (x _u, y _u) the installation desired position the user agent device desires (Step S3705). The method by which the computing device 230 obtains the desired installation position (x _u , _yu ) is arbitrary (as described above).

Next, the computing device 230 refers to the agent device table 233 for the device name received in step S3701, and acquires information on a place suitable for the agent device and a device and a place to be avoided. Then, the arithmetic device 230 uses the function F, which increases the score as it approaches a device and a place to be avoided but increases the score as it moves away, and calculates the score score (x, y) when the agent device is placed at the position (x, y). ) Is calculated according to the above equation (6) under the condition of the installation position shown in the above equation (8) (step S3706).

Next, the arithmetic device 230 calculates the score score (x, y) = F (x, y | Θ, Ψ) shown in the above equation (7) under the condition of the installation position shown in the above equation (8). Solving the optimization problem of searching for the maximum position, allowing the furniture to move, and considering the distance from the outlet, determines the optimal installation position (x _opt , y _opt ) of the agent device ( Step S3707). If there are a plurality of candidate positions, the calculation is also performed for the second and third positions. In addition, prioritization of each candidate position is performed based on the score.

Next, the arithmetic device 230 calculates a score (x _opt , y _opt ) at the optimal installation position (x _opt , y _opt ) determined in step S3606 according to the above equation (9), and the score is determined by a predetermined threshold th. It is checked whether it is the above (step S3708).

Optimal placement (x _{_opt,} y _opt) score in score (x _{_opt,} y _opt) if is the predetermined threshold th or more (Yes in step S3708), the computing device 230, captured image received in step S3701 Above, the image of the agent device is superimposed on the installation position (x _opt , y _opt ) of the agent device determined in step S3606, and the image of the furniture after movement is synthesized, and the image is transmitted to the companion device 220 of the request source. (Step S3709), and the process ends.

On the other hand, when the score score (x _opt , y _opt ) at the optimal installation position (x _opt , y _opt ) is less than the predetermined threshold th (No in step S3708), the arithmetic device 230 acquires in step S3705. installation desired position of the user (x _u, y _u) is notified that no suitable location of the agent device in companion device 220 (step S3710), the process ends.

J. Conclusion According to the storage location recommendation system 200 according to the present embodiment, the optimum position where the agent device 210 is installed is presented only by transmitting the image of the room photographed by the companion device 220 to the arithmetic device 230.

The companion device 220 is realized in the form of executing a companion application on an information terminal such as a smartphone or a tablet, for example. The computing device 230 is a server on the cloud. Therefore, the user can simply and conveniently perform an ordinary and simple operation of uploading an image captured by the smartphone to the cloud, so that the optimal installation position of the agent device 210 is presented, which is convenient and convenient.

Further, the companion device 220 performs image processing such as blurring or mosaicing information that is not desired to be disclosed, or performs encryption processing to replace the information with other information when transmitting the captured image of the room to the computing device 230. Because of this, it is possible to enjoy the service of recommending the location of the agent device 210 while protecting the privacy of the user.

Further, according to the storage location recommendation system 200 according to the present embodiment, the storage location of the agent device 210 is recommended in consideration of the movement of the furniture in the room. And recommending the location of the agent device 210.

The technique disclosed in the present specification has been described above in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the gist of the technology disclosed in this specification.

The technology disclosed in this specification can be applied not only to a voice agent, but also to installing various devices in which an agent application such as an information home appliance or an IoT device resides.

In short, the technology disclosed in this specification has been described by way of example, and the contents described in this specification should not be interpreted restrictively. In order to determine the gist of the technology disclosed in this specification, the claims should be considered.

The technology disclosed in the present specification may have the following configurations.
(1) a receiving unit that receives first information including information about an environment in which an agent device is installed;
A processing unit configured to generate, based on the first information, second information that suggests an installation position of the agent device in the environment;
A transmitting unit that returns the second information;
An information processing apparatus comprising:
(1-1) a receiving step of receiving first information including information on an environment in which an agent device is installed;
A processing step of generating second information that proposes an installation position of the agent device in the environment based on the first information;
A transmitting step of returning the second information;
An information processing method comprising:
(2) the first information includes a captured image of a room where the agent device is installed,
The second information includes an image in which the agent device is superimposed on a proposed installation position in the captured image,
The information processing device according to (1).
(3) The processing unit proposes an installation position of the agent device based on a result of object recognition of the captured image and a function or specification of the agent device.
The information processing device according to (2).
(4) The processing unit compares the result of object recognition of the captured image with a place where the agent device is suitable and a device or place to be avoided, and proposes an installation position of the agent device.
The information processing device according to any one of (2) and (3).
(4-1) The processing unit proposes, for the agent device equipped with a speaker playback function, an installation position that avoids a place where a television receiver or another speaker is located,
The information processing device according to (4).
(4-2) The processing unit proposes an installation position at the center of the room to the agent device equipped with a omnidirectional or wide-angle camera photographing function,
The information processing device according to (4).
(4-3) The processing unit proposes an installation position near a white flat wall with respect to the agent device equipped with a projector display function,
The information processing device according to (4).
(5) when there are a plurality of candidate positions for installing the agent device, the processing unit generates the second information in which a priority is assigned to each candidate position;
The information processing device according to any one of (1) to (4).
(6) The processing unit proposes an installation position of the agent device in consideration of movement of furniture in a room where the agent device is installed.
The information processing device according to any one of (1) to (5).
(7) The processing unit proposes an installation position of the agent device in consideration of movement of furniture in the room based on a movement cost for each furniture.
The information processing device according to (6).
(8) The processing unit generates the second information including a composite image of a room where the agent device is installed by moving furniture.
The information processing device according to (7).
(9) The processing unit proposes an installation position of the agent device in consideration of an installation cost based on a distance from an outlet,
The information processing device according to any one of (1) to (9).
(10) a transmitting unit that transmits first information including information on an environment in which the agent device is installed;
A receiving unit that receives second information regarding the installation position of the agent device proposed based on the first information;
A presentation unit for presenting the second information;
An information processing apparatus comprising:
(11) further including a photographing unit,
The transmission unit transmits the first information including an image of the room in which the agent device is installed by the imaging unit,
The receiving unit receives the second information including an image in which the agent device is superimposed on a suggested installation position in the captured image,
The presentation unit displays the image in which the agent device is superimposed on a proposed installation position,
The information processing device according to (10).
(12) The second information includes a plurality of candidate positions for installing the agent device, and a priority assigned to each candidate position,
The presentation unit presents each priority of the plurality of candidate positions,
The information processing device according to any one of (10) and (11).
(13) an information encryption unit for encrypting predetermined information included in the first information;
An information decryption unit that recovers based on the encrypted predetermined information included in the second information;
Further comprising,
The information processing apparatus according to any one of (10) to (12).
(14) the information encryption unit performs predetermined image processing on a human face area in an image included in the first information;
The information processing device according to (13).
(15) The information encryption unit replaces an image included in the first information with a generalized or generalized image.
The information processing device according to (13).
(16) The information encryption unit replaces an image included in the first information with an illustration drawn by a user.
The information processing device according to (13).
(17) The information encryption unit replaces an image included in the first information with position information of each object reflected in the image.
The information processing device according to (13).
(18) further comprising a sensor unit,
The transmitting unit transmits the first information in response to the sensor unit detecting a change in the environment in which an agent device is installed,
The information processing apparatus according to any one of (10) to (17).
(19) a transmitting step of transmitting first information including information on an environment in which the agent device is installed;
A receiving step of receiving second information on an installation position of the agent device proposed based on the first information;
A presentation step of presenting the second information;
An information processing method comprising:
(20) transmitting first information including information on an environment in which an agent device is to be installed, and receiving second information about an installation position of the agent device proposed based on the first information, A first device for presenting second information;
A second device that receives the first information from the first device and returns the second information;
An information processing system comprising:

200 ... Place recommendation system 210 ... Agent device, 211 ... Control unit, 212 ... Exposed unit 213 ... Sensor unit, 214 ... Communication unit 220 ... Companion device, 221 ... Control unit, 222 ... Exposed unit 223 ... Shooting unit, 224: Communication unit 225: Information encryption unit, 226: Information decryption unit 230: Arithmetic device, 231: Control unit, 232 ... Communication unit 233: Agent device table, 234: Furniture information table 2801: 6-axis sensor, 2802: Distance sensor , 2803: human sensor 2811: self-orientation detection unit, 2812: wall distance estimation unit 2813: installation position evaluation unit

Claims

A receiving unit that receives first information including information on an environment in which the agent device is installed;
A processing unit configured to generate, based on the first information, second information that suggests an installation position of the agent device in the environment;
A transmitting unit that returns the second information;
An information processing apparatus comprising:
The first information includes a captured image of a room where the agent device is installed,
The second information includes an image in which the agent device is superimposed on a proposed installation position in the captured image,
The information processing device according to claim 1.
The processing unit, based on the result of object recognition of the captured image and the function or specification of the agent device, proposes an installation position of the agent device,
The information processing device according to claim 2.
The processing unit compares the result of object recognition of the captured image with a place where the agent device is suitable and a device or place to be avoided, and proposes an installation position of the agent device.
The information processing device according to claim 2.
When there are a plurality of candidate positions for installing the agent device, the processing unit generates the second information in which a priority is assigned to each candidate position.
The information processing device according to claim 1.
The processing unit, considering the movement of furniture in the room where the agent device is installed, proposes an installation position of the agent device,
The information processing device according to claim 1.
The processing unit also considers the movement of furniture in a room based on the movement cost of each piece of furniture, and proposes an installation position of the agent device.
The information processing device according to claim 6.
The processing unit generates the second information including a composite image of a room where the agent device is installed by moving furniture.
The information processing device according to claim 7.
The processing unit also considers an installation cost based on a distance from an outlet, and proposes an installation position of the agent device.
The information processing device according to claim 1.
A transmitting unit that transmits first information including information on an environment in which the agent device is installed;
A receiving unit that receives second information regarding the installation position of the agent device proposed based on the first information;
A presentation unit for presenting the second information;
An information processing apparatus comprising:
It further includes a shooting unit,
The transmission unit transmits the first information including an image of the room in which the agent device is installed by the imaging unit,
The receiving unit receives the second information including an image in which the agent device is superimposed on a suggested installation position in the captured image,
The presentation unit displays the image in which the agent device is superimposed on a proposed installation position,
The information processing device according to claim 10.
The second information includes a plurality of candidate positions for installing the agent device, and a priority assigned to each candidate position,
The presentation unit presents each priority of the plurality of candidate positions,
The information processing device according to claim 10.
An information encryption unit that encrypts predetermined information included in the first information;
An information decryption unit that recovers based on the encrypted predetermined information included in the second information;
Further comprising,
The information processing device according to claim 10.
The information encryption unit performs predetermined image processing on a human face region in an image included in the first information;
The information processing device according to claim 13.
The information encryption unit replaces an image included in the first information with a generalized or generalized image,
The information processing device according to claim 13.
The information encryption unit replaces an image included in the first information with an illustration drawn by a user.
The information processing device according to claim 13.
The information encryption unit replaces an image included in the first information with position information of each object reflected in the image,
The information processing device according to claim 13.
Further comprising a sensor unit,
The transmitting unit transmits the first information in response to the sensor unit detecting a change in the environment in which an agent device is installed,
The information processing device according to claim 10.
Transmitting first information including information on an environment in which the agent device is installed;
A receiving step of receiving second information on an installation position of the agent device proposed based on the first information;
A presentation step of presenting the second information;
An information processing method comprising:
Transmitting first information including information on an environment in which the agent device is installed, and receiving second information on an installation position of the agent device proposed based on the first information, A first device for presenting information;
A second device that receives the first information from the first device and returns the second information;
An information processing system comprising: