JP2022155665A - In-home display system - Google Patents

Abstract

To encourage users to become attached to the system that provides advice and suggestions, and to provide an in-home display system using augmented reality for improving the usage frequency of the system.SOLUTION: An in-home display system is provided with a user information acquisition unit 3 for acquiring user information relating to the user's state and the user's position within a house 1, display devices 4 and 5 used by a user to display a virtual reality character superimposed on the interior scenery of the house 1 in the real world, and performance control means communicably connected to each of the user information acquisition unit and the display device and controlling rendering of the virtual reality character displayed on the display device (a display control unit 6, a first server 7 and a second server 8). The performance control means controls performances including output of message information to be transmitted to the user generated based on the user information and animation display of the virtual reality character corresponding to the message information.SELECTED DRAWING: Figure 2

Description

The present invention relates to an indoor display system using augmented reality.

In the past, not only for the purpose of energy conservation, but also for analyzing the behavior patterns of residents using sensors installed in houses and sensors installed in housing equipment, and taking into account the information of conversations with users. A technique related to a housing agent server has been proposed that supports user actions by means of a home (see Patent Literature 1).
According to such technology, it is possible to assist the user's actions without impairing the living comfort of the house.

JP 2015-156125 A

As a part of supporting the behavior of the user, it is required to give advice and suggestions to the user so that the user can act accordingly.
However, since the advice and suggestions from the system are strictly based on computer programs, there are cases where the user's actions cannot be well supported even if the advice or suggestions are followed. At such times, the user attempts to evaluate the performance of the system as a machine, such as, for example, "this machine has low accuracy in advice and suggestions." As a result, users do not develop attachment to the system, leading to a decrease in the frequency of use, and as a result, the development of technology in that field may be delayed.

The present invention has been made in view of the above circumstances, and an object of the present invention is to encourage users to become attached to a system that provides advice and suggestions, and to increase the frequency of use of the system.

The invention according to claim 1 includes, for example, as shown in FIGS.
display devices 4 and 5 used by the user U to display a virtual reality character C superimposed on the interior scenery S1 and S2 of the house 1 in the real world;
Effect control means (display A control unit 6, a first server 7, a second server 8),
The effect control means controls effects including output of message information generated based on the user information to be transmitted to the user U, and animation display of the virtual reality character C corresponding to the message information. characterized by performing

According to the first aspect of the invention, the effect control means controls the effect including the output of message information to be transmitted to the user U generated based on the user information. contains information about the user U himself, making it easier for the user U to be interested in the message information. In addition, since the effect control means controls effects including animation display of the virtual reality character C corresponding to such message information, it is possible to make the user U more interested in the virtual reality character C. The user U is more likely to feel attached to the virtual reality character C. As a result, it is possible to encourage the user U to develop an attachment to the system that outputs message information such as advice and proposals, and to improve the frequency of use of the system.

The invention according to claim 2 is, for example, as shown in FIGS. 1 to 8, in the in-home display system according to claim 1,
It further comprises an in-home information acquisition unit 2 for acquiring in-house information related to devices, facilities, etc. in the house 1,
The effect control means is communicably connected to the in-house information acquisition unit 2, and outputs the message information generated based on the in-house information and the user information, and the virtual reality corresponding to the message information. It is characterized by performing control of production including animation display of character C.

According to the second aspect of the invention, the effect control means controls the effect including the output of the message information generated based on the in-house information and the user information. Information about the house 1 in which the user resides and information about the user U himself are included, and it is possible to make the user U more interested in the message information. In addition, since the effect control means controls effects including animation display of the virtual reality character C corresponding to such message information, it is possible to make the user U more interested in the virtual reality character C. The user U is more likely to feel attached to the virtual reality character C. As a result, it is possible to encourage the user U to develop an attachment to the system that outputs message information such as advice and proposals, and to further improve the frequency of use of the system.

The invention according to claim 3 is, for example, as shown in FIGS. 5 and 8, in the in-home display system according to claim 1 or 2,
Indoor coordinate information 10 is defined in the indoor space of the house 1,
The effect control means is characterized by performing control to display the virtual reality character C at a position associated with the home coordinate information 10 in the virtually constructed virtual world.

According to the third aspect of the invention, the effect control means performs control to display the virtual reality character C at the position associated with the indoor coordinate information 10 in the virtually constructed virtual world. The virtual reality character C can be displayed at any position associated with the indoor coordinate information 10 . As a result, since the user U can enjoy the performance of the virtual reality character C at various places in the house 1, the user U is likely to feel attached to the virtual reality character C, and as a result, the frequency of use of the system is reduced. can contribute to improvement.

The invention according to claim 4 is, for example, as shown in FIG. 6, in the in-home display system according to any one of claims 1 to 3,
The effect control means further includes an analysis unit 7d for estimating a degree of intimacy between the user U and the virtual reality character C based on a conversation between the user U and the virtual reality character C. do.

According to the fourth aspect of the present invention, the effect control means includes the analysis unit 7d for estimating the degree of intimacy between the user U and the virtual reality character C based on the conversation between the user U and the virtual reality character C. Furthermore, since it is prepared, the favorable impression of the virtual reality character C to the user U is increased or decreased according to the level of the estimated intimacy, and as the favorable impression increases, the attitude of the virtual reality character C changes from stranger to intimate. change to As a result, since it is possible to perform effects such as making remarks and taking actions that consider the user U according to the favorability, the user U is more likely to feel attached to the virtual reality character C, and as a result, the system can contribute to increasing the frequency of use of

The invention according to claim 5 is, for example, as shown in FIG. 5, in the in-home display system according to any one of claims 1 to 4,
It is characterized by further comprising an in-home device operation means 22 that cooperates with the effect control means and operates an operation target device provided in the house 1 .

According to the fifth aspect of the invention, since the in-home device operation means 22 that cooperates with the effect control means and operates the operation target device provided in the house 1 is further provided, the in-home device that cooperates with the effect control means is further provided. The device operation means 22 allows the user U and the virtual reality character C to interact with each other and to operate the device to be operated. As a result, the user U can not only enjoy the performance of the virtual reality character C, but also enjoy the pleasure of controlling the facilities of the house 1 through the virtual reality character C. Moreover, the user U did not directly operate the device to be operated, and through conversation with the user U, the user U said, "(Apparently) the virtual reality character C operated the device to be operated for me." can be felt, the attachment to the virtual reality character C is likely to develop, and as a result, the system can be used more frequently.

According to the present invention, it is possible to encourage a user to develop a fondness for a system that provides advice and suggestions, thereby increasing the frequency of use of the system.

1 is a diagram showing an overview of an in-home display system; FIG. 1 is a diagram showing the configuration of an in-home display system; FIG. FIG. 4 is a diagram showing a display example of a display device; FIG. 4 is a diagram showing a display example of a display device; It is a figure which shows the structure of a display control part. It is a figure which shows the structure of a 1st server. It is a figure which shows the structure of a 2nd server. 1 is a conceptual diagram of a virtual world corresponding to the real world; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below have various technically preferable limitations for carrying out the present invention, but the technical scope of the present invention is not limited to the following embodiments and illustrated examples. do not have.

Reference numeral 1 in FIG. 1 indicates a house in which user U resides. Although this house 1 is a detached house, it may be a house in a condominium, an apartment, or the like, and is not particularly limited. A plurality of rooms are provided in a house 1. - 特許庁Also, the number of users U living in the house 1 may be one or more.
In such a house 1, display devices 4 and 5 for displaying the virtual reality character C superimposed on the real-world interior scenes S1 and S2 are used to display the virtual reality character C to the user U. An in-home display system has been introduced that allows the user to recognize the user, receive advice and suggestions through the virtual reality character C, and have conversations with the user.

As shown in FIGS. 2 and 3, the in-home display system includes an in-home information acquisition unit 2, a user information acquisition unit 3, display devices 4 and 5 (display device 4 and display device 5), and display control. A unit 6 , a first server 7 and a second server 8 are provided. In this embodiment, the display control unit 6, the first server 7, and the second server 8 constitute production control means.
It should be noted that the term "inside the house" or "inside the house" in the present embodiment does not refer only to the house itself where the user U resides, but may or may not include the site on which the house is built. also includes

[About the home information acquisition part]
First, the in-home information acquiring unit 2 is a means for acquiring in-house information related to devices, facilities, etc. in the house 1, and is connected to the display control unit 6 so as to be communicable. The in-house information acquiring unit 2 is, for example, various sensors arranged in various places in the house 1 or systems, apparatuses, devices, etc. using various sensors.
The home information acquired from the home information acquisition unit 2 affects the remarks (advice, proposals, etc.) and performances of the virtual reality character C to the user U. Based on the information, it is possible to control the speech and presentation of the virtual reality character C displayed on the display devices 4 and 5. FIG.

More specifically, the indoor information acquisition unit 2 includes a power sensor, water flow meter, gas flow meter, temperature/humidity sensor, opening/closing sensor, inventory management system, tilt sensor, carbon dioxide sensor, and the like. In the in-home display system installed in the house 1, at least a power sensor and a temperature/humidity sensor among the various sensors described above are employed as the in-home information acquisition unit 2.
It is not limited to this, and can be changed as appropriate without departing from the gist of the present invention. Also in the following description of the present embodiment, it is assumed that various sensors and the like are appropriately selected as the optimum in-home information acquisition unit 2 .

The power sensor is connected to, for example, a distribution board, a private power generation facility such as a photovoltaic power generation system, a storage battery facility, each outlet in the house 1, and the like, and can measure the amount of power used in the house 1. .
By appropriately arranging and connecting such power sensors in the house 1, not only the power consumption of the entire house 1, but also the power consumption of each room depending on the configuration of the distribution board and the number of power sensors used. It is possible to measure the amount of use and the amount of electricity used by each home appliance. By measuring the amount of power consumption, it is possible to issue a warning of excessive power consumption, to perceive that no power is being used as an anomaly, and to make use of life information provided to the user U. In addition, it can be used for behavior estimation and behavior prediction of the user U, and can also be used for abnormality detection of various home electric appliances used in the house 1, other sensors, devices, and the like.

The water flowmeter is connected to, for example, a water meter, and can measure the amount of water used in the house 1 . By measuring the amount of water used in the house 1, it is possible to warn of overuse of water, to perceive that no water is being used as an anomaly, and to make use of life information provided to the user U. can be done. It can also be used for behavior estimation and behavior prediction of the user U, and abnormality detection in various equipment such as hot water supply equipment.

The gas flowmeter is connected to, for example, a gas meter, and can measure the amount of gas used in the house 1 . It is possible to warn of overuse of gas, detect that gas is not being used at all as an anomaly, and provide the user U with living information. It can also be used for behavior estimation and behavior prediction of the user U, and abnormality detection in various equipment such as hot water supply equipment.

The temperature/humidity sensor is provided in each room in the house 1 and outdoors, and can measure the ambient temperature and humidity where the temperature/humidity sensor is provided. By measuring the temperature and humidity, it is possible to confirm whether the thermal environment in the house 1 is normal or not, and to make use of the life information provided to the user U. In addition, the user U's preferred thermal environment and the like can be stored and learned.

The opening opening/closing sensor can detect the opening/closing of fittings provided at a window or doorway. By detecting the opening and closing of the fittings, it is possible to confirm the entrance and exit of the user U to the house and rooms, and to make use of the security of the house 1 . In addition, it is possible to give a warning about forgetting to close fittings provided at a window or an entrance, or to estimate the movement trajectory of the user U inside the house.
Advice and proposals given to the user U by the virtual reality character C can be appropriately changed based on the weather information and the opening/closing detection information of fittings.

The inventory management system is a system introduced into the house 1 for inventory management of articles stored in a predetermined place (refrigerator, pantry, and other storage space) in the house 1 . It is designed to detect an increase or decrease in the number of items based on the images captured by the camera, the measurement results of the weight sensor, and the stored inventory data.

The tilt sensor, also called an acceleration sensor, is provided in a structural portion of the house 1 so as to measure vibrations occurring inside the house 1 . This tilt sensor constitutes a seismometer installed in the house 1, and by linking the seismometer and the in-house display system, the measurement results measured by the seismometer can be confirmed by the display devices 4 and 5. becomes possible.
For example, the seismometer detects vibrations generated in the house 1 and pre-stored house information related to the structural members and structure of the house 1. When a horizontal force acts on the house 1 due to an earthquake, It is possible to calculate the inter-story displacement angle that occurs in the

The carbon dioxide sensor is for measuring the carbon dioxide concentration inside the house 1, and can detect carbon dioxide when the carbon dioxide concentration inside the house 1 becomes high. When high-concentration carbon dioxide is detected, the advice or proposal given to the user U by the virtual reality character C can be appropriately changed based on the detection information.

[About the user information acquisition part]
Subsequently, the user information acquisition unit 3 is means for acquiring user information related to the state of the user U, the position in the house 1, and the like, and is connected to the display control unit 6 so as to be communicable. This user information acquisition unit 3 is, for example, various devices or various sensors arranged in various places in the house 1 .
More specifically, the user information acquisition unit 3 includes a camera, a microphone, a beacon, a piezoelectric sensor, a radio range sensor, a wearable terminal, and the like. In the in-home display system introduced into the house 1, at least a camera among the various devices and various sensors described above is employed as the user information acquisition unit 3, but the present invention is not limited to this, and is not limited to this. It can be changed appropriately as long as it does not occur. Also in the following description of the present embodiment, it is assumed that various devices and various sensors are appropriately selected as the optimum user information acquisition unit 3 .

A plurality of cameras are installed at various locations in the house 1, and of course photograph the user U, but also photograph various places in the house 1, and photograph articles in the refrigerator and storage space. be able to.
The camera may be provided with a face authentication function for identifying the face of the user U. In that case, the camera may be installed at various locations in the house 1 so that it can track where the user U has moved. It's becoming That is, the positional information of the user U within the house 1 can be obtained.
Furthermore, the camera may be provided with a thermal image sensor (thermography) function. It can detect changes in body temperature based on undulations and use it in conversation.
Also, the state of the user U can be acquired from the image captured by the camera. The state of the user U refers to the user's behavior and daily actions such as standing, sitting, lying down, and walking.
However, it is desirable not to install the camera in a place where privacy protection is required. Therefore, in places where privacy protection is required, information on the user U is acquired based on information from sensors, devices, etc. other than the camera.

A plurality of microphones are installed at various locations in the house 1, and can pick up the voice of the user U and other sounds. The location information of the user U can be acquired by listening to the voice of the user U with a microphone. Furthermore, by using a microphone, it is possible to enable the user U and the virtual reality character C to have a conversation, and to hear the sounds of daily life and abnormal sounds different from the sounds of daily life. In addition, the voice of the user U picked up by the microphone and other voice information are transmitted to the first server 7 via the display control unit 6 .
Such a microphone functions as the user information acquisition unit 3 to acquire the state in which the user U is speaking, and also functions as the in-home information acquisition unit 2 to acquire other surrounding sounds. Further, since the microphone is used for conversation between the user U and the virtual reality character C as described above, a speaker for outputting the voice of the virtual reality character C may be provided near the microphone. That is, the user information acquisition unit 3 may include a speaker as well as a microphone.

A beacon is a signal generator that emits a beacon signal, and is installed in a plurality of locations in the house 1. The beacon signal includes position information of the beacon signal generator.
The beacon signal transmitted from the beacon signal generator is received by the display devices 4 and 5, and the display devices 4 and 5 receive the position information included in the beacon signal and the identification information for identifying the display devices 4 and 5. is set to be transmitted to the display control unit 6. The display control unit 6 can thereby obtain information on the current positions of the display devices 4 and 5 .

The piezoelectric sensor generates strain by pressure to generate voltage, and is built into the floor of the house 1 . If such a piezoelectric sensor is built into the floor, user U's position information can be easily acquired. Further, if the detection range is a narrow area, it can be determined that the user U is standing or walking, and if the detection range is large, it can be determined that the user U is sitting or lying down. .

The radio ranging sensor is a sensor for acquiring position information of the user U by transmitting radio waves into a room and receiving radio waves reflected by the user U in the room. As radio waves to be transmitted, millimeter waves and microwaves are used together, and in particular, it is preferable to be provided in a highly private room such as a bathroom, a dressing room, a toilet, a bedroom, and the like.
Some radio sensors can measure a person's heartbeat and respiration rate under specific conditions (close to the radio sensor, subject not moving, etc.). A radio sensor for acquiring information may also be used.

The wearable terminal is worn on the body of the user U, and can transmit the position information of the user U to the display control unit 6, or acquire information related to health management of the user U and transmit it to the display control unit 6. can. In the present embodiment, in particular, a device capable of ascertaining the health condition of the user U, such as the user's U blood pressure, heartbeat, and body temperature, is employed.

[About the display device]
3, 4 and 8, the display devices 4 and 5 are means for displaying the virtual reality character C superimposed on the real world indoor scenery S1 and S2. It is communicably connected to the control unit 6 . The display devices 4 and 5 can receive the control signal from the display control section 6, operate based on the control signal, and display the virtual reality character C on the respective display sections 4a and 5a.
Further, the display devices 4 and 5 receive a beacon signal from a beacon, and transmit the position information included in the beacon signal and the identification information for identifying the display devices 4 and 5 to the display control unit 6. have a function.
The display devices 4 and 5 are assumed to have a calculation unit, a storage unit, a communication unit, etc. necessary for exhibiting these functions, and the storage unit stores various data necessary for displaying the virtual reality character C. and various programs (applications) are stored.

Examples of such display devices 4 and 5 include a smartphone 4, AR glasses 5, a hologram device, a projector, and the like.
In the in-home display system installed in the house 1, at least the smartphone 4 and/or the AR glasses 5 among the various devices described above are preferably employed as the display devices 4 and 5, but the present invention is not limited to this. , can be changed as appropriate without departing from the gist of the present invention. Also in the following description of the present embodiment, it is assumed that the optimum display device is appropriately selected.

The smartphone 4 is one of mobile information terminals, and a mobile information terminal such as a tablet terminal may be used instead of the smartphone 4 . As such a smartphone 4, as shown in FIG. 3, in addition to functions as a portable information terminal such as a calculation unit, a storage unit, a communication unit, etc., at least a camera (out camera) not shown and a display unit 4a, a speaker 4b, a microphone (not shown), and an in-camera 4c.
The storage unit stores various data and various programs necessary to display the virtual reality character C on the display unit 4a. Performs various processing including processing to display on

The display unit 4a is a touch panel that allows the user to operate the smartphone 4 by touching it with a finger or the like. Along with displaying, the virtual reality character C is also displayed.

The speaker 4b emits sounds corresponding to the image of the virtual reality character C (speech of the virtual reality character C, sound effects, music, etc.). In addition, it can substitute by connecting an earphone.

The microphone can pick up the user U's voice and other surrounding sounds. The microphones in the smartphone 4 are used for the same purpose as the above-described multiple microphones provided in various places in the house 1 . That is, by using a microphone, it is possible to allow the user U and the virtual reality character C to have a conversation, or to hear the sounds of daily life or abnormal sounds different from the sounds of daily life. Therefore, the microphone in this smartphone 4 can acquire the state in which the user U speaks as the user information acquisition unit 3 .
Also, instead of using a microphone, the intention of the user U may be conveyed by text input or button operation. In this case, the utterances of the virtual reality character C are also output as text on the display unit 4a. That is, the conversation between the user U and the virtual reality character C may be conducted by written communication.

The smartphone 4 is used by the user U holding it in his or her hand, for example. Then, by activating a dedicated application that uses the camera function incorporated in the smartphone 4, the real-world indoor scenery S1 displayed on the display unit 4a of the smartphone 4 is displayed according to the control signal received from the display control unit 6. The image of the virtual reality character C based on the image can be superimposed and displayed.

If the display device 4 is the smart phone 4, user registration can be performed by entering a user ID through a dedicated application. That is, when there are a plurality of users U in the house 1 (for example, when they live together with their family members), the users U can be identified.

The AR glasses 5 are a head-up display or a head-mounted display, and two transparent display units 5a are arranged in front of the line of sight of the user U as shown in FIG. Further, it has a calculation unit, a storage unit, a communication unit, etc. necessary for exhibiting the same functions as when the dedicated application is activated in the case of the smartphone 4 described above. As a result, the information (virtual reality character C) displayed on the display unit 5a is displayed overlapping the user's U normal field of view. That is, the image of the virtual reality character C based on the control signal received from the display control unit 6 can be superimposed and displayed on the indoor scenery S2 in the real world.
The storage unit stores various data and various programs necessary for displaying the virtual reality character C on the display unit 5a. Performs various processing including processing to display on

Since the AR glasses 5 in this embodiment are eyeglass-shaped, they include a rim 5b that holds the display unit 5a, a bridge 5c between the two rims 5b, a pad 5d that contacts the nose, and an endpiece 5e that protrudes from the rim 5b. and a foldable temple 5f rotatably connected to the end piece 5e.
Also, although not shown, the temple 5f arranged near the ear of the user U is provided with a bone conduction speaker (a normal speaker may be used), and a sound corresponding to the image of the virtual reality character C (virtual You can listen to the utterances of the real character C, sound effects, music, etc.).
Furthermore, although not shown, a microphone for collecting the voice of the user U may be provided. That is, the microphone in the AR glasses 5 can acquire the state in which the user U speaks as the user information acquisition unit 3 . If no microphones are provided, a conversation between the user U and the virtual reality character C is established using the plurality of microphones provided at various locations in the house 1 .
A goggle type or a helmet type may be used instead of the spectacle type.

Such AR glasses 5 have a head tracking function and a positional tracking function. The head tracking function refers to a function that changes the image according to the direction of viewing, such as the movement of the head. The positional tracking function refers to a function that changes the image according to the location by spatial recognition. With these tracking functions, it is easy to create a sense of immersion and unity, and high-quality augmented reality is possible.

Note that user registration can also be performed with the AR glasses 5 , but considering that a plurality of users U use the AR glasses 5 , it is not necessary to specify the user U with the AR glasses 5 .

A hologram device, although not shown, refers to a device that projects a three-dimensional image (hereinafter referred to as a 3D image) using 3D hologram technology. In other words, 3D images can be projected onto an empty space, making it appear as if the real thing is right in front of you. In general 3D images using non-3D holograms, the images can only be seen in three dimensions from the front, but with 3D images using 3D hologram technology, you can see the real thing from the sides and back without the need for 3D glasses. It is possible to see stereoscopic images.
The hologram device according to this embodiment includes an irradiation unit that emits light for creating a 3D image, a speaker, and a microphone. By using such a hologram device, it is possible to display the virtual reality character C in space without having to hold or wear a device like the smartphone 4 or the AR glasses 5 . That is, the image of the virtual reality character C based on the control signal received from the display control unit 6 can be superimposed and displayed on the real world scenery of the house (the scenery of the room in which the hologram device is located).
The hologram device may employ similar technologies such as pepper's ghost, volume display, and fan type display.

The projector is a device that projects an image onto the wall surface of a room provided in the house 1 (not only the wall surface, but also the ceiling, the floor, the surface of the furniture, etc.), and is installed at a high position in the room. The projector in this embodiment is configured integrally with a ceiling light and installed on the ceiling.
The projector in this embodiment has a projection unit that projects an image, a speaker, and a microphone. The projector can perform an effect of moving the virtual reality character C over a wide range in the room by providing a mechanism that rotates the entire device or a projection unit with a swinging function.
By using such a projector, it is possible to display the virtual reality character C on the wall surface of a room or the like without having to hold or wear a device like the smartphone 4 or the AR glasses 5 . That is, the image of the virtual reality character C based on the control signal received from the display control unit 6 can be superimposed and displayed on the real world scenery of the house (the scenery of the room with the projector).

[About the display control part]
Subsequently, the display control unit 6 transmits a control signal for displaying the virtual reality character C to the display devices 4 and 5 to operate the display devices 4 and 5. This is for displaying the virtual reality character C on the display units 4a and 5a.
As shown in FIG. 5, the display control unit 6 includes a calculation unit 6a, a storage unit 6b, communication units 6c to 6f, and have With such a configuration, the display control unit 6 collects data related to various sensors and various devices in the house 1, analyzes the collected data, judges and judges, and the result is Based on this, the display devices 4 and 5 can be controlled.

The display control unit 6 in this embodiment is configured by, for example, a home gateway. As communication units 6c to 6f, the home gateway has a router function that creates connection settings to the provider, etc., a wireless LAN function that enables Wi-Fi connection, and a voice call and fax transmission/reception by connecting telephones and faxes. It refers to a device that has both the function of relaying between a VoIP gateway telephone network and an IP network and the function of a line terminating device for an optical line.
Although the display control unit 6 in this embodiment is configured by, for example, a home gateway, it is not limited to this, and may be configured by other computer devices such as personal computers, smartphones, tablet terminals, etc. good.

As described above, the display control unit 6 in the present embodiment can provide the home gateway 4 with necessary functions to realize an in-home display system. Therefore, the in-home display system in this embodiment may be constructed without cooperation with an external device such as the first server 7 or the second server 8, which will be described later. That is, all the functions of the external devices such as the first server 7 and the second server 8 to be described later may be integrated into the display control unit 6 .
Also, the degree of cooperation with an external device such as the first server 7 or the second server 8, which will be described later, may be changed as appropriate. Therefore, it is assumed that the functions provided by the external device may be applied to the display control unit 6 . On the contrary, the display control unit 6 in this embodiment is simply used as a repeater having functions such as a router and a hub, and in cooperation with external devices such as the first server 7 and the second server 8, the in-home display in this embodiment A display system may be constructed.
Note that setup, setting changes, and maintenance of the in-home display system can be performed through the smartphone 4, a personal computer, or the like.

Of the display control unit 6 as described above, the calculation unit 6a is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and integrates the processing operations of each unit in the in-home display system. control effectively. Specifically, the CPU of the calculation unit 6a reads various processing programs stored in the ROM, develops them in the RAM, and performs various processing in cooperation with the programs.

The storage unit 6b is composed of a HDD (Hard Disk Drive), a non-volatile memory, etc., and stores various data and various programs necessary for realizing the in-home display system. Specifically, for example, it stores in-home coordinate information 10 preset in the house 1, a display control program 20 for displaying the virtual reality character C on the display devices 4 and 5 and operating them appropriately, and the like. there is The storage unit 6b also stores an in-home device operation program 21 for operating an operation target device linked to the in-home display system.
The in-house information acquired from the in-house information acquisition unit 2, the user information acquired from the user information acquisition unit 3, and the position information of the display devices 4 and 5 are stored in the memory of the calculation unit 6a. You may make it memorize|store in the memory|storage part 6b for database construction. Each piece of information stored in the storage unit 6b is transmitted to the first server 7, which will be described later, and used for data collection and database construction.

The communication units 6c to 6f are composed of network interfaces and the like, and transmit and receive data to and from other devices connected via the communication network N. FIG. The other devices include the home information acquisition unit 2, the user information acquisition unit 3, the display devices 4 and 5, the first server 7, the second server 8, as well as a personal computer used in the house 1. Information terminals, communication equipment such as telephones, various household appliances, other sensors, devices, etc. are included.
The communication units 6c to 6f in the present embodiment include indoor communication units 6c to 6e and an external communication unit 6f.

The indoor communication units 6c to 6e include a wireless communication unit 6c, a wired communication unit 6d, and a telephone communication unit 6e.
The wireless communication unit 6 c is a wireless communication interface that enables wireless communication with other devices that wirelessly access the display control unit 6 .
The wired communication unit 6d is a wireless communication interface that enables wireless communication with other devices that access the display control unit 6 by wire.
The telephone communication unit 6e is a voice call connection interface that enables proper connection with a telephone (not shown).
The in-house communication unit may also include a communication interface that enables connection with a USB device. An example of the USB device is a USB memory, and various data and various programs necessary for realizing the in-home display system may be stored in the USB memory.
In addition, a communication network by LAN (Local Area Network) is constructed in the house 1, and this in-home communication network includes various networks such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), and NFC (Near Field Communication). It may be a collective communication network in which the communication networks are communicatively connected to each other.

The out-of-home communication unit 6f is a communication interface for connection with a communication network N constructed between external devices such as a first server 7 and a second server 8, which will be described later.
Communication network N includes various communication line networks such as telephone lines, ISDN lines, optical fibers, mobile communication networks, communication satellite lines, CATV lines, and other dedicated lines, and Internet service providers ( That is, the Internet) and the like may be included. Alternatively, it may be a collective communication network in which various communication networks such as LAN, WAN (Wide Area Network), WiFi, Bluetooth, and NFC are communicably connected to each other. The form of connection may be wired, wireless, or a mixture of wired and wireless.

Such a display control unit 6 executes the display control program 20 while communicating with other devices through the communication units 6c to 6f to display the display units 4a and 5a of the display devices 4 and 5. A virtual reality character C can be displayed. The display control program 20 is one of the main constituent elements in the effect control means, and displays the virtual reality character C on the display units 4a and 5a of the display devices 4 and 5, and controls the virtual reality character C based on the user information. It is a program that controls the speech and actions of C.
At this time, the location information (in-house coordinates 12) of the user U using the display devices 4 and 5 is collated with the in-house coordinate information 10, and the virtual reality character C corresponding to the location in the house 1 is produced. can be done.
Further, the display control program 20 can control not only the virtual reality character C but also the display of other information. In other words, it is possible to control all the information displayed on the display units 4a and 5a of the display devices 4 and 5 so as to be superimposed on the indoor scenery S1 and S2 of the house 1 in the real world.

Further, the display control unit 6 can operate the operation target device by executing the in-home device operation program 21 constituting the in-house device operation means. This home device operation program 21 may be executable only when the user U is using the display devices 4 and 5 .
It should be noted that the device to be operated indicates, for example, various smart home appliances, IoT devices, and various types of housing equipment. Examples of various smart home appliances and IoT devices include air conditioners, washing machines, health meters, televisions, robot vacuum cleaners, smart LED lighting, security cameras, and the like, all of which can communicate with the display control unit 6. connected and working together.
Various types of housing equipment include windows, doors, shutter devices, water heaters, storage batteries, and the like, all of which are communicably connected to and cooperate with the display control unit 6 .
The in-home device operation program 21 is basically executed through conversation between the user U and the virtual reality character C. FIG. For example, it may be executed by an instruction or approval from the user U (speech or operation action in augmented reality), or by advice or suggestion from the virtual reality character C. As a result, the user U can feel that "the virtual reality character C has operated the operation target device for me," although this is apparently the case.
For example, in an emergency such as when the user U does not move from a fixed place for a fixed time or longer, or when the wearable terminal determines that the health condition is suspected to deteriorate, the in-home device operation program can be used even if the user U and the virtual reality character C do not talk. 21 alone, the telephone may be operated, or an emergency response such as reporting to a medical institution may be performed.

The in-home coordinate information 10 stored in the storage unit 6b can be calculated based on the coordinate information that can be acquired from the camera used as the user information acquisition unit 3, for example. That is, the storage unit 6b of the display control unit 6 stores an in-home coordinate calculation program 22, which is executed by the calculation unit 6a to calculate the in-home coordinate information 10. FIG.
The camera for calculating the indoor coordinate information 10 is preferably installed on the ceiling of the room so that the entire room can be overlooked.
The indoor coordinate information 10 includes coordinate information that can be obtained from a radio range sensor, design information including the floor plan of the house 1, coordinate information that can be obtained from the above-described robot cleaner that is the operation target device, and distance measurement of the smartphone 4. It is possible to make corrections based on measurement information, etc., by the application. In particular, since the robot cleaner has a space sensor that accurately grasps the distance and direction to objects (fixed installations, furniture, etc.) in the surroundings, it is possible to acquire coordinate information on a daily basis.

FIG. 8 is a conceptual diagram of a virtual world corresponding to the real world, and in-home coordinate information 10 calculated by the in-home coordinate calculation program 22 is associated with the virtual world. In other words, the indoor coordinate information 10 calculated by the indoor coordinate calculation program 22 defines the indoor space of the house 1 in the real world, and the virtual world is conceptually constructed accordingly.
The virtual reality character C can appear at any coordinate position in the virtual world associated with the indoor coordinate information 10 . Therefore, the virtual reality character C can suddenly appear at a certain coordinate, or can move from one coordinate, which is the starting point, to another coordinate, which is the reaching point.

As shown in FIG. 8, a house 1 is provided with various stationary installations such as a kitchen table 1a. Since such a fixed installation does not move from its place unless renovation is performed, coordinates are given when calculating the indoor coordinate information 10 . In-home coordinates 11 are also given to the kitchen table 1a shown in FIG. That is, in the virtual world, the location of the kitchen table 1a in the house 1 in the real world is associated with the indoor coordinates 11 indicating the kitchen table 1a.
It should be noted that coordinates may be assigned when calculating the in-home coordinate information 10 for objects that are not fixedly installed but that do not move frequently or that do not move easily, such as furniture and home appliances.
The virtual reality character C is displayed based on the indoor coordinates 11 outside the surface portion of the fixed installation so as not to enter the inside of the fixed installation. Similarly, the virtual reality character C is displayed so as not to get inside the wall or floor. That is, the in-house coordinate information 10 is set on the surface of each room including fixed installations and the like.

Also, as shown in FIG. 8, the position of the user U (home coordinates 12) in the virtual world corresponds to the position of the user U in the real world. When the user U moves around the indoor space of the house 1 in the real world, the user U also moves around within the coordinate range of the indoor coordinate information 10 that defines the virtual space in the virtual world. Specifically, the position information of the user U that can be acquired by the user information acquisition unit 3 is reflected.

Although the in-home coordinate information 10 in this embodiment is three-dimensional coordinates including the x-axis, the y-axis, and the z-axis, it is not limited to this, and two-dimensional coordinates including the x-axis and the y-axis. may be In the case of two-dimensional coordinates, the height position at which the virtual reality character C appears corresponds to the height position of the display portions 4a and 5a in the display devices 4 and 5, respectively.

[About the first server]
Next, the first server 7 is a computer device having a function for supporting actions of the user U, and is connected to the display control section 6 via the communication network N so as to be communicable. Then, from the display control unit 6, the in-house information acquired by the in-house information acquisition unit 2, the user information (including the voice information of the user U) acquired by the user information acquisition unit 3, and the position information of the display devices 4 and 5 etc., and transmits message information generated based on the received information to the display control unit 6 . The message information is transmitted to the user U through the virtual reality character C, and by giving advice and suggestions to the user U, it is possible to support the action of the user U.
As shown in FIG. 6, such a first server 7 includes at least a calculation section 7a, a storage section 7b, a communication section 7c, and an analysis section 7d. In addition, an operation unit such as a mouse and a keyboard, a display unit such as a display, and the like are appropriately provided.

The calculation unit 7a is composed of a CPU, a ROM, a RAM, and the like, and controls the processing operations of the first server 7 in an integrated manner. Specifically, the CPU of the calculation unit 7a reads various processing programs stored in the ROM, develops them in the RAM, and performs various processing in cooperation with the programs.

The storage unit 7b is composed of an HDD, a non-volatile memory, or the like, and stores various data and various programs necessary for realizing the in-home display system. Specifically, for example, a message generation program 30 and the like for generating message information based on information received from the display control unit 6 is stored.
The message generation program 30 is one of the main components in the effect control means, and generates messages (including advice and suggestions) that the virtual reality character C speaks based on the information received from the display control unit 6. The virtual reality character C speaks on the display devices 4 and 5 based on the message information generated by the first server 7 .
The message information generated by the message generating program 30 may be output as voice or as text on the display devices 4 and 5. FIG.
When audio is output, it may be output from the speakers 4 b of the display devices 4 and 5 or may be output from a plurality of speakers in the user information acquisition section 3 . Since the types of message information are diverse and it is difficult to prepare voice in advance, so-called Vocaloid technology is adopted.
When the text is output to the display units 4a and 5a, it is expressed like a balloon in a comic, expressed like subtitles in a movie, or employs expressions of other video contents. As a result, for example, even if the user U has a hearing impairment, it is possible to comprehend what the virtual reality character C is saying.

The storage unit 7b also has a database 40 that stores information received from the display control unit 6, message information generated by the message generation program 30, and home coordinate information and favorable ratings of virtual reality characters. Various information stored in the database 40 is accumulated and optimized.

The communication unit 7c is configured by a network interface or the like, and transmits and receives data to and from an external device (the display control unit 6, the second server 8) connected via the communication network N.

The analysis unit 7d is an AI (artificial intelligence), and is implemented by the cooperation of the CPU and various learning programs stored in the analysis unit 7d. good too. Further, the analysis unit 7d can perform machine learning on the information accumulated in the storage unit 7b and the database 40. FIG. As a result, the user U can have a conversation with the virtual reality character C as if it were a breeding game.

More specifically, the analysis unit 7d performs a text analysis of the conversation exchanged between the user U and the virtual reality character C, and estimates the degree of intimacy between the user U and the virtual reality character C. For this estimation, in addition to using a known familiarity estimation method for the text, teacher data that associates the text of the conversation between the user U and the virtual reality character C with the familiarity may be used.
Then, the likability of the virtual reality character C to the user U is increased or decreased according to the level of the estimated degree of intimacy, and as the likability increases, message information is generated by the message generation program, and the virtual reality character is controlled by the effect control means. The animation display of C is controlled, and the attitude of the virtual reality character C is changed from rude to intimate. Thereby, it is possible to perform an effect such as taking a thoughtful remark or action for the user U according to the favorability. In terms of conversation information, not only the content of the conversation but also the amount of conversation is important in measuring the degree of intimacy. In other words, increasing the number of conversations (increasing points of contact) between the user U and the virtual reality character C is also emphasized. Therefore, in the intimacy estimation described above, the intimacy may be estimated to be higher when the amount of conversation is large, or the intimacy may be estimated from only the amount of conversation.
The analysis unit 7d may also analyze the user's hobbies, preferences, personality, etc. by performing natural language processing analysis on the information (conversation content) of the conversation exchanged between the user U and the virtual reality character C. . In this case, the greater the amount of extracted user information such as the user's hobbies, tastes, and personality, the higher the degree of intimacy. Also, the amount of user information may be accumulated, and the favorability may be set in proportion to this accumulated amount. It should be noted that the contents of the user's hobbies, tastes and personalities themselves may be reflected in the message information generated by the message generation program 30 .
Furthermore, the analysis unit 7d accumulates and stores information on the timing at which the user U started to talk and information on the timing at which the user U refuses to talk as learning data, and performs machine learning to perform machine learning. Analyze when is the best time to start. As a result, it is possible to learn when to speak or when it is better not to speak.
When analyzing the timing, not only the above timing information, but also the position information of the user U before and after the conversation, the face recognition information of the user U by the camera, the voice recognition information of the user U by the microphone, and the wearable device. The analysis may be performed by taking into consideration biological information such as perspiration, body temperature, and pulse from the terminal. As a result, it is possible to learn in what kind of situations it should be spoken, or in which situations it is better not to speak.

In addition, the analysis unit 7d accumulates face recognition information of the user U using a camera, voice recognition information of the user U using a microphone, biometric information such as sweat, body temperature, and pulse from a wearable terminal, outing information of the user U, weather information, and the like. User U's health condition is analyzed by memorizing and performing machine learning. Depending on the result of the analysis, the virtual reality character C is set so as to change the effects such as utterances and actions thrown at the user U.
For example, when the user U is sweating or feeling hot, the virtual reality character C asks the user U whether it is hot based on learning data. The device operation program 21 is executed to operate the air conditioner.
In addition, when the temperature reaches a preset threshold value or higher, the virtual reality character C also makes an effect such as remarks or actions that make the character feel hot. Conversely, when the temperature falls below the preset threshold value, the virtual reality character C also makes an effect such as utterances or actions that make the character feel cold.

In addition, the analysis unit 7d also receives advice and suggestions from the virtual reality character C, accumulates and stores how the user U responds as information, and performs machine learning, so that the user U's Analyze behavioral patterns, preferences, etc. Furthermore, in response to the user U's remarks and actions, what kind of remarks and effects the virtual reality character C performed is stored as information, and machine learning is performed to analyze the user's U's behavior patterns, preferences, and the like. do. Depending on the result of the analysis, the virtual reality character C is set so as to change the effects such as utterances and actions thrown at the user U.

Note that the remarks (message information) and actions of the virtual reality character C are required to correspond firmly to the remarks and actions of the user U. In other words, in the conversation between the user U and the virtual reality character C, it is required that both answers are established, but they do not necessarily have to be regular. Message information is generated by the message generation program 30 according to the result of the analysis by the analysis unit 7d. may be randomly generated to have a conversation with the user U.

[About the second server]
Next, the second server 8 is a computer device for acquiring various information from various external information distribution servers and providing them to the in-home display system. and is communicatively connected.
Various types of information obtained from various external information distribution servers include, for example, weather information, traffic information, shopping information provided by various stores, event information such as sports, art, and fireworks displays, cherry blossom blooming information, and autumn colors. Information ・Seasonal information related to excursions and sightseeing such as the amount of snow at ski resorts is included.
As shown in FIG. 7, such a second server 8 includes at least a calculation section 8a, a storage section 8b, and a communication section 8c. In addition, an operation unit such as a mouse and a keyboard, a display unit such as a display, and the like are appropriately provided.

The calculation unit 8a is composed of a CPU, a ROM, a RAM, and the like, and controls the processing operations of the second server 8 in an integrated manner. Specifically, the CPU of the calculation unit 8a reads various processing programs stored in the ROM, develops them in the RAM, and performs various processing in cooperation with the programs.

The storage unit 8b is composed of an HDD, a non-volatile memory, or the like, and stores various data and various programs necessary for realizing the in-home display system. Specifically, for example, an information acquisition program for acquiring various information from various external information distribution servers, or an information provision request signal from the display control unit 6 or the first server 7 is received and the corresponding information is transmitted. It stores an information transmission program, etc. for performing

The communication unit 8c is configured by a network interface or the like, and transmits and receives data to and from external devices (the display control unit 6, the first server 7, various external information distribution servers) connected via the communication network N. .

Note that the second server 8 is a server specialized for the function of acquiring various information from external information distribution servers and providing the information to the in-home display system, but may be integrated with the first server 7 . That is, various data and various programs stored in the storage unit 8b of the second server 8 are stored in the storage unit 7b of the first server 7 so that they can be executed by the calculation unit 7a.

[About virtual reality characters]
Next, the virtual reality character C will be described in detail.
The virtual reality character C is basically one for each house, but is not limited to this, and may be multiple for each house. When multiple objects are used, for example, they may be of the same type with different colors, or may be of completely different types so that multiple objects can appear at the same time.
When the setting is such that only one virtual reality character C can appear in one house 1, it is preferable that the virtual reality character C does not appear at the same time even if a plurality of display devices 4 and 5 are activated. If the display devices 4 and 5 are separated from each other, it may be set so that the display devices 4 and 5 can appear simultaneously in order to improve convenience.
If the setting is such that a plurality of virtual reality characters C can appear, they may appear one by one in each of the display devices 4 and 5, or may appear simultaneously. When one character appears for each of the display devices 4 and 5, the type of virtual reality character C to appear may be completely random or may be assigned to each display device 4 and 5. FIG.

The virtual reality character C is not treated as a mere "servant", but is set to have personalities and emotions, and to have values such as likes/dislikes. As a result, the user U may develop affection and affection for the virtual reality character C, feel a sense of familiarity with him, have a strong attachment to him, and become an existence that cannot be left alone.
It is preferable that the appearance of the virtual reality character C is such that it is easy to feel attachment and familiarity. Although the virtual reality character C in this embodiment has the appearance of a "fairy", it is not limited to this. For example, a deformed animal or a character that is a collaboration with a child's favorite TV animation may appear, a celebrity with a sponsorship contract may appear in live action, or other original characters may be created. You may Further, it is possible to keep the interest of the user U by updating each time, acquiring new gestures and facial expressions, replacing the virtual reality character C, or making the virtual reality character C appear for a limited time. good.

The virtual reality character C can be displayed on the display units 4a and 5a of the display devices 4 and 5, and is set to appear anywhere according to the positions and orientations of the display devices 4 and 5. FIG. Furthermore, it operates by animation technology, and various productions can be performed according to the place where the virtual reality character C appears, and gestures and facial expressions can be changed according to the production.
For example, the virtual reality character C shown in FIG. 4 is in a state of appearing in the real world indoor scene S2 (kitchen). In the kitchen in the real world, a pot is set on fire, and the virtual reality character C is directed to cook together. Although not shown, the virtual reality character C has a gesture and facial expression of pouring seasonings on, and the user U feels a sense of familiarity with the virtual reality character C who helps with cooking. Also, in the dining room, they make gestures to make the meal enjoyable and eat together.

The utterance (message information) of the virtual reality character C may be randomly generated by the message generation program 30 . As a result, the virtual reality character C can be made to look like it has a capricious personality, and the performance is performed to make the virtual reality character C think that the virtual reality character C has the personality, sense of values, and way of thinking of the virtual reality character C. can be done. The settings related to the temperament of the virtual reality character C may be changed by a dedicated application on the smartphone 4, for example. The personality of the virtual reality character C includes, in addition to capriciousness, seriousness, politeness, attention-seeking, timidity, optimism, pessimism, and humor.

Note that the virtual reality character C in the present embodiment supports the action of the user U by giving advice and suggestions to the user U, and by conversing with the user U, but is limited to this. is not. For example, the appearance may be changed to a deformed animal so that it can be used simply as a pet virtual pet.

The virtual reality character C as described above is displayed based on the indoor coordinate information 10 preset in the house 1 as described above. In other words, it can appear anywhere as long as the coordinates are set. Then, it is displayed on the display units 4a and 5a of the display devices 4 and 5, moves around in the display units 4a and 5a, sometimes disappears, and operates while making gestures, and makes remarks including advice and suggestions. to have a conversation with user U.
In addition, through communication with the user U, the in-home device operation program 21 can be executed to operate various in-home devices.

Furthermore, when a portable information terminal such as a smart phone 4 is used as the display device 4, the virtual reality character C can be taken outside the house 1 (outside the premises). In other words, the function of bringing out the virtual reality character C is given to the dedicated application. When taken outside, for example, using a positioning system such as GPS (Global Positioning System), information related to the location where the dedicated application was started (information that can be obtained through the second server 8), and the virtual reality character C You may perform the production|presentation which links remarks and actions|movements.
Even if the display device 5 other than the smart phone 4 is provided with the same function as the smart phone 4, the virtual reality character C can be brought out to the outside of the house 1 (outside the premises).

[Functions of in-home display system]
The in-home display system in this embodiment functions as a user interface with the virtual reality character C displayed on the display units 4a and 5a of the display devices 4 and 5. take advantage of The in-home display system appropriately combines the configurations of the above units (the in-home information acquisition unit 2, the user information acquisition unit 3, the display devices 4 and 5, the display control unit 6, the first server 7, and the second server 8). , for example, to perform the following functions: That is, the following functions are executed through interaction between the user U and the virtual reality character C.

● Determine the usage status of the energy (electricity, water, gas) used in House 1. Advice and proposals are given to the user U according to the determination result.
● The temperature and humidity inside the house 1 are controlled. That is, the thermal environment information (temperature and humidity, power usage status, etc.) in the house 1 acquired by a temperature and humidity sensor or the like, and the state of the user U acquired by a conversation with the user U or the user information acquisition unit 3 (feels hot). The system provides advice and proposals on how to operate devices in the home, taking into consideration the fact that the user is currently out of the house, is about to go out, etc.).
● When it is detected that the user U is having trouble operating the home appliance through a conversation with the user U or the user information acquisition unit 3, the virtual reality character C is displayed around the home appliance, and the virtual reality character C is displayed. It teaches how to operate using hands and feet, and explains by superimposing virtual marks and marks on the buttons of home appliances.
When it is detected that the user U is having trouble with the cooking method through a conversation with the user U or the user information acquisition unit 3, the virtual reality character C is displayed around the kitchen table 1a, and the virtual reality character C is displayed around the kitchen table 1a. It teaches how to operate using hands and feet, and explains by superimposing virtual landmarks and marks on ingredients.
● When it is detected that the user U is having trouble with his/her studies through a conversation with the user U or by the user information acquiring unit 3, the virtual reality character C is displayed around the user U, and the virtual reality character C is used as a hint or hint. Teach your approach to the correct answer.
● When it is detected that the user U is searching for an item through a conversation with the user U or the user information acquisition unit 3 , the user U and the virtual reality character C tells the user U what to look for.
• Inventory food and other storage items. It detects changes in food and goods and makes shopping suggestions.
● In addition to managing food inventory, propose recipes considering user U's mood (I want to eat ○○ today) acquired through conversation with user U, and suggest shopping as necessary. conduct.
●Notify information including weather information, disaster information and disaster prevention information. Of the weather information, information such as today's weather may be notified in the form of simple display in the corners of the display screens of the display units 4a and 5a.
●Notify traffic information. The traffic information is provided in consideration of the conversation with the user U and the state of the user U acquired by the user information acquisition unit 3 (about to go out).
●Notify customers of shopping information provided by various stores. It may be linked to inventory management of food and other articles, and advertisement income may be obtained from various stores and the like to carry out advertising activities. In that case, the in-home display system may be linked with a product sales system (for example, a mail-order system, a home delivery system) constructed by various stores.
●Notify information about events such as sports, art, and fireworks festivals. Advertisement activities may be carried out by obtaining advertising income from event organizers, promoters, ticket sellers, and the like. In that case, the in-home display system and the ticket sales system may be linked.
●Announcement of seasonal information related to excursions and sightseeing, such as cherry blossom blooming information, autumnal coloring information, and amount of snowfall at ski resorts.
● Conversation, etc., according to the location and situation where the user U is using the display devices 4 and 5 .
● Reads the facial expression and tone of voice of user U, and conducts a conversation according to the expression and tone of voice. For example, when the user U is happy, he/she has a happy conversation. When the user U looks sad, he/she carries out conversations and actions that include words of encouragement accordingly.
● Accumulates and stores conversations and the like with the user U, analyzes the user U's behavior patterns, preferences, and the like, and performs conversations and actions based on the analysis results.
● Accumulates and stores the behavior of the user U in the house 1, analyzes the behavior pattern and preferences of the user U, and conducts conversations and actions based on the analysis results. In addition, the behavior of the user U is predicted, and advice and proposals regarding improvement of safety in life in the house 1 are given.
● Accumulates and stores information such as user U's physical measurements and health condition using a health meter, wearable terminal, etc., and provides health advice and proposals to user U accordingly.
●Improve crime prevention in House 1. That is, since the user U can be identified by the user information acquiring unit 3 including the camera arranged in the house 1, it is easy to identify a person other than the user U, and when a person other than the user U is confirmed, transmits the information to the smartphone 4, or reports to the police or a security company in some cases.
When the user U is notified while the display devices 4 and 5 are not in use, a speaker placed near the user U among the speakers placed in the house 1 emits a sound or a doorbell I will inform you with a signal sound such as. Alternatively, the smartphone 4 may be notified.
● A function for taking out the virtual reality character C can be added to a dedicated application in a mobile information terminal such as the smart phone 4 , and the virtual reality character C can be brought out to the outside of the house 1 (outside the premises). The photographed pictures and the information of the places visited are accumulated and stored so that they can be utilized in future conversations with the user U.例文帳に追加
●Navigation to the destination is performed by linking with the map information system and the positioning system when going out.
● When you go out, you will be notified of store information in the surrounding area and store information captured by the camera of a mobile information terminal such as a smartphone 4.
●Notify the inventories of food and other storage items when going out.
● Make a statement or the like that creates an opportunity for conversation between the user U and a third party (a person appearing in the photographed photo) who is with the user U.

The functions of the in-home display system according to the present embodiment are not limited to those described above, and additions and changes can be made as appropriate without departing from the scope of the present invention.

According to this embodiment, the following excellent effects can be obtained.
That is, the effect control means (display control unit 6, first server 7, second server 8) controls effects including output of message information to be transmitted to user U generated based on user information. , the message information inevitably includes information about the user U himself, so that the user U can easily be interested in the message information. In addition, since the effect control means controls effects including animation display of the virtual reality character C corresponding to such message information, it is possible to make the user U more interested in the virtual reality character C. The user U is more likely to feel attached to the virtual reality character C. As a result, it is possible to encourage the user U to develop an attachment to the system that outputs message information such as advice and proposals, and to improve the frequency of use of the system.

In addition, the production control means (display control unit 6, first server 7, second server 8) controls production including output of message information generated based on the in-house information and user information. necessarily includes information about the house 1 where the user U resides and information about the user U himself, so that the user U can easily be interested in the message information. In addition, since the effect control means controls effects including animation display of the virtual reality character C corresponding to such message information, it is possible to make the user U more interested in the virtual reality character C. The user U is more likely to feel attached to the virtual reality character C. As a result, it is possible to encourage the user U to develop an attachment to the system that outputs message information such as advice and proposals, and to further improve the frequency of use of the system.

In addition, the effect control means (display control unit 6, first server 7, second server 8) places the virtual reality character C at a position associated with the indoor coordinate information 10 in the virtually constructed virtual world. Since the display is controlled, the virtual reality character C can be displayed at any position associated with the indoor coordinate information 10 . As a result, since the user U can enjoy the performance of the virtual reality character C at various places in the house 1, the user U is likely to feel attached to the virtual reality character C, and as a result, the frequency of use of the system is reduced. can contribute to improvement.

In addition, the effect control means (display control unit 6, first server 7, second server 8) determines the intimacy between the user U and the virtual reality character C based on the conversation between the user U and the virtual reality character C. Since the analysis unit 7d for estimating is further provided, the degree of favorability of the virtual reality character C to the user U is increased or decreased according to the estimated level of intimacy, and as the favorability increases, the attitude of the virtual reality character C is changed to that of another person. Change from polite to intimate. As a result, since it is possible to perform effects such as making remarks and taking actions that consider the user U according to the favorability, the user U is more likely to feel attached to the virtual reality character C, and as a result, the system can contribute to increasing the frequency of use of

In addition, in-home device operation means 22 that cooperates with the effect control means (display control program 20 of the display control unit 6, the first server 7, the second server 8) and operates the operation target device provided in the house 1 (In-home device operation program 21) is further provided, so that the in-home device operation means 22 that cooperates with the effect control means can operate the operation target device through conversation and actions between the user U and the virtual reality character C. As a result, the user U can not only enjoy the performance of the virtual reality character C, but also enjoy the pleasure of controlling the facilities of the house 1 through the virtual reality character C. Moreover, the user U did not directly operate the device to be operated, and through conversation with the user U, the user U said, "(Apparently) the virtual reality character C operated the device to be operated for me." can be felt, the attachment to the virtual reality character C is likely to develop, and as a result, the system can be used more frequently.

U User C Virtual reality character N Communication network S1 Interior view (room with a bookshelf)
S2 Home view (pot being cooked)
1 house 1a kitchen table 2 home information acquisition unit 3 user information acquisition unit 4 display device (smartphone)
4a display unit 4b speaker 4c in-camera 5 display device (AR glasses)
5a display unit 6 display control unit 6a calculation unit 6b storage unit 6c wireless communication unit 6d wired communication unit 6e telephone communication unit 6f external communication unit 7 first server 7a calculation unit 7b storage unit 7c communication unit 7d analysis unit 8 Second server 8a Calculation unit 8b Storage unit 8c Communication unit 10 Home coordinate information 11 Home coordinates (position of kitchen table)
12 home coordinates (user's position)
20 display control program 21 home device operation program 22 home coordinate calculation program 30 message generation program 40 database

Claims (5)

a user information acquisition unit for acquiring user information relating to the user's state and the user's position within the house;
a display device used by the user to display a virtual reality character superimposed on the indoor scenery of the house in the real world;
an effect control means that is communicably connected to each of the user information acquisition unit and the display device and controls the effect of the virtual reality character displayed on the display device,
The production control means controls production including output of message information generated based on the user information to be transmitted to the user, and animation display of the virtual reality character corresponding to the message information. An in-home display system characterized by: In the in-home display system according to claim 1,
further comprising an in-home information acquiring unit for acquiring in-house information related to equipment and facilities in the house,
The effect control means is communicably connected to the in-home information acquisition unit, outputs the message information generated based on the in-home information and the user information, and outputs the virtual reality character corresponding to the message information. An in-home display system characterized by controlling the animation display of and the production including. In the in-home display system according to claim 1 or 2,
In-house coordinate information is defined in the indoor space of the house,
The in-home display system, wherein the effect control means performs control to display the virtual reality character at a position associated with the in-home coordinate information in a virtually constructed virtual world. In the in-home display system according to any one of claims 1 to 3,
The in-home display system, wherein the presentation control means further includes an analysis unit that estimates a degree of intimacy between the user and the virtual reality character based on a conversation between the user and the virtual reality character. In the in-home display system according to any one of claims 1 to 4,
An in-home display system, further comprising in-home device operation means for operating an operation target device provided in the home in cooperation with the effect control means.

Images