JP7216940B2 - Communication system, communication method and communication program - Google Patents

Description

The present invention relates to a communication system, communication method and communication program.

In recent years, HMDs (Head Mounted Displays) and the like have been proposed that allow users to experience virtual reality (VR) and augmented reality (AR). Among these, as an HMD using AR technology, AR glasses, which are worn on the user's head and used in the same manner as general eyeglasses, have been proposed.

The AR glasses capture an image of the real space in front of the user, and display the captured image of the real space on a display corresponding to the lens portion of the AR glasses. In addition, AR glasses display virtual content generated by CG on a display by superimposing it on a captured image of the real space. As a result, a user wearing AR glasses can experience an AR space in which virtual content is superimposed on a real space.

Japanese Patent No. 6655751

A system has been proposed that allows a user to experience an RPG (Role-Playing Game) by displaying RPG (Role-Playing Game) content on an AR glass display. There has been a demand for this system to provide users with a realistic AR experience.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a communication system, a communication method, and a communication program capable of providing a user with a realistic virtual space experience.

In order to solve the above-described problems and achieve the object, the communication system of the present invention is a communication system that includes a display device used by a user, a smart device carried by the user, and a server that communicates with the smart device. The server determines the user's surroundings based on at least the location information of the smart device and the map information, and creates virtual content corresponding to the user's surroundings such as animal and plant character images or animal and a transmission unit configured to transmit virtual content selected by the selection unit to a smart device, and the smart device detects at least location information of the smart device. and a control unit for outputting virtual content to a display device.

According to the present invention, it is possible to provide a user with a realistic virtual space experience.

FIG. 1 is a diagram showing an example of an image displayed on a display of AR glasses. FIG. 2 is a diagram illustrating the configuration of the communication system according to the embodiment. FIG. 3 is a block diagram showing an example of the configuration of the communication system according to the embodiment. 4 is a block diagram showing an example of the configuration of the AR glasses shown in FIG. 3. FIG. FIG. 5 is a block diagram showing an example of the configuration of the smart device shown in FIG. 3. As shown in FIG. FIG. 6 is a block diagram showing an example of the configuration of the server shown in FIG. 3; FIG. 7 is a diagram exemplifying a map of an island visited by the user. FIG. 8 is a diagram for explaining virtual content output by AR glasses. FIG. 9 is a diagram for explaining virtual content output by AR glasses. FIG. 10 is a sequence diagram illustrating a procedure of communication processing in the communication system according to the embodiment; FIG. 11 is a diagram showing a computer executing a program.

Hereinafter, embodiments of a communication system, a communication method, and a communication program according to the present application will be described in detail with reference to the drawings. Note that the communication system, communication method, and communication program according to the present application are not limited by this embodiment.

In the following embodiments, a communication system, a communication method, and a processing flow of a communication program according to the embodiments will be described in order, and finally effects of the embodiments will be described.

[Embodiment]
First, an embodiment will be described. In the embodiment, AR glasses capable of displaying virtual RPG content, for example, are used as the display device used by the user who is sightseeing in a predetermined area, specifically, a sightseeing spot (for example, an island). We will explain a system that allows you to experience RPG while wearing it and sightseeing in tourist spots.

FIG. 1 is a diagram showing an example of an image displayed on a display of AR glasses. As illustrated in FIG. 1, a monster content C1 and a coin content C2 are superimposed on an image of an actual island (for example, an image of a forest) and displayed on the AR glass, and the user defeats the monster and collects the coin. It enables game experiences such as collecting and working on missions. In addition, communication is possible between users wearing AR glasses.

In the embodiment, the entire small island is used to allow the user to move his or her body and communicate with friends, thereby enabling an RPG experience unique to the area. Furthermore, in the embodiment, a realistic AR experience is provided by outputting virtual content to the AR glasses according to the surrounding conditions of the user who is actually sightseeing by the user wearing the AR glasses. The virtual contents are, for example, animal and plant character images created by CG or the like, or sounds such as animal cries and footsteps. Note that the character image is not limited to a game character-like image, and may be an image created from a photograph or the like, and includes moving images.

[Configuration of communication system]
A configuration of a communication system according to an embodiment will be described. FIG. 2 is a diagram illustrating the configuration of the communication system according to the embodiment. FIG. 3 is a block diagram showing an example of the configuration of the communication system according to the embodiment.

As shown in FIGS. 2 and 3, the communication system 1 according to the embodiment includes an AR glass 10 (display device), a motion controller 20 held by a user, a smart device 30 carried by a user, a data center, and the like. has a server 40 located in

The AR glasses 10 are worn on the user's head like general eyeglasses. The AR glasses 10 have an imaging function, an audio input function, a communication function, a display function, and an audio output function. The AR glasses 10 capture an image of the real space, and display the captured image of the real space on a display corresponding to the lens portion of the AR glasses 10 . In addition, under the control of the smart device 30, the AR glasses 10 superimpose the virtual content generated by CG on the captured image of the real space and display it on the display. Under the control of the smart device 30, the AR glasses 10 also output virtual content according to the user's surroundings.

The motion controller 20 is a controller held by the user. The motion controller 20 has, for example, an acceleration sensor and a communication function for transmitting detection results of the acceleration sensor to the smart device 30 .

The smart device 30 is a device carried by a user, for example, a smart device such as a tablet or a smartphone, and has a position detection function, an image recognition function, and a voice recognition function. The smart device 30 controls the AR glasses 10 . The smart device 30 acquires the motion of the user's hand based on the detection result of the acceleration sensor transmitted from the motion controller 20 . The smart device 30 connects to the server 40 via the network N and transmits the user's position detection result, image recognition result, and voice recognition result to the server 40 . The smart device 30 receives virtual content output from the AR glasses 10 via the network N from the server 40 . The smart device 30 controls output of virtual content provided by the server 40 to the AR glasses 10 .

The server 40 holds map information and virtual content in the AR space, and determines the user's surroundings based on the user's position detection results, image recognition results, and voice recognition results transmitted from the smart device 30. do. The server 40 then selects virtual content according to the user's surroundings and transmits it to the smart device 30 .

[AR glass]
Next, the AR glasses 10 will be explained. FIG. 4 is a block diagram showing an example of the configuration of the AR glasses 10 shown in FIG. 3. As shown in FIG.

As shown in FIG. 4 , the AR glasses 10 have an imaging unit 11 , an audio input unit 12 (recording unit), a communication unit 13 , a display unit 14 and an audio output unit 15 .

The imaging unit 11 images the real space. For example, when the imaging direction of the imaging unit 11 is in front of the AR glasses 10 , the imaging unit 11 images the real space in front of the AR glasses 10 . The imaging unit 11 generates an image of a real space by processing a photoelectric conversion element such as a CCD or a CMOS and a photoelectric conversion signal output from the imaging element. The imaging unit 11 acquires a two-dimensional image of three colors of R (red), G (green), and B (blue), for example.

The voice input unit 12 is configured by a device that receives input of voice and records the input voice.

The communication unit 13 performs wireless communication with the smart device 30 . The communication unit 13 transmits the image captured by the imaging unit 11 and the audio recorded by the audio input unit 12 to the smart device 30 . The communication unit 13 receives virtual content transmitted from the smart device 30 . The virtual contents are character images of animals and plants, sounds such as animal cries and footsteps.

The display unit 14 is configured by a display. The display unit 14 displays the image of the real space captured by the imaging unit 11 . The display unit 14 displays the animal and plant character images included in the virtual content received by the communication unit 13 so as to be superimposed on the real space image. Note that the display unit 14 may be a display capable of transmissive display, and may display character images of animals and plants among the virtual contents.

The audio output unit 15 outputs audio. The audio output unit 15 outputs sounds such as animal barks and footsteps from the virtual content received by the communication unit.

[Smart device]
Next, the smart device 30 is described. FIG. 5 is a block diagram showing an example of the configuration of the smart device 30 shown in FIG. 3. As shown in FIG. As shown in FIG. 5 , the smart device 30 has a communication section 31 , a storage section 32 and a control section 33 .

The communication unit 31 performs wireless or wired communication with other devices. The communication unit 31 is a communication interface that transmits and receives various information to and from another device connected via a network or the like. The communication unit 31 is realized by a NIC (Network Interface Card) or the like, and performs communication between another device and the control unit 33 (described later) via an electric communication line such as a LAN (Local Area Network) or the Internet. For example, the communication unit 31 receives images and sounds from the AR glasses 10 and transmits virtual content to the AR glasses 10 . The communication unit 31 transmits position detection results, image recognition results, and voice recognition results to the server 40 and receives virtual content from the server 40 .

The storage unit 32 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disc, or the like. Note that the storage unit 32 may be a rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 32 stores an OS (Operating System) and various programs executed by the smart device 30 . Furthermore, the storage unit 32 stores various information used in executing the program. The storage unit 32 stores image data 321 and audio data 322 transmitted from the AR glasses 10 and content information 323 including virtual content transmitted from the server 40 .

The control unit 33 controls the smart device 30 as a whole. The control unit 33 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 33 also has an internal memory for storing programs defining various processing procedures and control data, and executes each processing using the internal memory. Further, the control unit 33 functions as various processing units by running various programs. The control unit 33 has a position detection unit 331, an image recognition unit 332, a voice recognition unit 333, a motion detection unit 334, and an AR glasses control unit 335 (control unit).

The position detection unit 331 detects the position of the smart device 30 . The position detection unit 331 can acquire the position information of the smart device 30 by GPS (Global Positioning System), for example. The position detection unit 331 periodically acquires the position of the smart device 30 and transmits the position information represented by latitude and longitude to the server 40 via the communication unit 31 .

The image recognition unit 332 processes the image captured by the AR glasses 10 and recognizes features of the image such as an object appearing in the image. The image recognition unit 332 recognizes animals, plants, landscapes, and buildings appearing in images captured by the AR glasses 10 . The image recognition unit 332 performs image recognition that has already learned image features such as animals, plants, landscapes, and buildings that are specific to a predetermined area, specifically, a sightseeing spot that the user is currently sightseeing, by, for example, machine learning. Using the model, the features of the image captured by the AR glasses 10 are recognized. The image recognition unit 332 may perform processing to recognize an animal, a plant, a landscape, or a building when there is a predetermined change or more in the image by comparing the images before and after the image. The image recognition section 332 transmits the image recognition processing result to the server 40 via the communication section 31 .

The voice recognition unit 333 processes the voice acquired by the AR glasses 10 and recognizes the features of the voice. The voice recognition unit 333 recognizes which animal's cry or footsteps the voice acquired by the AR glasses 10 is. The voice recognition unit 333 recognizes whether the voice acquired by the AR glasses 10 is the sound of the river. The voice recognition unit 333 has already learned, for example, by machine learning, voice features such as animal cries and footsteps, river sounds, and the like peculiar to a predetermined area, specifically, a tourist spot where the user is sightseeing. A voice recognition model is used to recognize the features of the voice acquired by the AR glasses 10 . The voice recognition unit 333 transmits the voice recognition processing result to the server 40 via the communication unit 31 . Note that the speech recognition unit 333 may process speech acquired by a speech input unit (not shown) of the smart device 30 .

Note that the server 40 may have the image recognition function and the voice recognition function. At that time, the smart device 30 transmits the image captured by the AR glasses 10 and the sound acquired by the AR glasses 10 to the server 40 .

The motion detection unit 334 acquires the user's hand motion based on the detection result of the acceleration sensor transmitted from the motion controller 20 .

The AR glasses control unit 335 controls the processing of the AR glasses 10 . For example, the AR glasses control unit 335 controls image display processing on the display of the lens portion of the AR glasses 10 and audio output processing by the AR glasses 10 . The AR glasses control section 335 has a content acquisition section 336 and a content synthesis section 337 .

The content acquisition unit 336 acquires virtual content transmitted from the server 40 . This virtual content is a character image superimposed on an image in the real space, and sounds such as animal cries and footsteps. The AR glasses control unit 335 causes the audio output unit 15 of the AR glasses 10 to output audio when the content acquired by the content acquisition unit 336 is a sound such as an animal's cries or footsteps.

When the content acquired by the content acquisition unit 336 is a character image superimposed on a real space image, the content synthesizing unit 337 performs synthesis processing of the character image and the real space image. After the composition processing by the content composition unit 337, the AR glasses control unit 335 causes the display unit 14 of the AR glasses 10 to display an image in which the character image is superimposed on the image of the real space. The superimposition position of the character image is set in advance, for example. Alternatively, the character image superimposition position is set by an instruction from the server 40 .

[server]
Next, the server 40 will be explained. FIG. 6 is a block diagram showing an example of the configuration of server 40 shown in FIG. As shown in FIG. 6 , the server 40 has a communication section 41 , a storage section 42 and a control section 43 .

The communication unit 41 performs wireless or wired communication with other devices. The communication unit 41 is a communication interface that transmits and receives various information to and from another device connected via a network or the like. The communication unit 41 is realized by a NIC (Network Interface Card) or the like, and performs communication between another device and the control unit 43 (described later) via an electric communication line such as a LAN (Local Area Network) or the Internet. For example, the communication unit 41 receives location information, image recognition results, and voice recognition results from the smart device 30 and transmits virtual content to the smart device 30 .

The storage unit 42 is a storage device such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disc, or the like. Note that the storage unit 42 may be a rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 42 stores an OS (Operating System) and various programs executed by the server 40 .

Further, the storage unit 42 stores various information used in executing the program. The storage unit 42 stores map information 421 including forests, rivers, mountains, plains, sights, etc. of sightseeing spots such as islands visited by the user, and content information 422 including various types of virtual content. The map information 421 may describe hidden spots, or may describe places that tend to be muddy. The various virtual contents are character images of animals and plants, or sounds of footsteps or cries of animals, and are provided in predetermined areas such as sightseeing spots, forests, rivers, mountains, plains, famous places, and the like. It is a content specific to each predetermined place in the sightseeing spot, which is created in advance in association with the place. Also, various contents may be created separately for enemies and allies. The content information 422 includes, for example, setting conditions for various types of content, which are setting information such as the display position and size of character images.

The control unit 43 controls the server 40 as a whole. The control unit 43 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 43 also has an internal memory for storing programs defining various processing procedures and control data, and executes each processing using the internal memory. Further, the control unit 43 functions as various processing units by running various programs. The control unit 43 includes a position information acquisition unit 431 , an image recognition result acquisition unit 432 that acquires the image recognition result transmitted from the smart device 30 , and a voice recognition result acquisition unit 433 that acquires the voice recognition result transmitted from the smart device 30 . , a condition setting unit 434, a content selection unit 435 (selection unit), and a content transmission control unit 436 (transmission unit).

The location information acquisition unit 431 detects the location of the user based on the location information received from the smart device 30 and the map information 421 . Based on the user's position, the position information acquisition unit 431 detects whether the user is in a tourist spot such as a forest, river, mountain, plain, famous place, or the like. The position information acquisition unit 431 may determine the positions of mountains and plains based on contour lines included in the map information 421 .

The condition setting unit 434 sets conditions for selection of contents related to animals to appear according to the user's surroundings and conditions for determining friend or foe for each predetermined area (for example, a tourist spot such as an island). For example, the condition setting unit 434 extracts a rule according to the area where the user is actually sightseeing from among the content selection rules and friend/enemy judgment rules created in advance for each sightseeing spot such as an island. , set. In this way, the condition setting unit 434 extracts content selection rules and friend/enemy determination rules according to the tourist destination that the user is actually visiting, and enables selection of content that is representative of the tourist destination. .

The content selection unit 435 determines the surrounding situation of the user based on the user's location information from the location information acquisition unit 431 and the image recognition result from the smart device 30 or the voice recognition result from the smart device 30 . Then, the content selection unit 435 selects virtual content in the AR space according to the surrounding circumstances of the user. When the content selection unit 435 determines that the surrounding situation of the user corresponds to any one of the predetermined locations in the predetermined area, the content selection unit 435 selects a character image of an animal or plant peculiar to the corresponding predetermined location or an animal cry peculiar to the predetermined location. Alternatively, footstep sounds are selected as virtual content.

At this time, the content selection unit 435 uses the content selection rule and the enemy/friend determination rule set by the condition setting unit 434 . The content selection unit 435 uses the content selection rule and the enemy/friend determination rule set by the condition setting unit 434. For example, when the user is sightseeing on an island, the content selection unit 435 selects characters of animals and plants peculiar to the island. Select an image as content. Also, when the content selection unit 435 determines that the user is walking in the forest, it selects the sounds, footsteps, or character images of animals unique to the forest as content. In addition, if the content selection unit 435 determines that the user is in a forest based on the position detection result, and further recognizes the presence of a specific bird that inhabits the forest based on the cry recognized in the voice recognition result, The bird's song sound is selected as content. In addition, the content selection unit 435 determines that the user is in a forest, and if the footsteps recognized in the image recognition result confirm the presence of a unique animal that inhabits the forest, the content selection unit 435 determines that the animal is in the forest. Select a character image as content.

At this time, the content selection unit 435 may select enemy or ally content according to the user's surroundings. For example, when the user reaches a hidden sightseeing spot on the island, the content selection unit 435 selects the character image in the state of cheering as an ally among the character images of flora and fauna peculiar to the island. . Furthermore, for example, the content selection unit 435 may select a character image of a friendly bird when recognizing a bird's cry, or may select a friendly character image when detecting that another user is nearby. You can increase it. In addition, when the content selection unit 435 determines that the surroundings of the user are muddy and the situation is bad based on the map information 421, the image recognition result, the weather information, etc., the content selection unit 435 selects the animal sounds peculiar to this island. A threatening call may be selected as a

The content transmission control unit 436 transmits the virtual content on the AR space selected by the content selection unit 435 to the smart device 30 . The content transmission control unit 436 also transmits the content output timing and the content display position to the smart device 30 together with the selected content.

[Content output example]
FIG. 7 is a diagram exemplifying a map of an island visited by the user. 8 and 9 are diagrams for explaining virtual content output by the AR glasses 10. FIG.

As shown in FIG. 7, the user is sightseeing an island with forests, rivers, mountains, plains, sights, and the like. As shown in FIG. 8, a case where the user U wearing the AR glasses 10 is walking in the forest located at the point P1 in FIG. 7 will be described as an example.

In communication system 1, server 40 determines that user U is in the forest at point P1. Then, if a wild boar exists in the forest ((1) in FIG. 8), the server 40 transmits a character image G1 of the wild boar as a companion to the smart device 30 as virtual content. The smart device 30 displays the wild boar character image G1 on the display of the AR glasses 10 so that the wild boar becomes a companion ((2) in FIG. 8). The AR glasses 10 make the wild boar appear by superimposing the character image of the wild boar on G1, the actual image of the forest. At this time, the smart device 30 may cause the AR glasses 10 to display the wild boar character image G1 as if it is near the user or as if it moves together with the user U.

The server 40 may also transmit the sound of the footsteps of a wild boar to the smart device 30 as virtual content. The smart device 30 may output the boar character image and footsteps only while the boar is in the forest, or may output the boar character until a predetermined distance away from the forest.

In addition, if there is a rare bird that lives only in the forest where the user U is located ((3) in FIG. 8), the server 40 uses the bird's cries as virtual content. Send to device 30 . The smart device 30 outputs the bird's song from the AR glasses 10 ((4) in FIG. 8). The smart device 30 may output the chirping of birds only while in the forest, or may output the sounds of the birds until a predetermined distance away from the forest.

In the communication system 1, the server 40 determines whether the user U is on the river at the point P2 based on the voice recognition result of the river sound ((1) in FIG. 9), the position detection result, and the map information by the smart device 30. determine that there is Then, if there are fish in the river ((2) in FIG. 9), the server 40 transmits the character image G2 of the companion dolphin to the smart device 30 as virtual content.

The smart device 30 displays the character image G2 of the dolphin on the display of the AR glasses 10 so that the dolphin becomes a companion ((3) in FIG. 9). The AR glasses 10 make the dolphin appear by superimposing the character image G2 of the dolphin on the actual image of the forest. It should be noted that these characters may be displayed as if they were nearby in a battle scene or the like, or as if they were moving together.

In this way, the communication system 1 determines the surrounding conditions of the user U wearing the AR glasses 10, such as the appearance of animals, actual topography (forests, rivers, mountains, etc.) and famous places. Then, the communication system 1 makes the user U hear the cries and footsteps of animals according to the surrounding situation of the user U, and makes animal and plant characters appear according to the user U's surrounding situation. Thereby, the communication system 1 provides the user U with a more realistic AR experience by allowing the user U to experience the presence of animals in the user U's surroundings.

The communication system 1 may display character images of animals and plants peculiar to tourist spots on the AR glasses 10 for each tourist spot such as an island, or let the user U hear the cries and footsteps of animals peculiar to tourist spots. good too.

In addition, when the communication system 1 makes the user U hear the sounds of animal cries and animal footsteps, or when displaying animal and plant character images on the AR glasses 10, the communication system 1, depending on the surrounding situation of the user U, You may change the voice and character image when becoming an ally. For example, when outputting an animal's cry, if it is an ally, it outputs a pleasant cry, and if it is an enemy, it outputs a menacing cry. Further, when the character images of animals and plants are displayed on the AR glasses 10, in the battle scene, if it is an ally, it may be displayed as if it is cheering, and if it is an enemy, it may be displayed as if it is attacking the user U. may be displayed.

Further, the communication system 1 may determine whether the user is an enemy or an ally according to the surrounding circumstances of the user. For example, when the user's surroundings are bad, for example, when it is muddy or dark, an enemy is made to appear. Also, when the user's surroundings are good, for example, when the view is good or when it is bright, an ally appears.

[Processing procedure of communication processing]
FIG. 10 is a sequence diagram illustrating a procedure of communication processing in the communication system according to the embodiment;

As shown in FIG. 10, each time the smart device 30 detects the location of the smart device 30 (step S1), it transmits location information to the server 40 (step S2).

Then, when the image is transmitted from the AR glasses 10 (step S3), the smart device 30 executes image recognition processing on the image (step S4), and transmits the image recognition result to the server 40 (step S5). When the voice is transmitted from the AR glasses 10 (step S6), the smart device 30 performs voice recognition processing on the voice (step S7), and transmits the voice recognition result to the server 40 (step S8).

The server 40 detects the user's position based on the received position detection result and the map information 421, and for each predetermined area (for example, a tourist spot such as an island), there is a Conditions for content selection and friend/enemy determination are set (step S9). The server 40 determines the surrounding situation of the user based on the position detection result, the image recognition result, and the voice recognition result (step S10). The server 40 selects virtual content in the AR space according to the user's surroundings (step S11), and transmits it to the smart device 30 (step S12).

If the content is a character image of an animal or plant, the smart device 30 synthesizes the character image and the real space image (step S13). Then, the smart device 30 instructs the AR glasses 10 to output content (step S14), and causes the AR glasses 10 to output virtual content (for example, animal and plant character images and sounds) in the AR space (step S14). S15).

[Effects of Embodiment]
As described above, in the present embodiment, the user can hear the cries and footsteps of animals according to the user's surroundings, or the characters of animals and plants according to the user's surroundings can appear. By allowing the user to feel the presence of the animal in the room, it is possible to provide the user with a more realistic AR experience in the virtual space.

In addition, in the present embodiment, by outputting content related to animals unique to a sightseeing spot such as an island visited by the user, it is possible to allow the user to experience an experience unique to the tourist spot. As a result, according to the present embodiment, it is possible to promote experiential tourism through game-like elements and content output unique to tourist spots.

[System configuration, etc.]
Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Furthermore, all or any part of each processing function performed by each device is realized by a CPU or GPU and a program analyzed and executed by the CPU or GPU, or as hardware by wired logic can be realized.

Further, among the processes described in this embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

[program]
It is also possible to create a program in which the processes executed by the AR glasses 10, the smart device 30, and the server 40 described in the above embodiments are described in a computer-executable language. For example, it is possible to create a program in which the processes executed by the AR glasses 10, the smart device 30, and the server 40 in the embodiment are described in a computer-executable language. In this case, the same effects as those of the above embodiments can be obtained by having the computer execute the program. Further, such a program may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read by a computer and executed to realize processing similar to that of the above embodiments.

FIG. 11 is a diagram showing a computer executing a program. As illustrated in FIG. 11, computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. , and these units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 as illustrated in FIG. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090 as illustrated in FIG. A disk drive interface 1040 is connected to the disk drive 1100 . A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100 . Serial port interface 1050 is connected to mouse 1110 and keyboard 1120, for example. Video adapter 1060 is connected to display 1130, for example.

Here, as illustrated in FIG. 11, the hard disk drive 1090 stores an OS 1091, application programs 1092, program modules 1093, and program data 1094, for example. That is, the above program is stored, for example, in hard disk drive 1090 as a program module in which instructions to be executed by computer 1000 are described.

Various data described in the above embodiments are stored as program data in the memory 1010 or the hard disk drive 1090, for example. Then, the CPU 1020 reads the program modules 1093 and program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 as necessary, and executes various processing procedures.

Note that the program module 1093 and program data 1094 related to the program are not limited to being stored in the hard disk drive 1090. For example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive or the like. . Alternatively, the program module 1093 and program data 1094 related to the program are stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.), and are transmitted via the network interface 1070. It may be read by CPU 1020 .

The above-described embodiments and modifications thereof are included in the scope of the invention described in the claims and their equivalents, as well as in the technology disclosed in the present application.

10 AR glasses 11 imaging unit 12 audio input unit 13, 31, 41 communication unit 14 display unit 15 audio output unit 20 motion controller 30 smart device 32, 42 storage unit 33, 43 control unit 40 server 321 image data 322 audio data 323, 422 content information 331 position detection unit 332 image recognition unit 333 voice recognition unit 334 motion detection unit 335 AR glasses control unit 336 content acquisition unit 337 content synthesis unit 421 map information 431 position information acquisition unit 432 image recognition result acquisition unit 433 voice recognition result Acquisition unit 434 Condition setting unit 435 Content selection unit 436 Content transmission control unit

Claims (5)

A communication system comprising a display device worn on the head of a user, a smart device carried by the user, and a server communicating with the smart device,
The display device
an imaging unit that captures an image of a real space;
a voice input unit for receiving input of voice;
a communication unit that communicates with the smart device and transmits the image of the real space and the sound to the smart device;
has
The communication system shall:
an image recognition unit that recognizes features of the image in the real space;
a speech recognition unit that recognizes the features of the speech;
has
The server is
Based on at least the location information of the smart device and the map information, the surrounding situation of the user is determined, and as virtual content according to the user's surrounding situation, character images of animals and plants, animal footsteps, or a selection unit for selecting the voice of the bark;
a transmission unit configured to transmit the virtual content selected by the selection unit to the smart device;
has
the smart device includes a location detection unit that detects at least location information of the smart device and transmits location detection results to the server;
a control unit for outputting the virtual content to the display device;
has
The selection unit determines a surrounding situation of the user based on the location information of the smart device, the map information, and the result of image recognition by the image recognition unit or the result of voice recognition by the voice recognition unit,
The virtual content is created separately for enemies or allies,
The communication system , wherein the selection unit selects enemy or ally content from the virtual content according to the user's surroundings . The virtual contents are character images of animals and plants peculiar to each prescribed location in a prescribed area or sounds of animal cries or footsteps peculiar to tourist spots,
When the selection unit determines that the surrounding situation of the user corresponds to any one of the predetermined locations in the predetermined area, the selection unit selects a character image of animals and plants peculiar to the predetermined location or an animal peculiar to the predetermined location. 2. The communication system according to claim 1, wherein sounds such as cries or footsteps are selected as the virtual contents. The display device
a display unit for displaying an image;
an audio output unit that outputs audio;
has
When the virtual content is a character image of the animal or plant, the control unit causes the display unit to display the virtual content, and causes the audio output unit to output the sound of footsteps or cries of the animal as the virtual content. 3. The communication system according to claim 1 or 2, characterized by: A communication method executed by a communication system having a display device worn on the head of a user, a smart device carried by the user, and a server communicating with the smart device, comprising:
a step in which the display device captures an image of a real space;
a step in which the display device accepts a voice input;
a step of the display device communicating with the smart device and transmitting the image of the real space and the sound to the smart device;
the smart device recognizing features of the image of the real space;
the smart device recognizing features of the voice;
The server determines the surrounding situation of the user based on at least the location information of the smart device and the map information, and generates character images of animals and plants as virtual content according to the user's surrounding situation, or selecting an animal footstep or call sound;
the server transmitting the virtual content selected in the selecting step to the smart device;
the smart device detecting at least location information of the smart device and transmitting the location detection result to a server;
a step of causing the server to output the virtual content to the display device;
the display device outputting the virtual content;
including
The step of selecting determines the surrounding situation of the user based on the location information of the smart device, the map information, and the result of image recognition in the step of recognizing or the result of voice recognition in the step of recognizing;
The virtual content is created separately for enemies or allies,
The communication method , wherein the selecting step selects enemy or ally content from the virtual content according to the user's surroundings . A communication program that causes a computer to perform the method, comprising:
A computer as a display device worn on the user's head ,
imaging a real space;
accepting voice input;
communicating with a smart device to transmit the real space image and the audio to the smart device;
and
computer as a server,
Based on at least the location information of the smart device carried by the user and the map information, the surrounding situation of the user is determined, and the virtual contents according to the user's surrounding situation are character images of animals and plants, or animals. selecting a sound of footsteps or cries of
transmitting the virtual content selected in the selecting step to the smart device;
and
a computer as the smart device;
recognizing features of the real space image;
recognizing features of said speech;
detecting location information of at least the smart device and transmitting the location detection result to a server;
a step of outputting the virtual content to a display device used by a user;
and
a computer as the display device,
executing the step of outputting the virtual content ;
The step of selecting determines the surrounding situation of the user based on the location information of the smart device, the map information, and the result of image recognition in the step of recognizing or the result of voice recognition in the step of recognizing;
The virtual content is created separately for enemies or allies,
A communication program according to claim 1, wherein said step of selecting selects enemy or ally content from said virtual content according to the user's surroundings .

Images