JP2018097664A - Information processing system, information processing method and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively convey an image and a sound suitable for a desired scene to a user by using a mechanism of virtual reality.SOLUTION: An information processing system according to an embodiment includes: an extraction unit that extracts image data for allowing a user to experience virtual reality and sound data independent of the image data from a database; and an output unit that outputs the image data extracted by the extraction unit to a display for displaying the image data to the user without substantially showing an actual landscape to the user and outputs the sound data extracted by the extraction unit to a loudspeaker.SELECTED DRAWING: Figure 3

Description

  One aspect of the present invention relates to an information processing system, an information processing method, and an information processing program for allowing a user to listen to a predetermined sound while experiencing virtual reality. This information processing system, information processing method, and information processing program are used for mental training, for example.

  2. Description of the Related Art Conventionally, a technique for transmitting voice advice to a user while allowing the user to experience virtual reality (VR) is known. For example, Patent Literature 1 below describes a golf practice assisting device that can actually swing and hit a ball while looking at the front or side of the golf practitioner himself. This device includes a head-mounted display that can simultaneously view an actually viewed landscape and a video image, and that can output sound so that the flow of body movement in a swing can be performed with a constant rhythm.

JP-A-10-127850

  The technique described in Patent Document 1 is specialized in golf practice and lacks versatility. Therefore, it is desired to effectively convey to the user images and sounds suitable for a desired scene using a virtual reality mechanism.

  An information processing system according to an aspect of the present invention is extracted by an extraction unit that extracts image data for allowing a user to experience virtual reality and audio data independent of the image data, and an extraction unit. Outputting the image data to a display that displays the image data to the user without substantially showing the actual scenery to the user, and an output unit that outputs the audio data extracted by the extraction unit to a speaker. Prepare.

  An information processing method according to an aspect of the present invention is an information processing method executed by an information processing system including a processor, and is independent of image data for allowing a user to experience virtual reality and the image data. Extracting the audio data from the database; and outputting the image data extracted in the extraction step to a display that displays the image data to the user without substantially showing the actual scenery to the user; An output step of outputting the audio data extracted in the extraction step to a speaker.

  An information processing program according to an aspect of the present invention is extracted in an extraction step of extracting image data for allowing a user to experience virtual reality and audio data independent of the image data from the database, and the extraction step. Outputting the image data to a display that displays the image data to the user without substantially showing the actual scenery to the user, and outputting the audio data extracted in the extraction step to a speaker; Let the computer system run.

  In such an aspect, image data for experiencing virtual reality is provided to the user via the display, and the user views the image without substantially viewing the actual landscape. Furthermore, audio independent of the image is provided to the user. By outputting the image and sound selected for the desired scene to the user in this way, it is possible to effectively convey the sound and sound to the user.

  According to an aspect of the present invention, an image and sound suitable for a desired scene can be effectively transmitted to a user using a virtual reality mechanism.

It is a figure which shows the specific example of the information processing system which concerns on embodiment. It is a figure which shows the hardware constitutions of the terminal which concerns on embodiment. It is a figure which shows the function structure of the information processing system which concerns on embodiment. It is a figure which shows the example of an interface screen. It is a flowchart which shows operation | movement of the information processing system which concerns on embodiment. It is a figure which shows the structure of the information processing program which concerns on embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  The function and configuration of the information processing system 1 according to the embodiment will be described with reference to FIGS. The information processing system 1 is a computer system that outputs a predetermined image and sound to a user for the purpose of exerting a predetermined psychological effect on the user. A user is a person who perceives images and sounds provided from the information processing system. A psychological effect is an effect or influence that acts on a person's mind or behavior. For example, the information processing system 1 may be a computer system that supports mental training. “Mental training” in this specification is training aimed at bringing a user's mind or behavior into a state desired by the user or approaching it. Examples of effects obtained by the user using the information processing system 1 include relaxation effects, improved motivation, improved concentration, relaxation of tension, improved performance, reduced stress, and the like. Of course, the purpose of use of the information processing system 1 is not limited to these, and the information processing system 1 may be used for any purpose (for example, a purpose other than mental training).

  An image is an image that can be viewed by a person by being displayed on a display, and may be a still image or a moving image. The image may be a still image or a moving image taken by a camera, or may be a still image or a moving image (animation) created by computer graphics. Alternatively, the image may be a mixture of captured images and computer graphics. The image used by the information processing system 1 is used for allowing the user to experience virtual reality. The user can experience virtual reality by viewing an environment that feels substantially like reality through the information processing system 1 (more specifically, a display). An example of an image for allowing a user to experience virtual reality is an image of a landscape including at least one of a natural object and an artificial object, but the object represented by the image is not limited at all. For example, the image may be a natural scene, a competition venue, or a concert hall. In view of effective virtual reality, the image may be a 360 ° image (360 ° still image or 360 ° video) that surrounds the user. Of course, the image need not be a 360 ° image.

  The sound is a sound or voice that can be heard by a person by being output from a speaker. The voice may include at least a spoken language with which the user can understand the meaning, and may include, for example, advice for mental training (advice by a mental training trainer). Alternatively, the voice may include other sounds or voices (eg, music, natural sounds, artificial sounds, humming, etc.). Alternatively, the voice may include both spoken language and other sounds or voices. In consideration of the psychological effect on the user, the sound may include a sound including a frequency of 528 hz or 4096 hz, or may include a 1 / f environmental sound.

  The information processing system 1 includes a terminal 10, a display 20, a speaker 30, and a biological measuring instrument 40. The terminal 10 can perform data communication with each of the display 20, the speaker 30, and the biometric instrument 40 via a wireless network or a wired network. For example, the terminal 10 may be connected to the display 20, the speaker 30, and the biological measuring instrument 40 by Bluetooth (registered trademark).

  The terminal 10 is a computer that plays a central role in the information processing system 1. For example, the terminal 10 transmits image data and audio data to the display 20 and the speaker 30, respectively, and receives viewpoint data and biological data from the display 20 and the biological measuring instrument 40, respectively. The display 20 is a device that processes image data received from the terminal 10 and displays an image to the user. Further, the display 20 transmits viewpoint data to the terminal 10. The viewpoint data is data indicating the portion of the image that the user is viewing, that is, the user's gaze point. The speaker 30 is a device that processes audio data received from the terminal 10 and outputs audio to the user. The biometric instrument 40 is a device that detects user biometric data and transmits the biometric data to the terminal 10. Biometric data is data indicating the physical or physiological state of a user. The biological data processed by the information processing system 1 is not limited at all. For example, the physiological data may be data indicating heartbeat, pulse, blood pressure, body temperature, electroencephalogram, sweating, eye movement, respiration, and the like. The biometric instrument 40 detects at least one type of biometric data.

  FIG. 1 shows a specific example of the information processing system 1. In this example, the terminal 10 is a tablet (tablet computer), the display 20 is a head-mounted display, the speaker 30 is a headphone, and the biometric device 40 is a wristwatch-type wearable terminal (smart watch). But the specific kind of each apparatus which comprises the information processing system 1 is not limited at all. For example, the terminal 10 may be a smartphone or a personal computer. The display 20 may be a projector, a television receiver, or a screen. The speaker 30 may be an earphone, a stationary speaker, or a surround system. The biometric instrument 40 may be an electroencephalograph or a glasses-type wearable terminal capable of measuring eye movement.

  FIG. 2 shows a general hardware configuration of the terminal 10 that is the key of the information processing system 1. The terminal 10 includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, a communication control unit 104, an input device 105, and an output device 106. The processor 101 is an electronic component that executes an operating system and application programs. The main storage unit 102 is an electronic component that temporarily stores programs to be executed and data, and includes, for example, a ROM and a RAM. The auxiliary storage unit 103 is an electronic component that permanently stores data to be processed or processed data, and includes a storage device such as a hard disk or a flash memory. The communication control unit 104 is an electronic component that transmits / receives data to / from another device via a wired or wireless connection, and includes, for example, a network card or a wireless communication module. The input device 105 is a device that accepts input from a user, and is, for example, a touch panel, a keyboard, or a mouse. The output device 106 is a device that outputs data designated or processed by the processor 101 in a manner that a person can recognize, and is, for example, a touch panel or a monitor. Of course, the hardware components constituting the terminal 10 may vary depending on the type of computer. For example, if the terminal 10 is a tablet, the touch panel functions as the input device 105 and the output device 106. If the terminal 10 is a stationary personal computer, the input device 105 includes a keyboard and a mouse, and the output device 106 Is a monitor.

  FIG. 3 shows a functional configuration of the information processing system 1 (particularly, a functional configuration of the terminal 10). The terminal 10 includes an image database 11, an audio database 12, an interface display unit 13, an extraction unit 14, a transmission unit 15, a reception unit 16, a data processing unit 17, and a medical record database 18 as functional components. These functional elements are realized by reading predetermined software (an information processing program P1 described later) on the processor 101 or the main storage unit 102 and executing the software. The processor 101 operates the communication control unit 104, the input device 105, or the output device 106 in accordance with the software, and reads and writes data in the main storage unit 102 or the auxiliary storage unit 103. Data or a database necessary for processing is stored in the main storage unit 102 or the auxiliary storage unit 103.

  The image database 11 is a functional element that stores one or a plurality of image data. When a plurality of image data are prepared in the image database 11 in advance, a plurality of options for images can be provided to the user. Image data is data indicating an image that can be processed by a computer. Accordingly, the image is recorded in a storage device such as a memory, and is output to the display 20 by the processing of the processor, so that the image can be viewed.

  The voice database 12 is a functional element that stores one or a plurality of voice data. When a plurality of sound data is prepared in the sound database 12 in advance, a plurality of options for sound can be provided to the user. The sound data is data indicating sound that can be processed by a computer. Therefore, the sound is recorded in a storage device such as a memory and is output to the speaker 30 by the processing of the processor, so that it can be heard. The audio data is data independent of the image data. Therefore, even if one piece of data is processed (for example, reproduced, modified, or deleted) assuming that there is one piece of audio data and one piece of image data, the process does not affect the other piece of data at all.

  In the present embodiment, the image database 11 and the audio database 12 will be described separately. However, a single database may store both image data and audio data.

  The interface display unit 13 is a functional element that displays an interface screen including information necessary for operation of the information processing system 1 and a user interface on the terminal 10. The interface is operated by a user or a person other than the user (for example, a trainer), and these persons are hereinafter referred to as “operators”. Although the configuration of the interface screen is not limited at all, in the present embodiment, the interface display unit 13 displays an interface screen 110 as shown in FIG.

  The interface display unit 13 displays the interface screen 110 in response to input of necessary information such as a user identifier (for example, user ID or name) and tapping or clicking a predetermined confirmation button. Note that the terminal 10 may establish communication with the display 20, the speaker 30, and the biological measuring instrument 40 at the timing when the interface screen 110 is displayed.

  In the present embodiment, the interface screen 110 includes a thumbnail area 111, a reproduction image area 112, a graph 113, a memo field 114, a MEMO button 115, and a ZIP button 116. The thumbnail area 111 is an area for displaying one or more thumbnails indicating one or more selectable image data and audio data. The reproduction image area 112 is an area indicating an image selected and displayed on the display 20 and a user's gazing point with respect to the image. A graph 113 is an area showing the transition of the biometric data in real time. The memo column 114 is an area for displaying a memo input by the operator. The MEMO button 115 is a user interface for displaying a keyboard for entering a memo on the screen. The ZIP button 116 is a user interface for saving medical chart data including biometric data.

  An example of an operation method of the interface screen 110 will be described. The operator selects one image data and one audio data by designating a thumbnail. The selected image data is displayed on the display 20, and the selected audio data is output from the speaker 30. When the image data is reproduced, the situation (the same image or frame as the user is viewing) is displayed in the reproduced image area 112. In addition, the biological data sent from the biological measuring instrument 40 is displayed as a graph 113 in real time. The operator can input a memo at an arbitrary timing. The interface display unit 13 enables the ZIP button 116 at a predetermined timing (for example, when a specific time (20 minutes or the like) has elapsed or when image reproduction has ended). The operator can save medical chart data including biometric data by tapping or clicking the ZIP button 116. The medical record data may further include a memo.

  The configuration of the interface screen may be arbitrarily determined. The interface screen 110 is merely an example, and the thumbnail area 111, the reproduction image area 112, the graph 113, the memo column 114, the MEMO button 115, and the ZIP button 116 are not essential components. The interface screen may include a user interface other than these components.

  The extraction unit 14 is a functional element that extracts image data and audio data selected via the interface screen 110 from the image database 11 and the audio database 12, respectively. The extraction unit 14 outputs the extracted data to the transmission unit 15.

  The transmission unit 15 is a functional element that transmits (outputs) the extracted image data to the display 20 and transmits (outputs) the extracted audio data to the speaker 30. That is, the transmission unit 15 functions as an output unit.

  The display 20 receives the image data and displays the image data to the user without substantially showing the actual scenery to the user. “Without showing the actual scenery” means that the user is more aware of the image displayed on the display than the surrounding scenery (real scenery) that the user can see with his / her own eyes. This means showing the data to the user. If the display 20 is a head-mounted display, the user's field of view is completely covered by the head-mounted display, so that the user can view an image without looking at the actual scenery. By displaying images in this way, it is possible to show the user only the world of virtual reality. As another example, a user may be placed in an enclosed space having a projector and a screen or screen function to project an image on the entire wall, or the entire wall and ceiling, or the entire floor, wall, and ceiling. Good. This case also corresponds to “displaying image data to the user without substantially showing the actual scenery to the user”.

  The speaker 30 receives audio data and outputs it to the user. If the speaker 30 is a headphone, the user can listen to the voice without worrying about ambient noise. However, the user who uses the information processing system 1 may be in a situation where sound or voice other than the sound from the speaker 30 can be heard.

  The receiving unit 16 is a functional element that receives viewpoint data from the display 20 and receives biological data from the biological measuring instrument 40. That is, the receiving unit 16 functions as an acquiring unit. The receiving unit 16 outputs the viewpoint data and biometric data to the data processing unit 17. Acquisition and transmission of viewpoint data and biometric data are repeatedly executed at predetermined intervals, and the receiving unit 16 receives these data and outputs them to the data processing unit 17 each time. This means that the terminal 10 acquires viewpoint data and biological data in real time.

  The data processing unit 17 is a functional element that processes viewpoint data and biometric data. In the present embodiment, the data processing unit 17 executes a process of updating the interface screen 110 based on these data and a process of storing medical chart data including biometric data in the medical chart database 18. Therefore, the data processing unit 17 also functions as a storage unit.

  The data processing unit 17 displays the gazing point in the reproduced image area 112 based on the viewpoint data. When the image displayed on the display 20 is a 360 ° image, the data processing unit 17 changes the angle of the image so that the portion including the gazing point is reflected in the reproduced image area 112. In addition, the data processing unit 17 draws the graph 113 based on the biometric data. The data processing unit 17 updates the display of the gazing point and the graph every time the viewpoint data and the biological data are input from the receiving unit 16. Therefore, the user's viewpoint and biometric data are displayed on the interface screen 110 in real time.

  When the ZIP button 116 is pressed, the data processing unit 17 includes a set of input biological data (a series of biological data obtained from the start to the end of measurement by the biological measuring instrument 40) and a user identifier. Medical chart data is generated, and the medical chart data is stored in the medical chart database 18. When a memo is input via the interface screen 110, the data processing unit 17 may generate and store medical chart data further including the memo. The data processing unit 17 may generate and store medical chart data further including information for specifying the reproduced image and sound (for example, file names of image data and sound data).

  The medical chart database 18 is a functional element that stores medical chart data. There is no limitation on how to use the chart data. For example, the terminal 10 may redisplay the medical chart data on the screen, may send the medical chart data to another computer such as a file server for sharing the medical chart data, or send it to a specific mail address by e-mail. You may send it.

  Note that logically one database may store medical chart data, image data, and audio data.

  Next, the operation of the information processing system 1 (particularly the operation of the terminal 10) will be described with reference to FIG. 5, and the information processing method according to the present embodiment will be described.

  First, the interface display unit 13 displays the interface screen 110 (step S11). When the user operates the interface screen 110 to select one image data and one sound data, the extraction unit 14 extracts the image data and the sound data from the image database 11 and the sound database 12, respectively (step S12, extraction). Step). Then, the transmission unit 15 transmits the image data to the display 20 and transmits the audio data to the speaker 30 (step S13, output step). As a result, the display 20 displays images, the speaker 30 outputs sound, and the user perceives these images and sounds.

  With the transmission of image data and audio data as an opportunity, the display 20 starts transmitting viewpoint data, the biometric instrument 40 starts transmitting biometric data, and the receiving unit 16 of the terminal 10 starts receiving the viewpoint data and biometric data. (Step S14). Then, the data processing unit 17 displays the viewpoint data and biometric data on the interface screen 110 (step S15). This step S15 is repeatedly executed while the receiving unit 16 continues to receive the viewpoint data and the biological data from the display 20 and the biological measuring instrument 40, respectively. Thereafter, in response to a user operation (for example, tap or click on the ZIP button 116), the data processing unit 17 stores the medical record data including the biometric data in the medical record database 18 (step S16).

  Next, an information processing program P1 for realizing the information processing system 1 (terminal 10) will be described with reference to FIG.

  The information processing program P1 includes a main module P10, an image storage module P11, an audio storage module P12, an interface display module P13, an extraction module P14, a transmission module P15, a reception module P16, a data processing module P17, and a medical record storage module P18. The main module P10 is a part that comprehensively controls processing in the terminal 10. The functions realized by executing the image storage module P11, the sound storage module P12, the interface display module P13, the extraction module P14, the transmission module P15, the reception module P16, the data processing module P17, and the medical record storage module P18 are respectively described above. The functions of the image database 11, the audio database 12, the interface display unit 13, the extraction unit 14, the transmission unit 15, the reception unit 16, the data processing unit 17, and the medical record database 18 are the same.

  The information processing program P1 may be provided after being fixedly recorded on a tangible recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the information processing program P1 may be provided via a communication network as a data signal superimposed on a carrier wave.

  As described above, an information processing system according to one aspect of the present invention includes an extraction unit that extracts, from a database, image data for allowing a user to experience virtual reality and audio data independent of the image data. The image data extracted by the extraction unit is output to a display that displays the image data to the user without substantially showing the actual scenery to the user, and the audio data extracted by the extraction unit is output to the speaker. An output unit for outputting.

  An information processing method according to an aspect of the present invention is an information processing method executed by an information processing system including a processor, and is independent of image data for allowing a user to experience virtual reality and the image data. Extracting the audio data from the database; and outputting the image data extracted in the extraction step to a display that displays the image data to the user without substantially showing the actual scenery to the user; An output step of outputting the audio data extracted in the extraction step to a speaker.

  An information processing program according to an aspect of the present invention is extracted in an extraction step of extracting image data for allowing a user to experience virtual reality and audio data independent of the image data from the database, and the extraction step. Outputting the image data to a display that displays the image data to the user without substantially showing the actual scenery to the user, and outputting the audio data extracted in the extraction step to a speaker; Let the computer system run.

  In such an aspect, image data for experiencing virtual reality is provided to the user via the display, and the user views the image without substantially viewing the actual landscape. Furthermore, audio independent of the image is provided to the user. By outputting the image and sound selected for the desired scene to the user in this way, it is possible to effectively convey the sound and sound to the user.

  The information processing system according to another aspect may further include an acquisition unit that acquires the biological data of the user from the biological measuring instrument, and a storage unit that stores the biological data acquired by the acquisition unit in a predetermined database. This mechanism makes it possible to objectively grasp the influence or effect that the image or sound has on the user.

  In an information processing system according to another aspect, voice data may indicate advice for mental training. By using this voice data, it becomes possible to bring the user's mind or behavior into or close to the state desired by the user.

  In the information processing system according to another aspect, the display may be a head mounted display. By using a head-mounted display, it is possible to prevent a user from seeing a real scene more reliably, so that a more complete virtual reality can be provided to the user.

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the gist thereof.

  The configuration of the information processing system is not limited to the above embodiment. For example, a distributed system (client server system) including a plurality of computers may execute the function of the terminal 10. For example, at least a part of the image database 11, the audio database 12, and the medical record database 18 may be mounted on a computer different from the terminal 10, and the terminal 10 may access these databases via a communication network. Alternatively, one computer or device may execute the functions of the terminal 10, the display 20, the speaker 30, and the biometric instrument 40. Alternatively, one device may execute the functions of the display 20 and the speaker 30, or the functions of the display 20, the speaker 30, and the biometric instrument 40.

  The information processing system may not include a biological measuring instrument or a function corresponding to this. Further, the information processing system may not acquire viewpoint data. Therefore, the information processing system may not include the receiving unit 16, the data processing unit 17, and the medical record database 18 in the above embodiment.

  The image data may include sound, and the information processing system may output both the sound data and the sound of the image data from a speaker. In this case, the user can hear these two kinds of sounds.

  The processing procedure of the operation method of the information processing system executed by at least one processor is not limited to the example in the above embodiment. For example, the information processing system (terminal) may omit some of the steps (processing) described above, or may execute the steps in a different order. Also, any two or more of the steps described above may be combined, or a part of the steps may be corrected or deleted. Alternatively, the information processing system (terminal) may execute other steps in addition to the above steps.

  DESCRIPTION OF SYMBOLS 1 ... Information processing system, 10 ... Terminal, 20 ... Display, 30 ... Speaker, 40 ... Biometric instrument, 11 ... Image database, 12 ... Audio database, 13 ... Interface display part, 14 ... Extraction part, 15 ... Transmission part ( (Output unit), 16 ... receiving unit, 17 ... data processing unit, 18 ... medical record database, 110 ... interface screen, P1 ... information processing program, P10 ... main module, P11 ... image storage module, P12 ... audio storage module, P13 ... Interface display module, P14 ... extraction module, P15 ... transmission module, P16 ... reception module, P17 ... data processing module, P18 ... chart storage module.

Claims (6)

An extraction unit for extracting from the database image data for allowing the user to experience virtual reality and audio data independent of the image data;
The image data extracted by the extraction unit is output to a display that displays the image data to the user without substantially showing the actual scenery to the user, and the audio data extracted by the extraction unit is output. An information processing system comprising: an output unit that outputs to a speaker. An acquisition unit for acquiring user biometric data from a biometric instrument;
The information processing system according to claim 1, further comprising: a storage unit that stores the biological data acquired by the acquisition unit in a predetermined database. The voice data indicates mental training advice,
The information processing system according to claim 1 or 2. The display is a head mounted display;
The information processing system according to any one of claims 1 to 3. An information processing method executed by an information processing system including a processor,
An extraction step for extracting from the database image data for allowing the user to experience virtual reality and audio data independent of the image data;
The image data extracted in the extraction step is output to a display that displays the image data to the user without substantially showing the actual scenery to the user, and the audio data extracted in the extraction step is An information processing method including an output step of outputting to a speaker. An extraction step for extracting from the database image data for allowing the user to experience virtual reality and audio data independent of the image data;
The image data extracted in the extraction step is output to a display that displays the image data to the user without substantially showing the actual scenery to the user, and the audio data extracted in the extraction step is An information processing program for causing a computer system to execute an output step of outputting to a speaker.

Images