JP6721727B1 - Information processing apparatus control program, information processing apparatus control method, and information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control program of an information processing device capable of recording an event such as a trouble in a virtual space. A control program of an information processing device connected to a virtual space system in which a user's motion and voice input via a user terminal are reflected in the motion and voice of a user's avatar in a virtual space The apparatus has a position specifying function for generating input information specifying a position in the virtual space, an acquisition function for acquiring image information of the virtual space at the position specified by the position specifying function, and one or more avatars included in the image information. Generates text information that describes information about the avatar's movement based on the analysis function that analyzes the movement of the avatar and the analysis result by the analysis function when the extracted avatar satisfies a predetermined condition. The function and the storage function of storing the text information generated by the generation function in a predetermined storage unit are realized. [Selection diagram]

Description

The present invention relates to a control program for an information processing device, a method for controlling the information processing device, and an information processing device, and more particularly to a control program for the information processing device used in a system that provides a virtual space.

2. Description of the Related Art In recent years, with the spread of head-mounted displays (HMDs), various services for providing virtual spaces using HMDs are expanding. In particular, a service in which a plurality of users in a virtual space communicate with each other via their avatars via a network has become popular (for example, Non-Patent Document 1). In order to realize such a virtual space, a technique has been proposed in which the movement and facial expression of the user in the real world are reflected in the avatar. (For example, patent document 1).

In the virtual space, various troubles may occur between avatars (users) as in the real world. For such a problem, for example, Patent Document 2 proposes a technique for early detection of an unauthorized user or an annoying user in order to manage conversations and chats in a virtual space soundly.

"VR CHAT", [online], [Search on December 5, 2018], Internet, <https://www.vrchat.net/#>

Japanese Patent Laid-Open No. 2016-53133 JP, 2008-92416, A

In Patent Document 2, a user as a whistleblower shoots an avatar that performs a nuisance, and information about the shot image is transmitted to the server as notification data. However, the technique described in Patent Document 2 has the following problems.

First, each user as a whistleblower has different criteria for determining whether or not the act is a nuisance. Therefore, unnecessary notifications may be transmitted, or conversely, true nuisance may not be reported. It is also possible for the administrator of the virtual space to provide the virtual space as a place where fraudulent acts such as exchanging information relating to criminal acts can be performed. In this case, even if the user reports, it is unlikely that the administrator will take appropriate measures.

It is also possible to record the operation history of each user so that the trouble that occurred in the virtual space can be confirmed later. However, it is considered that the operation information of each user for each avatar includes operation information for moving the avatar, voice, and, in the future, information such as tactile sense. It is considered that such information will be enormous due to the number of users, and it is not realistic to store such information.

Therefore, the present invention provides a control program for an information processing apparatus, which can solve the above problems and record an event such as a trouble in a virtual space.

Control of an information processing apparatus connected to a virtual space system in which a user's action and voice input via a user terminal are reflected in the action and voice of a user's avatar in a virtual space according to an embodiment of the present invention. A program, in the information processing device, a position input function that generates input information that specifies a position in the virtual space, an acquisition function that acquires image information of the virtual space at the position specified by the position input function, and an image. Extracts one or more avatars included in the information, and if the extracted avatars meet a predetermined condition, an analysis function that analyzes the avatar's movements and information about the avatar's movements based on the results of the analysis by the analysis function. A generation function of generating text information to be described and a storage function of storing the text information generated by the generation function in a predetermined storage unit are realized.

In the control program according to the embodiment of the present invention, the acquisition function acquires the voice information together with the image information of the virtual space, the analysis function extracts the voice information of the avatar together with the avatar, and the generation function performs the predetermined condition. The voice information may be further used for the determination.

A control program according to an embodiment of the present invention further causes an information processing apparatus to realize an association function of associating annotation information, which is information specifying an avatar's action in a virtual space, and the generation function extracts as a predetermined condition. When the annotation information associated with the created avatar includes a predetermined keyword, text information may be generated.

In the control program according to the embodiment of the present invention, the position input function generates input information designating a plurality of positions within a predetermined distance from the avatar when the avatar extracted by the analysis function satisfies a predetermined condition. However, the acquisition function may acquire image information about the avatar at a plurality of positions.

In the control program according to the embodiment of the present invention, the associating function may update the annotation information to be associated with the avatar based on the result of learning the series of actions of the avatar.

In the control program according to the embodiment of the present invention, the storage function may use a digital watermark or a distributed ledger technology to store the text information generated by the generation function.

In the control program according to the embodiment of the present invention, the storage function further stores registration information registered in the virtual space system as one avatar, and the position input function inputs the position of one avatar in the virtual space system. However, the acquisition function may acquire the image information of the virtual space as the field of view of one avatar.

In the control program according to the embodiment of the present invention, one avatar may be realized as a transparent object that makes a background transparent in the virtual space.

Control of an information processing apparatus connected to a virtual space system in which a user's action and voice input via a user terminal are reflected in the action and voice of a user's avatar in a virtual space according to an embodiment of the present invention. The method is such that the information processing apparatus generates input information designating a position in the virtual space, obtaining image information of the virtual space at the designated position, and one or more avatars included in the image information. If the avatar extracted in the extracting step satisfies a predetermined condition, text information describing information about the avatar's movement is analyzed based on the analysis step of analyzing the avatar's movement and the analysis step. The generating step and the step of storing the text information generated in the generating step in a predetermined storage unit are executed.

An information processing apparatus according to an embodiment of the present invention, which is connected to a virtual space system in which a user's action and voice input via a user terminal are reflected in the action and voice of a user's avatar in a virtual space, A position input unit that generates input information that specifies a position in the virtual space, an acquisition unit that acquires image information of the virtual space at the position specified by the position input unit, and one or more avatars included in the image information are extracted. If the avatar extracted by the extraction unit satisfies a predetermined condition, an analysis unit that analyzes the motion of the avatar, and generation of text information that describes information about the motion of the avatar based on the analysis result by the analysis unit And a storage control unit that stores the text information generated by the generation unit in a predetermined storage unit.

According to the present invention, it is possible to provide a control program of an information processing apparatus for recording an event such as a trouble in a virtual space.

1 is a schematic diagram of a virtual space recording system configuration according to an embodiment of the present invention. 1 is an example of a hardware configuration of a recording server (information processing device), a computer, and an HMD according to an embodiment of the present invention. It is an example of a software configuration of a computer according to an embodiment of the present invention. 1 is an example of a software configuration of a recording server (information processing device) according to an embodiment of the present invention. (A) is a schematic diagram illustrating a virtual space recording system according to an embodiment of the present invention, and (b) is an example of information stored in the virtual space recording system according to an embodiment of the present invention. It is an example of a sequence according to an embodiment of the present invention. 3 is an example of annotation information generated in the virtual space recording system according to the embodiment of the present invention. It is a schematic diagram explaining the virtual space recording system concerning other embodiments of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

<System configuration>
FIG. 1 is a diagram showing a configuration example of a virtual space recording system according to an embodiment of the present invention. As shown in FIG. 1, a virtual space recording system 700 includes a recording server (information processing device) 100, a virtual space providing server 200, and HMD systems 500A to 500C of respective users, which are connected to each other via a network 600. including. The recording server 100 records an event that occurs in the virtual space provided by the virtual space providing server 200 (details will be described later).

The virtual space providing server 200 is a server on the manager side of a virtual space providing service (hereinafter referred to as “virtual space service”), and provides the virtual space service to the HMD systems 500A to 500C of users A to C, respectively. To do. The HMD systems 500A to 500C are systems for users to use virtual space services, and include HMDs 400A to 400C and computers 300A to 300C as user terminals, respectively. It should be noted that although FIG. 1 shows three users, user A, user B, and user C, the number of users is not limited to this, and may be more or less. Further, the HMDs 400A to 400C do not have to be of the same type, and the types thereof do not matter as long as the virtual space service provided by the virtual space providing server 200 can be used.

Note that, as described above, the HMD system 500, the computer 300, and the like corresponding to the user X are expressed as the HMD system X, the computer 300X, and the like, and the user information in the virtual space service is expressed as the user information X. Here, the user information is user information associated with an account used by the user in the virtual space service. The user information is, for example, the user's name, the user's avatar information, the user's age, the user's sex, the user's address, the user's hobby, or the telephone number, which is input by the user or given by the virtual space service. The information may be information associated with the user, such as the user's contact information such as a mail address, a mail address, or the user's identifier (user ID (IDentifier)).

In the following description, the HMD systems 500A to 500C will be collectively referred to as the HMD system 500, the HMDs 400A to 400C will be collectively referred to as the HMD 400, and the computers 300A to 300C will be collectively referred to as the computer 300, unless otherwise specified.

The network 600 has a role of connecting the virtual space providing server 200, the recording server 100, and each computer 300. The network 600 means a communication network that provides a connection path so that data can be transmitted and received between them. The network 600 may include a wireless network or a wired network. Specifically, the network 600 includes a wireless LAN (WLAN), a wide area network (WAN), ISDNs (integrated service digital networks), wireless LANs, LTE (long term evolution), LTE-Advanced, The fourth generation (4G), the fifth generation (5G), CDMA (code division multiple access), and the like. The network 600 is not limited to these examples. For example, the public switched telephone network (PSTN), Bluetooth (registered trademark), optical line, ADSL (Asymmetric Digital Subscriber LINE) line, satellite. It may be a communication network or the like. Further, the network 600 may be a combination of these.

The virtual space providing server 200 may be typically realized on the cloud. The virtual space providing server 200 has a function as an application server and executes various processes in response to a request from an application installed in the computer 300.

In the computer 300, the virtual space, the avatars of the user of the computer 300 and other users, the output of voice, and the like are performed based on the application program. The computer 300 transmits, to the HMD 400, information for providing a virtual space to a user wearing the HMD 400. The information for providing the virtual space includes display information for displaying the virtual space, audio information in the virtual space, and tactile information in the future. The moving image information may be included. The computer 300 also acquires from the HMD 400 motion information regarding the motion of the user wearing the HMD 400 and voice information regarding the voice generated by the user (hereinafter referred to as “motion information etc.”), and according to the acquired motion information. Change the display of the virtual space. In addition, the computer 300 operates the avatars of other users in the virtual space according to the operation information of the other users transmitted from the virtual space providing server 200 via the network 600.

The HMD 400 is mounted on the user's head. The HMD 400 is connected to the computer 300 to provide a virtual space to the user. The HMD 400 may include both a so-called head mounted display including a monitor and a head mounted device to which a terminal having a monitor such as a smartphone can be mounted. It should be noted that although the drawings and the present disclosure describe the mode in which the application of the virtual space is installed in the computer 300, the present invention is not limited to this, and the application is installed in the HMD 400, and the HMD 400 is installed in the network 600. It may be directly connected to provide the user with a virtual space.

It should be noted that the recording server 100, the virtual space providing server 200, the computer 300, and the HMD 400 are any information processing devices as long as they can realize the functions and/or methods described in the present disclosure. May be. For example, examples of these information processing devices include smartphones, mobile phones (feature phones), computers (for example, desktops, laptops, tablets, etc.), server devices, media computer platforms (for example, cables, satellite set-top boxes, digital video). A recorder, etc.), a handheld computer device (eg PDA (Personal Digital Assistant), e-mail client etc.), a wearable terminal (eg glasses-type device, clock-type device etc.), another kind of computer or a communication platform. ..

<Hardware configuration>
The hardware configurations of the recording server 100, the computer 300, and the HMD 400 included in the virtual space recording system 700 will be described with reference to FIG.

(1) Hardware Configuration of HMD System 500 In one embodiment of the present invention, an existing virtual space providing service can be used, and the virtual space providing server 200 and the HMD system 500 may be existing technologies. Therefore, detailed description of the virtual space providing service and the virtual space providing server 200 is omitted. The virtual space providing service is not limited to the description of the present disclosure.

The computer 300 includes a processor 301, a storage 302, an input/output interface (input/output I/F) 303, a memory 304, and a communication interface (communication I/F) 305. The respective constituent elements are connected to each other via a bus B.

The processor 301 executes a function or method realized by a code or instruction included in a program stored in the storage 302. The memory 304 temporarily stores the program loaded from the storage 302 and provides a work area to the processor 301. Various data generated while the processor 301 is executing the program is also temporarily stored in the memory 304. The storage 302 stores a program.

The communication I/F 305 executes communication with the virtual space providing server 200 and the recording server 100, which are other information processing devices, via the network 600. The communication I/F 305 transmits various data to another information processing device according to an instruction from the processor 301. Further, the communication I/F 305 receives various data transmitted from another information processing device and transfers the data to the processor 301.

The input/output I/F 303 is a connection interface with the HMD 400 and performs data communication with the HMD 400. The input/output I/F 303 includes, for example, USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), and the like. Data communication between the HMD 400 and the computer 300 may be performed via wireless communication (near field communication (Bluetooth, Wi-Fi, etc.)) or wired communication. Further, the input/output I/F 303 may be connected to another sensor (not shown) provided separately from the HMD 400, such as an infrared sensor or a camera. The camera may be provided so as to detect the facial expression of the user, and the change in the facial expression of the user may be reflected in the change in the facial expression of the avatar.

The HMD 400 includes a gyro sensor 410, a line-of-sight sensor 420, a camera 430, a microphone 440, a monitor 450, and a speaker 460. The monitor 450 is realized as, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor, and displays a three-dimensional virtual space. More specifically, the image for the right eye and the image for the left eye are displayed on the monitor 450 so that the user can recognize the three-dimensional image based on the parallax of both eyes. Although the monitor 450 is described here, retinal projection may be used.

The gyro sensor 410 detects the inclination of the HMD 400. The HMD 400 may include an acceleration sensor, a geomagnetic sensor, and other sensors in addition to the gyro sensor 410, and can detect the movement (rotation, inclination, orientation, acceleration, etc.) of the head of the user wearing the HMD 400. .. Further, the computer 300 may be configured to detect the position and the inclination of the HMD 400 based on the output of the sensor included in the HMD 400 or the above-described separate sensor.

The camera 430 is configured to be able to acquire depth information from the camera 430 to an object, and acquires data for displaying in a three-dimensional manner an object in the line-of-sight direction of the user wearing the HMD 400 (for example, the user's hand). .. For example, the camera 430 may be a TOF (Time Of Flight) camera, a stereo camera, an infrared camera, or the like. The camera 430 outputs the acquired depth information of the target object to the computer 300.

The speaker 460 converts the audio signal received from the computer 300 into audio and outputs the audio to the user. The microphone 440 converts the user's utterance into an electric signal and outputs the electric signal to the computer 300. The line-of-sight sensor 420 detects a direction (line of sight) in which the right and left eyes of the user are directed.

(2) Hardware Configuration of Recording Server 100 The recording server 100 includes a processor 101, a storage 102, an input/output interface (input/output I/F) 103, a memory 104, and a communication interface (communication I/F) 105. including. The respective constituent elements are connected to each other via a bus B.

The recording server 100 realizes the functions and methods described in the present embodiment by the cooperation of the processor 101, the storage 102, the input/output I/F 103, the memory 104, and the communication I/F 105. For example, the functions or methods of the present disclosure are realized by the processor 101 executing instructions included in the program read into the memory 104.

The processor 101 executes a function and/or a method realized by a code or an instruction included in a program stored in the storage 102. The processor 101 is, for example, a central processing unit (CPU), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), microprocessor (microprocessor), processor core (processor core), multiprocessor (multiprocessor), ASIC (Application- Specific integrated circuit), FPGA (Field Programmable Gate Array), etc., and implemented by a logic circuit (hardware) or dedicated circuit formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)) or the like. Each process disclosed in the embodiment may be realized. Further, these circuits may be realized by one or a plurality of integrated circuits, and the plurality of processes shown in each embodiment may be realized by a single integrated circuit. Further, the LSI may be referred to as VLSI, super LSI, ultra LSI, or the like depending on the degree of integration.

The memory 104 temporarily stores the program loaded from the storage 102 and provides a work area to the processor 101. The memory 104 also temporarily stores various data generated while the processor 101 is executing the program. The memory 104 includes, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like.

The storage 102 stores a program. The storage 102 includes, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, and the like.

The communication I/F 105 is implemented as hardware such as a network adapter, communication software, and a combination thereof, and transmits/receives various data via the network 600. The communication may be performed by wire or wireless, and any communication protocol may be used as long as mutual communication can be performed. The communication I/F 105 executes communication with the virtual space providing server 200 and the computer 300, which are other information processing devices, via the network 600. The communication I/F 105 transmits various data to another information processing device according to an instruction from the processor 101. Further, the communication I/F 105 receives various data transmitted from another information processing device and transfers the data to the processor 101.

The input/output I/F 103 includes an input device that inputs various operations on the recording server 100 and an output device that outputs a processing result processed by the recording server 100. In the input/output I/F 103, the input device and the output device may be integrated, or the input device and the output device may be separated. The input device is realized by any of all types of devices that can receive an input from a user and transmit information related to the input to the processor 101, or a combination thereof. The input device includes, for example, a hardware key such as a touch panel, a touch display, and a keyboard, a pointing device such as a mouse, a camera (operation input via an image), and a microphone (operation input by voice). The output device outputs the processing result processed by the processor 101. The output device includes, for example, a touch panel, a speaker, and the like.

<Functional configuration>
Next, the functional configurations of the computer 300 and the recording server 100 will be described with reference to FIGS. 3 and 4.

(1) Functional Configuration of Computer 300 FIG. 3 shows a schematic diagram of the functional configuration of the computer 300. As shown in FIG. 3, the computer 300 includes a display control unit 320 and a virtual space control unit 330 as the virtual space application 310. Further, the computer 300 includes a communication control unit 340, a storage unit 350, and an input/output control unit 360.

The communication control unit 340 controls communication with another information processing device via the communication I/F 305. That is, the communication control unit 340 transmits the data acquired through the communication with the virtual space providing server 200, the recording server 100, and the other computer 300, which are other information processing devices, to the virtual space application 310 and the storage unit 350. Or transmit data from the virtual space application 310 or the storage unit 350 to another information processing device.

The input/output control unit 360 controls communication with the HMD 400, for example, via the input/output I/F 303. That is, the input/output control unit 360 transmits the user's motion information and voice information transmitted from the HMD 400 to the virtual space application 310 and the storage unit 350, and the virtual space display information transmitted from the virtual space application 310. The voice information is transmitted to the HMD 400.

The display control unit 320 controls an image displayed on the monitor 450 of the HMD 400 in the virtual space. The display control unit 320 includes a position input unit 321, a viewpoint input unit 322, a visual field image generation unit 323, a motion detection unit 324, an angular velocity detection unit 325, and a visual line detection unit 326.

The position input unit 321 sets the position in the virtual space according to the user's motion information transmitted from the HMD 400. That is, the position input unit 321 sets the moving destination and the moving distance in the virtual space according to the walking motion or the running motion of the user. The operation information of the user is not limited to that transmitted from the HMD 400, and may be transmitted from a controller (not shown). A known controller can be used as the controller. Alternatively, the position input unit 321 may set the position in the virtual space based on the destination in the virtual space directly input by the user. The viewpoint input unit 322 sets the visual field area of the virtual space of the user according to the inclination of the HMD 400 (that is, the output of the gyro sensor 410) and the position set by the position input unit 321. The visual field image generation unit 323 generates a visual field image of the virtual space displayed on the monitor 450, based on the set visual field area.

The motion detection unit 324 detects the motion of the user transmitted from the HMD 400. The detection unit 326 detects the inclination of the HMD 400 as the line of sight of the user based on the outputs from the gyro sensor 410 and other sensors (not shown). The angular velocity detection unit 325 converts the output of the gyro sensor 410 into an angular velocity, and outputs the converted angular velocity to the viewpoint input unit 322 and the line-of-sight detection unit 326.

The virtual space control unit 330 controls the virtual space provided to the user. The virtual space control unit 330 includes a virtual space generation unit 331, a virtual object generation unit 332, and an avatar generation unit 333.

The virtual space generation unit 331 generates virtual space data representing a virtual space and outputs it to the HMD 400. The virtual object generation unit 332 generates an object arranged in the virtual space. The objects may include, for example, artificial objects such as urban areas, roads, and shops arranged according to the theme or story set in the virtual space, landscapes including forests, mountains, etc., and objects simulating natural objects such as animals. The virtual object generation unit 332 may generate an object that can be operated by the user in the virtual space. The operable object may be, for example, a user's hand (user's avatar's hand) or another user's avatar's hand. Further, the virtual object generation unit 332 may display an arbitrary object generated based on the program by the registered user of the virtual space service. The objects generated (displayed) in the virtual space are not limited to those described above, and various objects such as virtual currency, characters, still images, and moving images can be included.

The avatar generation unit 333 generates data of an avatar placed in the virtual space. The avatar placed in the virtual space may be an avatar object of another user of another computer 300 connected via the network 600, or an avatar object of the user. Note that the avatar object may include various objects set by the user, such as a person, an animal, a robot, a character, and the like.

The avatar generation unit 333 reflects the movement of the HMD 400 on the avatar object of the user based on the angular velocity output by the angular velocity detection unit 325. In addition, the avatar generation unit 333 reflects the user's motion including the motion of the user's hand in the physical space on the avatar object based on the output of the motion detection unit 324. Also, the avatar generation unit 333 changes the facial expression of the avatar according to the change of the facial expression of the user acquired by the camera and the content of the voice/speech of the user. This can be performed by a known three-dimensional image generation technique or a technique of estimating a facial expression by providing a myoelectric sensor in a portion of the HMD that comes into contact with the face of the user. In addition, the avatar generation unit 333 moves and displays the avatar object corresponding to another user of the other computer 300 based on the angular velocity information input from the other computer 300.

The storage unit 350 holds data used by the computer 300 to provide a virtual space to a user. The storage unit 350 includes virtual space information 351 and user information 352. The virtual space information 351 includes one or more templates defined for providing a virtual space, the above-mentioned objects, and the like.

The user information 352 holds a program for causing the computer 300 to function as a control device of the HMD system 500, an application program of a virtual space service, contents available in the virtual space service, and the like. These programs and contents may be acquired by downloading a data on a cloud or a computer of a provider of the virtual space service (for example, the virtual space providing server 200), a computer of the provider of the content. ..

Note that a part of the virtual space application 310 can be realized by using Unity (registered trademark) provided by Unity Technologies, Inc., for example. Alternatively, the virtual space application 310 may be realized as a combination of circuit elements that realize each process.

(2) Functional Configuration of Recording Server 100 FIG. 4 shows a schematic diagram of the functional configuration of the recording server 100. As shown in FIG. 4, the recording server 100 includes a display control unit 120 and a virtual space control unit 130 as the virtual space application 110. Furthermore, the recording server 100 includes a communication control unit 140, a storage unit 150, an operation input information generation unit 160, an image recognition unit 170, a text generation unit 181, and an association unit 182.

Here, prior to the description of each functional unit of the recording server 100, the difference in the functional configuration between the computer 300 and the recording server 100 will be described.

Like the computer 300, the recording server 100 is installed with the application program of the virtual space service, and acquires the data regarding the virtual space based on the program. Therefore, there is no functional difference between the virtual space application 310 in the computer 300 and the virtual space application 110 in the recording server 100. That is, in implementing the virtual space recording system according to the embodiment of the present invention, it is not necessary to impose any special measures on the administrator side of the virtual space service.

In the recording server 100 and the computer 300, the HMD 400 is not connected to the recording server 100, and in the recording server 100, the input information for designating the place (position) and the direction of recording the virtual space is the operation input information generation unit. There is one difference in the points generated by 160. Therefore, in FIG. 4, the display control unit 120 of the virtual space application 110 does not show the motion detection unit, the angular velocity detection unit, and the line-of-sight detection unit regarding the HMD 400. However, in the embodiment of the present invention, when the application program of the virtual space service is used, these motion detection unit, angular velocity detection unit, and line-of-sight detection unit may be included as functions.

As described above, the virtual space application 110 is the same as the virtual space application 310, and therefore, the description of the same parts will be omitted and different parts will be described below.

The operation input information generation unit 160 generates information generated by the HMD 400 in the HMD system 500. That is, the position input information generation unit 161 generates input information that specifies (sets) the position in the virtual space. The viewpoint input information generation unit 162 generates input information that specifies the field of view of the virtual space. The input information generated by the operation input information generation unit 160 may be generated in a data format according to the data format transmitted from the HMD 400. Alternatively, the origin may be set and the relative position from the origin may be designated by a numerical value, or the coordinates may be designated directly. Further, the information generated by the viewpoint input information generation unit 162 includes a rotation angle from the current position, a rotation direction, and the like.

The image recognition unit 170 processes the image data regarding the virtual space generated by the virtual space application 110. The image recognition unit 170 includes a visual field image acquisition unit 171 and an analysis unit 172. The visual field image acquisition unit 171 acquires information (image information) on the visual field image generated by the visual field image generation unit 123 in the virtual space application 110. The analysis unit 172 extracts one or more avatars in the image acquired by the view image acquisition unit 171. Extraction of the avatar in the analysis unit 172 can be performed using an existing image analysis technique.

The analysis unit 172 determines whether the extracted avatar satisfies a predetermined condition. Although the details will be described later, the predetermined condition is a condition with which it is possible to determine the occurrence of a trouble in the virtual space or a state that may lead to the occurrence of the trouble. When it is determined that the extracted avatar satisfies a predetermined condition, the analysis unit 172 further analyzes the motion of the avatar. In the present disclosure, analysis means obtaining statistical data to be analyzed, and analysis of avatar's movement refers to data about how the avatar's movement has changed over time. Means to get. The analysis of the avatar's motion and the analysis of the avatar's facial expression by the analysis unit 172 can also use the existing image analysis technology. In addition, artificial intelligence (AI: Artifficial Intelligence), for example, an analysis technique by deep learning may be used for the analysis unit 172. The text generation unit 181 generates text information that describes information regarding an avatar's action based on the analysis result by the analysis unit 172.

The communication control unit 140 controls communication with another information processing device via the communication I/F 105. That is, the communication control unit 140 transmits data acquired by communication with the virtual space providing server 200, which is another information processing device, and the plurality of computers 300 to the virtual space application 110 and the storage unit 150, and the virtual space application 110 and the storage unit 150. Data from the spatial application 110 and the storage unit 150 is transmitted to another information processing device.

The storage unit 150 includes virtual space information 151, user information 152, and a condition database (DB) 153. The virtual space information 151 includes one or more templates defined for providing a virtual space, the above-mentioned objects, and the like. The condition database 153 holds the above-mentioned predetermined conditions. Furthermore, the storage unit 150 holds the text information generated by the text generation unit 181.

<Recording process in virtual space>
Next, the outline of the recording server 100 recording an event in the virtual space will be described with reference to FIG. FIG. 5A is a diagram in which the virtual space 10 is represented by a hemisphere. In the figure, the virtual space 10 is shown to include avatars 20A and 20B.

In the figure, the avatar 30U corresponds to one account of the recording server 100. As shown in the figure, according to the embodiment of the present invention, the recording server 100 has an account in the virtual space service, and the avatar 30U can be arranged in the virtual space in association with the account. Then, the operation input information generation unit 160 can generate information for moving the avatar 30 and input information for moving the viewpoint of the avatar 30, and transmit the information to the virtual space application 110. This makes it possible to acquire a view field image in the virtual space.

At this time, the avatar 30U may be realized as a transparent object that allows the background to be transparent. In addition, the avatar 30U does not need to be bound by the movement rule of a normal avatar in the virtual space, and may be, for example, capable of acquiring a bird's-eye view image from above while floating in the space. That is, the avatar 30U of the recording server 100 according to the present embodiment can, so to speak, acquire image information in the virtual space as a transparent surveillance camera. As a result, it becomes possible for the user of the virtual space to perform a natural operation without detecting the recording system.

In FIG. 5A, a field surrounded by a dotted line extending from the avatar 30U represents a virtual view area of the avatar 30U. In the figure, the field of view of the avatar 30U includes the avatars 20A and 20B, and the field-of-view image acquisition unit 171 can acquire image information including the avatars 20A and 20B. The analysis unit 172 extracts the avatars 20A and 20B and determines whether these avatars satisfy a predetermined condition.

The predetermined condition is a state that may lead to a trouble in the virtual space, and is, for example, “approach between users”, “interaction with gesture”, “face expression is classified as negative emotion”, “ It may be a condition such as “delivery of an object whose delivery is prohibited” or a combination thereof. The analysis unit 172 analyzes the image including the avatar by a known image analysis technique, and determines that the predetermined condition is satisfied when the avatar has an action that meets the predetermined condition.

For example, in FIG. 5, it is assumed that the avatar 20A and the avatar 20B approach each other and satisfy a predetermined condition. Therefore, the analysis unit 172 starts analysis of the motions of the avatar 20A and the avatar 20B. The text generation unit 181 generates text information describing the action of the avatar based on the analysis result of the analysis unit 172. At that time, the text generation unit 181 may generate text information including the start time and end time of the operation, and information about the place in the virtual space. FIG. 5B shows an example of the generated text information. As shown in the figure, the content of the event is recorded together with the date and the location (location ID) of the event.

As described above, according to the embodiment of the present invention, the action of the avatar can be analyzed from the image and described as text information. At this time, since text information is generated only for avatars that satisfy a predetermined condition, it is possible to make uniform the determination of fraudulent behavior that may differ for each user of the virtual space service, and also to burden the user. There is no. Furthermore, since all the events that occur in the virtual space are not recorded, it is possible to reduce the amount of recorded data. Further, since the application provided by the virtual space service can be used to implement the embodiment of the present invention, the administrator of the virtual space service is not burdened. Moreover, since the recording of the event in the virtual space is performed by analyzing the visual field image and it is not necessary to acquire the three-dimensional data of each avatar, it is not necessary to acquire a huge amount of data that reflects the motion of the avatar.

Note that the voice information of the avatar may be further used to determine the predetermined condition. The visual field image acquisition unit 171 can acquire the audio information from the virtual space application 110 together with the image information of the virtual space. The analysis unit 172 extracts the avatar and the voice information of the avatar from the image information, and determines whether the extracted voice information satisfies a predetermined condition. The predetermined condition relating to the voice information can be, for example, “busy word”, “argument”, “use of secret word”, or the like. Note that the determination may be made by combining a predetermined condition regarding voice information and a predetermined condition regarding operation.

According to the embodiment of the present invention, it is possible to appropriately record an event that may cause a trouble because it is determined whether or not the event is recorded in the virtual space using the audio information.

Here, a recording system according to an embodiment of the present invention will be described with reference to FIG. FIG. 6 is a sequence diagram between the recording server 100, the HMD system 500A, and the virtual space providing server 200 in the recording process. Although the HMD system 500A of the user A is shown as the HMD system 500, the other HMD systems 500B and 500C are similarly connected to the network 600, and the actions of the user B and the user C are reflected in each avatar in the virtual space. Note that this is done.

First, in the HMD system 500A, the HMD 400A acquires the operation information and the voice information of the user A and transmits them to the computer 300A (step S21). The computer 300A transmits the display information and the voice information for reflecting the operation information and the voice information of the user A on the avatar A to the HMD 400A (step S22). In the recording server 100, the position in the virtual space is specified according to the input information generated by the position input information generation unit 161, and the viewpoint (visual field area) is specified according to the input information generated by the viewpoint input information generation unit 162. (Step S26). The visual field image acquisition unit 171 of the recording server 100 acquires the visual field image generated by the virtual space application 110, and the analysis unit 172 extracts one or more avatars included in the visual field image (step S27). On the HMD 400A, the action of the avatar A is displayed and the voice is reproduced (step S25).

The virtual space providing server 200 transmits the motion information and the voice information of the avatar A to the recording server 100 (step S23). The visual field image generation unit of the recording server 100 generates a visual field image that reflects the motion of the avatar A. The field-of-view image acquisition unit 171 acquires information about the field-of-view image, that is, image information about the field-of-view image that displays the action of the avatar A and reproduces sound (if any), and the analysis unit 172 causes the avatar included in the image information. A is extracted (step S27). Then, the analysis unit 172 determines whether the avatar A satisfies a predetermined condition (step S28). When the predetermined condition is not satisfied (NO in step S28), the process returns to step S26 and the position in the virtual space is changed or the process ends. If the predetermined condition is satisfied (YES in step S28), the analysis unit 172 analyzes a series of motions of the avatar A, and the text generation unit 181 generates text information that describes information related to the motion of the avatar A (step). S29).

Note that in the sequence of FIG. 6, the order of steps is not strict, and the user's motion information transmitted from each HMD system 500 is reflected on the avatar in the virtual space, and the avatar in the virtual space is recorded by the recording server 100. Can perform the analysis in real time.

In addition, the recording server 100 may further include an associating unit 182 that associates annotation information that is information that identifies the action of the avatar in the virtual space. Annotation information refers to information that describes an operation or state. According to one embodiment of the present invention, the action of each avatar is learned based on the image information of the virtual space, and the action of the avatar is predicted based on the regularity of the action of the avatar. For example, the avatar's facial expression and the accompanying actions (please enjoy with the expression that raises the corner of the mouth, get angry with the expression that lowers the corner of the mouth, be happy with the expression that narrows your eyes, be surprised with the expression that opens your eyes, be pleased with the movement of raising your hands, and shrug your shoulders And get angry after screaming, etc.) and associate the annotation information for each facial expression, each action, and each voice, and store them in the user information 152 of the storage unit 150.

FIG. 7 shows an example of avatar information relating to the association of annotation information. In the table TB1, annotation information of “glazing” and “anger” is associated with a staring facial expression. In the table TB2, the action of the user when the voice is uttered is associated with the voice information as annotation information. These tables are preferably generated for each user, that is, for each avatar.

The text generation unit 181 may generate text information when the annotation information associated with the extracted avatar includes a prescribed keyword as the prescribed condition. The predetermined keyword is a keyword that represents the occurrence of a trouble or a state that may lead to the occurrence of the trouble, and is a keyword having a generally negative meaning such as "quarrel", "anger", "crying", and the like. For example, when the annotation information associated with a certain facial expression includes the keyword “anger” by image analysis, the analysis unit 172 can determine that the avatar satisfies a predetermined condition.

The facial expression, voice, etc. associated with a certain action vary depending on the user, that is, the avatar. However, as described above, according to one embodiment of the present invention, since annotation information that identifies the action of the avatar is associated with each avatar, it is possible to appropriately determine whether the action of each user may lead to a trouble. It becomes possible to judge.

The associating unit 182 may update the annotation information associated with the avatar, based on the result of learning the series of actions of the avatar. This enables more accurate determination.

Next, another embodiment of the present invention will be described with reference to FIG. FIG. 8 is a schematic diagram of the virtual space 10. The position input information generation unit 161 may specify a plurality of positions within a predetermined distance from the avatar when the avatar extracted by the analysis unit 172 satisfies a predetermined condition. For example, as shown in FIG. 8, it is assumed that the avatar 500C is tilted in the virtual space 10 and the analysis unit 172 determines that the avatar 500C satisfies a predetermined condition. In this case, the position input information generation unit 161 generates input information designating a plurality of positions within a predetermined distance from the avatar 500C. This corresponds to input information for arranging the avatars 30X and 30Y in addition to the avatar 30U around the avatar 500C, for example. This can be realized by registering accounts for the avatars 30X and 30Y in the virtual space service in advance and causing the position input information generation unit 161 to generate input information for the plurality of accounts. Then, the viewpoint input information generation unit 162 generates input information for setting the viewpoint so that the avatar 500C enters the visual field, and the visual field image acquisition unit 171 acquires image information corresponding to the avatars 30U, 30X, 30Y. May be.

According to the above configuration, it is possible to generate text information that describes the action of the avatar determined to satisfy the predetermined condition in more detail.

The storage unit 150 may use a digital watermark or a distributed ledger technology to store the text information generated by the text generation unit 181. As a result, the security and confidentiality of the text information as the analysis result can be ensured, and the possibility of data tampering by the administrator of the virtual space service can be reduced.

The text information generated by the text generation unit 181 may be transmitted as evidence to a predetermined institution, if necessary.

Note that real-time analysis processing is preferable, but it is also conceivable that the load increases due to image analysis processing occurring in parallel or a long recording period. In this case, the moving image data may be stored for a predetermined time (for example, 10 minutes), and the stored moving image data may be analyzed when the load is low without another analysis process. In this case, although the storage capacity will be temporarily squeezed, the moving image data may be deleted as soon as the analysis process is completed. If the avatar satisfies a predetermined condition, the image information as evidence may be stored together with the text information.

Although the present invention has been described based on the drawings and the embodiments, it should be noted that those skilled in the art can easily make various variations and modifications based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the functions and the like included in each constituent unit and each step can be rearranged so as not to logically contradict, and a plurality of constituent units and steps can be combined or divided into one. Is. In addition, the configurations described in the above embodiments may be combined as appropriate. For example, each component described as being included in the recording server 100 may be distributed and realized by a plurality of servers.

The program of each embodiment of the present disclosure may be provided in a state of being stored in a storage medium readable by the information processing device. The storage medium can store the program in a “non-transitory tangible medium”. The programs include, for example, software programs and information processing device programs.

The storage medium is, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (eg, field programmable gate array (FPGA), application specific IC (ASIC), etc.), hard Disk drive (HDD), hybrid hard drive (HHD), optical disk, optical disk drive (ODD), magneto-optical disk, magneto-optical drive, floppy diskette, floppy disk drive (FDD), magnetic tape, solid-state drive (SSD), RAM drive, secure digital card or drive, any other suitable storage medium, or any suitable combination of two or more thereof. Storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Further, the program of the present disclosure may be provided to the recording server 100 via any transmission medium (communication network, broadcast wave, or the like) capable of transmitting the program.

Each embodiment of the present disclosure can also be realized in the form of a data signal embedded in a carrier wave in which a program is embodied by electronic transmission.

The program of the present disclosure is implemented using, for example, a script language such as Java Script (registered trademark), Python, C language, Go language, Swift, Koltin, Java (registered trademark), or the like.

100 recording server (information processing device)
200 virtual space providing server 300 (300A to 300C) computer 400 network 500 (500A to 500C) HMD system 600 network 700 virtual space recording system 301 processor 302 storage 304 memory 410 gyro sensor 420 line-of-sight sensor 430 camera 440 microphone 450 monitor 460 speaker 101 Processor 102 Storage 104 Memory 310 Virtual space application 320 Display control unit 321 Position input unit 322 Viewpoint input unit 323 Visual field image generation unit 324 Detection unit 325 Angular velocity detection unit 326 Line of sight detection unit 330 Virtual space control unit 331 Virtual space generation unit 332 Virtual object Generation unit 333 Avatar generation unit 340 Communication control unit 350 Storage unit 351 Virtual space information 352 User information 360 Input/output control unit 110 Virtual space application 120 Display control unit 123 View image generation unit 130 Virtual space control unit 140 Communication control unit 150 Storage unit 160 Operation Input Information Generation Unit 161 Position Input Information Generation Unit 162 Viewpoint Input Information Generation Unit 170 Image Recognition Unit 171 Visual Field Image Acquisition Unit 172 Analysis Unit 181 Text Generation Unit 182 Association Unit 151 Virtual Space Information 152 User Information 153 Condition Database 10 Virtual Space 10A, 10B, 20A, 20B, 30U, 30X, 30Y Avatar

Claims (9)

A control program of an information processing device connected to a virtual space system, in which a user's action and voice input via a user terminal are reflected in the action and voice of the user's avatar in the virtual space,
In the information processing device,
A position input function for generating input information designating a position in the virtual space,
An acquisition function for acquiring image information of the virtual space at the position designated by the position input function,
An analysis function of extracting one or more avatars included in the image information, and analyzing the action of the avatar when the extracted avatar satisfies a predetermined condition,
A generation function for generating text information describing information regarding the movement of the avatar, based on the result of analysis by the analysis function;
A storage function of storing the text information generated by the generation function in a predetermined storage unit,
An associating function for associating annotation information, which is information specifying at least one of the action, facial expression, and voice of the avatar in the virtual space,
Is realized ,
The control program , wherein the generation function generates the text information when the annotation information associated with the extracted avatar includes a predetermined keyword as the predetermined condition . The acquisition function acquires audio information together with image information of the virtual space,
The analysis function extracts voice information of the avatar together with the avatar,
The generation function further uses the voice information to determine the predetermined condition,
The control program according to claim 1, wherein: The position input function generates input information designating a plurality of positions within a predetermined distance from the avatar when the avatar extracted by the analysis function satisfies the predetermined condition,
The acquisition function acquires moving image information regarding the avatar at the plurality of positions,
The control program according to claim 1 or 2 , characterized in that. The associating function updates annotation information associated with the avatar based on a result of learning a series of actions of the avatar.
The control program according to any one of claims 1 to 3 , characterized in that: The storage function uses a digital watermark or a distributed ledger technology to store the text information generated by the generation function,
Control program according to any one of claims 1 to 4, characterized in that. The storage function further stores registration information registered in the virtual space system as one avatar,
The position input function generates input information designating a position of the one avatar in the virtual space system,
The acquisition function acquires image information of the virtual space as a field of view of the avatar of 1,
Control program according to any one of claims 1 to 5, characterized in that. The first avatar is realized as a transparent object that allows a background to be transparent in the virtual space.
The control program according to claim 6 , wherein A control method of an information processing device connected to a virtual space system, wherein a user's action and voice input via a user terminal are reflected in the action and voice of said user's avatar in a virtual space,
The information processing device is
Generating input information designating a position in the virtual space;
Acquiring image information of the virtual space at the designated position,
Extracting one or more avatars included in the image information, and analyzing the action of the avatar when the extracted avatar satisfies a predetermined condition,
Generating text information describing information about the movement of the avatar based on the result of the analysis by the analyzing step,
Storing the text information generated in the generating step in a predetermined storage unit,
Associating annotation information, which is information specifying at least one of the action, facial expression, and voice of the avatar in the virtual space,
The execution,
The control method of the information processing device , wherein, in the step of generating, when the predetermined keyword is included in the annotation information associated with the extracted avatar as the predetermined condition, the text information is generated . An information processing device connected to a virtual space system in which a user's motion and voice input via a user terminal are reflected in the motion and voice of the user's avatar in the virtual space,
A position input unit that generates input information that specifies a position in the virtual space,
An acquisition unit that acquires image information of the virtual space at the position specified by the position input unit;
Out extraction of one or more avatar included in the image information, when extracted the avatar satisfies a predetermined condition, an analysis unit for analyzing the behavior of the avatar,
A generation unit that generates text information that describes information about the action of the avatar, based on the result of analysis by the analysis unit;
A storage control unit that stores the text information generated by the generation unit in a predetermined storage unit;
An associating unit for associating annotation information, which is information specifying at least one of the action, facial expression, and voice of the avatar in the virtual space,
Equipped with
The information processing apparatus , wherein the generation unit generates the text information when the annotation information associated with the extracted avatar includes a predetermined keyword as the predetermined condition .

Images