JP7386583B1 - Program, information processing device and method - Google Patents

Abstract

The present invention provides a program, an information processing device, and a method that facilitate operations for receiving various services in a virtual space. [Solution] In the avatar service providing system, a method using a terminal device 10 is to acquire a photographed image of a person with a camera, analyze the acquired photographed image, identify skeletal information of the person based on the analysis result, and identify the person. A pointer control process that detects a person's motion from the skeletal information obtained and displays a pointer on the screen that corresponds to the person's hand and indicates the position on the screen, and according to the first motion of the detected person, A plurality of predetermined icons are displayed on the screen S311, a plurality of icons are moved on the screen according to the second motion of the detected person, S312, and the hand motion is changed according to the hand motion of the person. At the same time, the pointer is moved to perform S313, and icon control processing is performed in S314 to execute a function associated with the icon according to the position with respect to one of the plurality of icons. [Selection diagram] Figure 7

Description

Application of Article 30, Paragraph 2 of the Patent Act Announcement via website (1) December 14, 2022_https://www. xrkaigi. com/pages/future_pitch (2) December 16, 2022_https://mirox. jimdosite. com/%E3%81%8A%E7%9F%A5%E3%82%89%E3%81%9B/(3) January 5, 2023_httpps://apps. apple. com/jp/app/mirrox/idl621968722 (4) January 5, 2023_https://prtmes. jp/main/html/rd/p/000000001.000114425. html(5) January 4, 2023_https://youtu. be/2oTwXA_ksZI

The present disclosure relates to a program, an information processing device, and a method.

Recently, it has become possible to take a picture of a person using a camera equipped with a mobile terminal such as a smartphone, display an avatar, which is a character associated with the photographed person, in a virtual space, and provide various services in the virtual space. It is being said. Such services provide various services without displaying the appearance of the actual person, so it is possible to receive various services without exposing the appearance of the person to unspecified users, so it is possible to receive various services without displaying the appearance of the person. of users.

Patent Document 1 discloses a technique for generating a composite image in which a human object is replaced with an avatar based on avatar information and skeleton information related to a user account.

Japanese Patent Application Publication No. 2021-157800

By the way, when an avatar of a person photographed with a mobile terminal is displayed in real time in a virtual space and various services are provided in the virtual space, another person may operate the mobile terminal to start providing the various services. Either the person in question had to perform operations such as starting or ending the process, or the person concerned had to operate the mobile terminal or controller themselves. Therefore, in order to receive such services, cumbersome operations are required, causing a loss of immersion and sense of unity with the avatar.

Therefore, in the present disclosure, a technology that facilitates operations for receiving various services in a virtual space will be described.

According to an embodiment of the present disclosure, a program is provided that is executed by a computer including a processor and a memory to generate an image to be displayed on a screen. The program causes the processor to acquire an image of a moving person taken with a camera, analyze the acquired image, identify the person's skeletal information based on the analysis result, and use the identified skeletal information. , a step of detecting a motion of the person; a step of displaying a pointer on the screen corresponding to the hand of the person indicating the position on the screen; displaying a predetermined icon (e.g., an icon button), and displaying a plurality of icons displayed on the screen in response to a second action of the detected person that is different from the first action. a step of moving the pointer; a step of controlling the pointer to move in accordance with the person's hand motion in accordance with the person's hand motion extracted from the detected person's motion; and a step of detecting the pointer motion and controlling the pointer to A step of executing a function associated with one icon in accordance with a position with one of the plurality of icons.

According to the present disclosure, a plurality of predetermined icons are displayed and moved on the screen according to the motion of the photographed person. The pointer is controlled to be moved in accordance with the movement of the person's hand, and the function associated with the first icon is executed according to the position of the pointer and the first icon. This facilitates operations for receiving various services in the virtual space, and it becomes possible to receive various services without losing the sense of immersion or oneness with the avatar.

1 is a configuration diagram schematically showing an avatar service providing system according to the present disclosure. FIG. 1 is a configuration diagram showing the overall configuration of an avatar service providing system 1 according to an embodiment. FIG. 1 is a block diagram showing a functional configuration of a terminal device 10 that constitutes an avatar service providing system 1 according to the present embodiment. FIG. 2 is a block diagram showing the functional configuration of a server 20 that constitutes the avatar service providing system 1 according to the present embodiment. It is a flowchart showing an example of the flow of user authentication processing performed by the avatar service providing system 1 of the present embodiment. It is a flowchart which shows an example of the flow of pointer control processing by avatar service provision system 1 of this embodiment. It is a flowchart which shows an example of the flow of icon control processing by avatar service provision system 1 of this embodiment. 3 is a diagram showing an example of a screen in which pointers 31L and 31R are displayed on a display screen 30. FIG. 3 is a diagram showing an example of a screen in which icons 32A, 32B, and 32C are displayed on a display screen 30. FIG. 3 is a diagram showing a virtual positional relationship between a person H and icons 32A to 32F in virtual space VR. FIG. FIG. 4 is a diagram showing an example of a screen in which an icon 32B displayed on the display screen 30 is selected.

Embodiments of the present disclosure will be described below with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<Summary>
The avatar service providing system according to the present disclosure will be described below. This avatar service provision system takes a picture of a person using a camera equipped with a mobile terminal such as a smartphone, displays an avatar, which is a character associated with the photographed person, in a virtual space, and provides various services in the virtual space. It is a system for providing. Various services provided in virtual space include, for example, entertainment such as games in which avatars are manipulated by the user's movements, online fitness services in which multiple avatars are displayed according to the movements of each user, and multiple services. Examples include communication services using avatars. The avatar service providing system according to the present disclosure is a system that provides various services performed through an avatar that is a projection of such a user, for example, as a web service, and provides so-called SaaS (Software as a Service) as a web service using a cloud server or the like. This is a system provided by In addition, the avatar service providing system is a system in which various services provided through an avatar that is a projection of such a user are provided by a mobile terminal such as a smartphone (which may or may not be via a web server). It is.

FIG. 1 is a configuration diagram showing an outline of an avatar service providing system according to the present disclosure. As shown in FIG. 1, the avatar service providing system includes a terminal device 10 such as a smartphone, and a display screen 30. The terminal device 10 is a mobile terminal such as a smartphone or a tablet. The display screen 30 is an image display device such as a display, a television, or a projector, and is configured to have a size that can be viewed from a certain distance. A virtual space is displayed on the display 132 of the terminal device 10, and when a person H is photographed with the terminal device 10, a character associated with the person H is displayed on the virtual space displayed on the display 132 of the terminal device 10. A certain avatar A is displayed. The virtual space and avatar A are also displayed on the display screen 30. Avatar A is configured to move in response to the body movements of person H, and various services are provided based on the movements of avatar A. Furthermore, the information on the avatars A may be shared by a web server or the like, and a plurality of shared avatars A may be displayed on the display screen 30 to provide various services.

Here, when trying to photograph the body movements of the person H using the terminal device 10, it is necessary to photograph the body at a certain distance from the person H. In this state, it is necessary to operate the terminal device 10 in order to start providing various services, and if the person H is doing it alone, the person H must move to the location of the terminal device 10 and operate the terminal device 10. , to start providing various services, and then to return to a position distant from the terminal device 10, which is a cumbersome operation for the person H.

Therefore, in the avatar service providing system according to the present disclosure, before the provision of various services is started, a pointer imitating the shape of a hand is displayed and the pointer is controlled to move in accordance with the hand movements of the person H. Various services can be started by operating the pointer. For example, icons for starting the provision of various services are displayed on the display screen 30, and the icon can be selected and executed by moving the pointer. Therefore, the person H does not need to return to the terminal device 10 and operate it, thereby eliminating the need for troublesome operations. This allows the avatar service providing system to smoothly provide various services.

Further, in the avatar service providing system according to the present disclosure, a plurality of predetermined icons are displayed according to a predetermined action (first action) of person H, and a plurality of predetermined icons are displayed according to a predetermined action (second action) of person H. to move multiple predetermined icons on the screen. One of the plurality of icons is made selectable, and when that icon is selected with the pointer, the function associated with the icon is executed. Specifically, a plurality of icons for starting provision of various services are displayed on the display screen 30, and the plurality of icons are all scrolled in accordance with a predetermined action (second action) of the person H. When one of these icons is displayed in front of the person H, the icon can be selected. This eliminates the need for troublesome operations and allows the avatar service providing system to smoothly provide various services.

<Embodiment>
The avatar service providing system 1 will be explained below. In the following description, for example, when the terminal device 10 accesses the server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

<1 Overall configuration of avatar service provision system 1>
FIG. 2 is a configuration diagram showing the overall configuration of the avatar service providing system 1. As shown in FIG. As shown in FIG. 2, the avatar service providing system 1 includes a plurality of terminal devices (in FIG. 2, a terminal device 10A and a terminal device 10B are shown. Hereinafter, they may be collectively referred to as "terminal devices 10"). and a server 20. The terminal device 10 and the server 20 are connected via a network 80 so that they can communicate with each other. The network 80 is configured by a wired or wireless network.

The terminal device 10 is a device operated by each user. The terminal device 10 is realized by a mobile terminal such as a smartphone or a tablet that is compatible with a mobile communication system. In addition, the terminal device 10 may be, for example, a portable notebook PC (Personal Computer) or a laptop PC. The user who operates the terminal device 10 is a person who displays an avatar in a virtual space in the avatar service providing system 1 and receives various services, and is, for example, a person who is registered as a member of the avatar service providing system 1. be.

The terminal device 10 is communicably connected to the server 20 via the network 80. The terminal device 10 is a wireless base station 81 compatible with communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN (Local Area Network) standard such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. It is connected to the network 80 by communicating with a corresponding communication device such as a wireless LAN router 82. As shown as a terminal device 10B in FIG. 2, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage section 16, and a processor 19.

The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 communicates with an external device. The input device 13 is an input device (for example, a touch panel, a touch pad, a keyboard, a pointing device such as a mouse, etc.) for receiving input operations from a user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 19 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The server 20 is a device that receives input from a user for providing various services, displays an avatar in a virtual space, and distributes and provides various services in the virtual space as a web service. Note that the avatar service providing system 1 may provide various services only with the terminal device 10, and in this case, the server 20 is not an essential configuration.

Server 20 is a computer connected to network 80. The server 20 includes a communication IF 22 , an input/output IF 23 , a memory 25 , a storage 26 , and a processor 29 .

The communication IF 22 is an interface for inputting and outputting signals so that the server 20 communicates with external devices. The input/output IF 23 functions as an interface with an input device for receiving input operations from a user and an output device for presenting information to the user. The memory 25 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 29 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal device 10>
FIG. 3 is a block diagram showing the functional configuration of the terminal device 10 that constitutes the avatar service providing system 1 of this embodiment. As shown in FIG. 3, the terminal device 10 includes a plurality of antennas (an antenna 111, an antenna 112), a wireless communication section (a first wireless communication section 121, a second wireless communication section 122) corresponding to each antenna, and an operation Reception unit 130 (including touch sensitive device 131 and display 132), audio processing unit 140, microphone 141, speaker 142, position information sensor 150, camera 160, storage unit 170, and control unit 180 including. The terminal device 10 also has functions and configurations that are not particularly illustrated in FIG. 3 (for example, a battery for holding power, a power supply circuit that controls the supply of power from the battery to each circuit, etc.). As shown in FIG. 3, each block included in the terminal device 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 111 receives radio waves from space and provides a received signal to the first wireless communication unit 121.

The antenna 112 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 112 receives radio waves from space and provides a received signal to the second wireless communication unit 122.

The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the terminal device 10 communicates with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the terminal device 10 communicates with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation/demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10 and provide received signals to the control unit 180.

The operation reception unit 130 has a mechanism for accepting user input operations. Specifically, the operation reception unit 130 is configured as a touch screen and includes a touch sensitive device 131 and a display (screen) 132. Note that the operation reception unit 130 may be configured by, for example, a keyboard, a mouse, or the like.

Touch sensitive device 131 receives input operations from the user of terminal device 10 . The touch sensitive device 131 uses, for example, a capacitive touch panel to detect the position of the user's touch on the touch panel. The touch sensitive device 131 outputs a signal indicating the user's touch position detected by the touch panel to the control unit 180 as an input operation.

The display 132 displays data such as images, videos, and text under the control of the control unit 180. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

The audio processing unit 140 modulates and demodulates the audio signal. The audio processing unit 140 modulates the signal provided from the microphone 141 and provides the modulated signal to the control unit 180. The audio processing unit 140 also provides an audio signal to the speaker 142. The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 140 . The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10 .

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current position of the terminal device 10 equipped with a GPS module based on the received signals.

The camera 160 is a device that receives light with a light receiving element and outputs it as a photographed image. The camera 160 is, for example, a depth camera that can detect the distance from the camera 160 to the object to be photographed.

The storage unit 170 is configured of, for example, a flash memory, and stores data and programs used by the terminal device 10. In one aspect, storage unit 170 stores user information 171.

The user information 171 is information about a user who uses the terminal device 10 to display an avatar in a virtual space and receive various services, which is a function of the avatar service providing system 1 . The user information includes information for identifying the user (user ID), password, information such as the user's name, pseudonym, and avatar image information.

The control unit 180 controls the operation of the terminal device 10 by reading a program stored in the storage unit 170 and executing instructions included in the program. The control unit 180 is, for example, an application installed in the terminal device 10 in advance. The control unit 180 functions as an input operation reception unit 181, a transmission/reception unit 182, a notification control unit 183, and a data processing unit 184 by operating according to a program.

The input operation reception unit 181 performs a process of accepting a user's input operation on an input device such as the touch sensitive device 131. The input operation reception unit 181 determines whether the user's operation is a flick operation, a tap operation, or a drag ( Determine the type of operation, such as whether it is a swipe operation. In addition, the input operation receiving unit 181 performs a process of receiving the user's voice from the microphone 141.

The transmitting/receiving unit 182 performs processing for the terminal device 10 to transmit and receive data with an external device such as the server 20 according to a communication protocol.

The notification control unit 183 performs a process of presenting information to the user. The notification control unit 183 performs processes such as displaying a display image on the display 132, outputting audio to the speaker 142, and generating vibrations on the camera 160.

The data processing unit 184 performs calculations according to a program on the data input by the terminal device 10, and outputs the calculation results to a memory or the like.

In addition, the data processing unit 184 acquires a captured image of a moving person captured by a camera as various modules by the processor 19 of the terminal device 10 performing processing according to a program, analyzes the captured captured image, and analyzes the captured image. The person's skeletal information is identified based on the analysis results, the person's motion is detected from the identified skeletal information, and a pointer indicating the position on the screen corresponding to the person's hand is displayed on the screen of the display 132, and the detection is performed. displaying a plurality of predetermined icons on the screen in response to a first action of the detected person; moving the plurality of icons displayed on the screen on the screen in response to a second action of the detected person; The pointer is controlled to move according to the person's hand movement, which is extracted from the detected person's movement, and the pointer movement is detected, and the pointer and one of the multiple icons are controlled to move according to the person's hand movement. Depending on the position with respect to the icon, it exhibits a function of executing a function associated with one icon. Further, the data processing unit 184 selectably displays only one of the plurality of icons, and moves the plurality of icons displayed on the screen in accordance with the second operation. It may be configured to change the icon.

<1.2 Functional configuration of server 20>
FIG. 4 is a block diagram showing the functional configuration of the server 20 that constitutes the avatar service providing system 1 of this embodiment. As shown in FIG. 4, the server 20 functions as a communication section 201, a storage section 202, and a control section 203.

The communication unit 201 performs processing for the server 20 to communicate with external devices.

The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores a user database 2021 and the like.

The user database 2021 is a database for holding information regarding users who display avatars in a virtual space and receive various services in the avatar service providing system 1. The information regarding the user includes information for identifying the user (user ID), password, information such as the user's name, name, and pseudonym, avatar image information, and the like.

The control unit 203 performs functions shown in a reception control module 2031, a transmission control module 2032, and a user authentication module 2033 as various modules by the processor of the server 20 performing processing according to a program.

The reception control module 2031 controls the process by which the server 20 receives signals from an external device according to a communication protocol.

The transmission control module 2032 controls the process by which the server 20 transmits a signal to an external device according to a communication protocol.

The user authentication module 2033 controls the process of authenticating that the user accessing the server 20 is a valid user. User authentication by the user authentication module 2033 is performed based on, for example, that the user ID is registered, that the same password as the password registered for the user ID is input, and so on.

<2 Operation>
Hereinafter, a user authentication process, a pointer movement process, and an icon control process, which are programs and methods executed by the avatar service providing system 1 in the first embodiment, will be described with reference to FIGS. 5 to 7.

FIG. 5 is a flowchart showing an example of the flow of user authentication processing performed by the avatar service providing system 1 of this embodiment. Note that the avatar service providing system 1 may provide various services only using the terminal device 10, and in this case, the user authentication process shown in FIG. 5 does not need to be performed.

In step S111, the input operation reception unit 181 of the terminal device 10 starts the dedicated application of the avatar service providing system 1 according to the user's operation. The transmitter/receiver 182 transmits user information such as a user ID and password to the server 20 . The transmitting/receiving unit 182 receives various information transmitted from the server 20 for displaying an avatar in a virtual space and receiving various services. The notification control unit 183 causes the display 132 to display the received various information.

In step S121, the user authentication module 2033 of the server 20 receives user information transmitted from the terminal device 10 via the communication unit 201. The user authentication module 2033 of the server 20 authenticates that the received user information is from a valid user. If the server 20 determines that the received user information is valid, it displays various information on the display 132 of the user's terminal device 10 to display an avatar in the virtual space and provide various services. In order to do so, the information is transmitted to the terminal device 10 via the communication unit 201.

FIG. 6 is a flowchart showing an example of the flow of pointer control processing by the avatar service providing system 1 of this embodiment.

In step S211, the data processing unit 184 of the terminal device 10 obtains a captured image of a moving person captured by the camera 160 in response to a user's operation. For example, as shown in FIG. 1, the data processing unit 184 uses a dedicated application of the avatar service providing system 1 to A captured image of a person H, for example a video, is acquired. Note that at this time, the device for photographing the person H is not limited to the camera 160 of the terminal device 10, but may be a camera as an external device, for example, a LIDAR (Light Detection and Ranging) camera (sensor), an optical or A mechanical (Inertial Measurement Unit) motion capture system or the like may be used. The data processing unit 184 of the terminal device 10 stores the captured image in the storage unit 170 as image data.

In step S212, the data processing unit 184 of the terminal device 10 analyzes the image data based on the photographed image acquired in step S211, and identifies skeletal information of the photographed person based on the analysis result. For example, by analyzing the image data, the data processing unit 184 identifies the body parts of the person in the photographed image using known image recognition technology, and connects the points indicating the respective body parts with straight lines. Information obtained from points and lines is assumed to be skeleton information. The data processing unit 184 uses the position of a point in the photographed image as coordinate information in the photographed image. This point is arranged, for example, at a joint part of the body, and is configured to change the angle with the straight part according to the movement of the body. This configuration makes it possible to express body movements using points and straight lines. The data processing unit 184 of the terminal device 10 causes the storage unit 170 to store the identified skeleton information.

In step S213, the data processing unit 184 of the terminal device 10 detects the motion of the photographed person from the skeleton information specified in step S212. For example, the data processing unit 184 identifies the body part of the person in the photographed image by analyzing the image data, extracts information about a point placed on the identified body part, for example, information about the coordinates of the point, Detect human movements. At this time, the data processing unit 184 may detect the person's movement as movement information of points (coordinates) placed on body parts, and may detect the person's movement by using annotations such as a machine learning model. The movement of the person may be detected as information on the movement of the person (for example, clapping hands, twisting the upper body, raising one or both hands, etc.) interpreted from the movement information of the person's body.

In step S214, the data processing unit 184 of the terminal device 10 displays on the display 132 of the terminal device 10 a pointer that corresponds to the person's hand and indicates the position on the display 132. The pointer is, for example, an image imitating the shape of a human hand, but may have any shape as long as it can be recognized where on the display 132 it is pointing, and may be in the shape of an arrow. The pointer to be displayed may be one of both hands, or one of either the left or right hand. The pointer image data is stored in the storage unit 170. Further, the data processing unit 184 of the terminal device 10 causes the pointer to be displayed on the display 132 and on the display screen 30 as shown in FIG.

In step S215, the data processing unit 184 of the terminal device 10 adjusts the motion of the person's hand, based on the skeleton information identified in step S212, to the hand motion of the person extracted from the person's motion detected in step S213. Then, the pointer displayed in step S214 is controlled to be moved.

As described above, in the avatar service providing system 1, a pointer imitating a hand is displayed on the screen, and the pointer is controlled to move in accordance with the movement of the person's hand.

FIG. 7 is a flowchart showing an example of the flow of icon control processing performed by the avatar service providing system 1 of this embodiment.

In step S311, the data processing unit 184 of the terminal device 10 displays a plurality of predetermined icons on the display 132 of the terminal device 10 in accordance with the predetermined motion (first motion) of the person detected in step S213. let This icon is an icon for starting the provision of various services provided by the avatar service providing system 1 on the screen, and is selected by the pointer to execute the function (various services) associated with the icon. The functions associated with the icon include, for example, as mentioned above, in a game where an avatar is controlled by the movement of a person, the functions include saving game play data, returning to the starting point, and displaying the user's own avatar ( Includes functions such as changing the avatar's equipment and clothing, changing various setting conditions of the virtual space, etc. Here, the predetermined action (first action) of the person includes, for example, the action of raising both hands by the person, the action of clapping hands, the action of raising one hand, and the voice of the person (for example, uttering "menu").

Furthermore, in step S311, the data processing unit 184 of the terminal device 10 is configured to selectably display only one of the plurality of icons when displaying a plurality of predetermined icons on the display 132. You can. For example, the data processing unit 184 allows the user to select only an icon located at a predetermined position on the display 132 (for example, an icon located in front of a person) from among the plurality of icons displayed on the display 132. , the icon at a predetermined position may be moved in step S312, which will be described later. Further, the data processing unit 184 may be configured to display only icons that can be selected by the user on the display 132.

Further, in step S311, the data processing unit 184 of the terminal device 10 displays a plurality of predetermined icons on the display 132, and allows one of the plurality of icons to be selected according to a predetermined action by the person. It may also be configured to be displayed. For example, if the user raises his or her right hand while a plurality of icons are displayed on the display 132, the data processing unit 184 determines that the icon placed on the right side has been selected, and the user If the user raises his or her left hand, it may be determined that the icon placed toward the left has been selected.

Furthermore, in step S311, the data processing unit 184 of the terminal device 10 may be configured to display a virtual space including a person on the display 132, and display a plurality of icons in this virtual space. . For example, the data processing unit 184 displays an avatar, which is a character associated with the person, on the display 132, and a plurality of icons exist in the virtual space including the avatar (for example, they are arranged so as to float in the virtual space). ), and the icon may be configured to be selectable by the avatar.

In step S312, the data processing unit 184 of the terminal device 10 selects a plurality of predetermined motions displayed on the display 132 of the terminal device 10 according to the predetermined motion (second motion) of the person detected in step S213. Move the icon on display 132. For example, the data processing unit 184 moves a plurality of predetermined icons on the display 132 in a direction indicated by the gesture, which is a predetermined action (second action) of the person. At this time, the data processing unit 184 may move all of the plurality of icons in the same manner (for example, in the same direction), or may move only some of the icons. Further, the data processing unit 184 may hide some of the plurality of predetermined icons displayed on the display 132, and display some of the icons again on the display 132. You can. Such an operation may be performed by scrolling the icon on the display 132 to display/hide the icon, or by causing the icon to appear/delete on the display 132 to display/hide the icon. Here, the predetermined action (second action) of the person is, for example, an action by the person of bringing both hands together, an action of twisting the upper body, an action of moving both hands in a predetermined direction at the same time, an action of clapping hands, an action of raising both hands, Although it includes the action of raising one hand and the voice of the person, it is a different action from the first action described above.

Further, in step S312, when moving the plurality of predetermined icons displayed on the display 132, the data processing unit 184 of the terminal device 10 moves the plurality of icons and changes one selectable icon. It may be configured to do so. For example, the data processing unit 184 allows the user to select only an icon located at a predetermined position on the display 132 (for example, an icon located in front of a person) from among the plurality of icons displayed on the display 132. , the icon at a predetermined position may be moved.

Furthermore, in step S312, the data processing unit 184 of the terminal device 10 displays the plurality of icons in the virtual space displayed in step S311 in the virtual space including the person displayed on the display 132. It may be configured to move by. For example, while an avatar, which is a character associated with the person, is displayed on the display 132, the data processing unit 184 may generate a virtual You may move multiple icons in space.

In step S313, the data processing unit 184 of the terminal device 10 detects the position of the pointer and the position of the icon on the screen. For example, the data processing unit 184 detects the position of the pointer on the screen as information on the coordinate position on the screen at any time in accordance with the movement of the person's hand. The data processing unit 184 also detects the position of the icon on the screen as information on the coordinate position on the screen, and detects the position of the icon moved in step S312 each time the icon is moved.

In step S314, when the data processing unit 184 of the terminal device 10 detects that the position of the pointer on the display 132 matches the position of the icon, the data processing unit 184 of the terminal device 10 selects the icon or executes the function associated with the icon. . At this time, for example, if the data processing unit 184 detects that the position of the pointer of one hand out of the pointers of both hands on the screen matches the position of the icon, the data processing unit 184 places the icon in the selected state (the selected state, and the function is When it detects that the position of both pointers on the screen matches the position of the icon, even if the icon is set to the execution state (the state in which the function associated with the icon is executed). good.

As described above, in the avatar service providing system 1, a plurality of predetermined icons are displayed according to a predetermined action (first action) of a person, and a plurality of predetermined icons are displayed according to a predetermined action (second action) of a person. Move a predetermined icon on the screen. One of the plurality of icons is made selectable, and when that icon is selected with the pointer, the function associated with the icon is executed. This eliminates the need for the user to return to the terminal device 10 and operate it, thereby eliminating the need for troublesome operations.

<3 Screen example>
Hereinafter, examples of screens for operations using a pointer in the avatar service providing system 1 will be described with reference to FIGS. 8 to 11.

FIG. 8 is a diagram showing an example of a screen in which pointers 31L and 31R are displayed on the display screen 30. The screen example in FIG. 8 shows a screen example in which a pointer that is moved in accordance with the hand movements of the person H is displayed on the display screen 30 shown in FIG. This corresponds to step S214 in FIG.

As shown in FIG. 8, an avatar A, which is a character associated with a person H, is displayed on the display 132 of the terminal device 10. Further, on the display screen 30, pointers 31L and 31R are displayed that are moved in accordance with the hand movements of the person H. The pointers 31L and 31R are moved in accordance with the movements of the left hand of the person H and those moved in accordance with the movements of the right hand of the person H, respectively.

When the person H moves his right hand, the pointer 31R moves vertically and horizontally on the display screen 30 in accordance with the movement. Similarly, when the person H moves his left hand, the pointer 31L moves vertically and horizontally on the display screen 30 in accordance with the movement. Thereby, it is possible to move the pointer in accordance with the movement of the person H's hand.

FIG. 9 is a diagram showing an example of a screen in which icons 32A, 32B, and 32C are displayed on the display screen 30. The screen example in FIG. 9 shows a state in which a plurality of predetermined icons 32A, 32B, and 32C are displayed on the display screen 30 shown in FIG. 1 in response to the user performing a predetermined action (first action). An example screen is shown. This corresponds to step S311 in FIG.

As shown in FIG. 9, icons 32A, 32B, and 32C are displayed on the display screen 30 along the horizontal direction of the display screen 30. These icons 32A, 32B, and 32C are used when the user performs a predetermined action (first action), for example, when the user performs a predetermined action (first action) with both hands, when only the pointers 31L and 31R are displayed, as in the display screen 30 shown in FIG. 8, for example. When the user performs an upward movement, the data processing unit 184 detects the movement and displays the icons 32A, 32B, and 32C in step S311 of FIG.

Further, in the state shown in FIG. 9, when the user performs a predetermined action (second action), for example, the user puts both hands together, the data processing unit 184 detects the action and proceeds to step S312 in FIG. Move the icons 32A, 32B, and 32C. The moving direction of the icons 32A, 32B, and 32C at this time is, for example, the direction in which the user moves both hands together. Due to this movement, for example, if the movement indicated by the user is to the left toward the display screen 30, the icons 32A, 32B, and 32C each move one by one to the left, and the icon 32A moves to the position of the icon 32B. The icon 32B moves to the position of the icon 32C, and the icon 32C moves out of the display range of the display screen 30. Further, a new icon may be displayed at the position of the icon 32A. In addition, at this time, when a certain icon (for example, icon 32B) is selected, the function performed by the icon 32B may be configured to change, and other icons (for example, icons 32A, 32C) may also be selected. The configuration may be such that the functions executed by the system change.

FIG. 10 is a diagram showing the virtual positional relationship between the person H and the icons 32A to 32F in the virtual space VR. The virtual space VR shown in FIG. 10 is an example in which the data processing unit 184 displays the virtual space VR including the user on the display 132, and the icons 32A, 32B, and 32C shown in FIG. FIG. 4 is a diagram schematically showing an example where there are icons 32A to 32F that include the icons 32A to 32F. This corresponds to step S311 in FIG.

As shown in FIG. 10, in the virtual space VR, icons 32A, 32B, 32C, 32D, 32E, and 32F are arranged in a ring shape around the person H in the virtual space VR. Icons 32A, 32B, 32C, 32D, 32E, and 32F shown in FIG. 10 are displayed on the display screen 30 when the user performs a predetermined action (first action). In the state shown in FIG. 10, only icons 32A, 32B, and 32C existing in the direction are displayed.

Furthermore, when the user performs a predetermined action (second action) in the state shown in FIG. direction) one by one along the ring. This allows the user to select the icon 32 without returning to the location of the terminal device 10.

FIG. 11 is a diagram showing an example of a screen in which the icon 32B displayed on the display screen 30 is selected. The screen example in FIG. 11 shows a screen example in which the positions of the pointers 31L and 31R match the position of the icon 32B on the display screen 30 shown in FIG. 1. This corresponds to step S314 in FIG.

As shown in FIG. 11, the display screen 30 displays pointers 31L and 31R that are moved in accordance with the hand movements of the person H. Further, icons 32A, 32B, and 32C are displayed on the display screen 30.

The pointers 31L and 31R move in accordance with the hand movements of the person H, and their positions on the screen are detected. In the state shown in FIG. 11, it is detected that the positions of the pointers 31L and 31R on the screen match the position of the icon 32B. At this time, the icon 32B becomes a selected state or an execution state. This allows the user to select the icon 32B and execute the function of the icon 32B without returning to the location of the terminal device 10.

Note that the operation of selecting the icon 32B and executing the function of the icon 32 is not limited to the sequence shown in FIG. 11. For example, if the position of either pointer 31L or 31R matches the position of icon 32B for a certain period of time or more, icon 32B is brought into a selected state, and the position of either pointer 31L or 31R matches the position of icon 32B twice within a certain period of time. 32B (so-called double click), the function of the icon 32B may be executed.

<Summary>
As described above, according to the present embodiment, the pointer is displayed, the pointer is controlled to move in accordance with the user's hand motion, and the pointer displays an icon that allows the pointer to start providing various services according to the pointer motion. It is configured so that it is possible to select and execute it. Therefore, the user does not have to return to the terminal device and operate it, eliminating the need for troublesome operations and making it possible to receive various services without losing the sense of immersion and unity with the avatar. This allows the avatar service providing system to smoothly provide various services.

Further, according to the present embodiment, a plurality of predetermined icons are displayed according to a predetermined action (first action) of the person, and a plurality of predetermined icons are displayed according to the predetermined action (second action) of the person. Move the icon on the screen. One of the plurality of icons is made selectable, and when that icon is selected with the pointer, the function associated with the icon is executed. Therefore, the user does not have to return to the terminal device and operate it, thereby eliminating the need for troublesome operations. This allows the avatar service providing system to smoothly provide various services.

Although the disclosed embodiments have been described above, they can be implemented in various other forms, and can be implemented with various omissions, substitutions, and changes. These embodiments and modifications, as well as omissions, substitutions, and changes, are included within the technical scope of the claims and their equivalents.

<Additional notes>
The matters described in each of the above embodiments are additionally described below.

(Additional Note 1) A program for generating an image to be executed by a computer equipped with a processor 29 and a memory 25 and displayed on a screen, the program having the processor 29 take a picture of a moving person with a camera. A step of acquiring a photographed image (S211), a step of analyzing the acquired photographic image and identifying skeletal information of the person based on the analysis result (S212), and a step of detecting the motion of the person from the identified skeletal information (S212). S213), a step of displaying a pointer on the screen that corresponds to the person's hand and indicates the position on the screen (S214), and a plurality of predetermined points are displayed on the screen in accordance with the first movement of the detected person. A step of displaying an icon (S311), and a step of moving a plurality of icons displayed on the screen on the screen according to a second motion of the detected person that is different from the first motion (S312). and a step (S215) of controlling the pointer to move in accordance with the person's hand motion in accordance with the person's hand motion extracted from the detected person's motion; detecting the pointer motion; A program that causes a program to execute a step (S314) of executing a function associated with one icon according to a position with respect to one of the plurality of icons.

(Additional note 2) In the step of displaying a plurality of icons on the screen, in the step of displaying only one icon among the plurality of icons on the screen in a selectable manner, and in the step of moving the plurality of icons, the second operation is performed. The program according to (Appendix 1), which moves a plurality of icons displayed on the screen and changes one selectable icon in accordance with the above.

(Additional Note 3) In the step of displaying a plurality of icons on the screen, a step of displaying a virtual space including a person on the screen, displaying the plurality of icons as if they exist in the virtual space, and moving the plurality of icons. The program according to Supplementary Note 2, wherein a plurality of icons in the virtual space are moved in the virtual space according to the second operation.

(Additional note 4) In the step of displaying a plurality of icons on the screen, in the step of displaying a plurality of icons in a ring shape around a person in the virtual space, and moving the plurality of icons, a second operation is performed. The program according to (Appendix 3), which moves a plurality of icons in a virtual space along a ring according to the above, and changes the one selectable icon.

(Additional Note 5) The program according to any one of (Appendix 1) to (Appendix 4), wherein the first action includes an action by the person of raising both hands and an action of clapping the hands.

(Additional Note 6) The program according to any one of (Appendix 1) to (Appendix 4), wherein the second action includes an action of twisting the upper body of the person and an action of moving both hands in a predetermined direction at the same time.

(Additional Note 7) An information processing device that includes a control unit 180 and a storage unit 170 and generates an image to be displayed on a screen, wherein the control unit 180 acquires a captured image of a moving person captured by a camera. (S211); a step (S212) of analyzing the acquired captured image and identifying skeletal information of the person based on the analysis result; and a step (S213) of detecting the motion of the person from the identified skeletal information. A step of displaying a pointer corresponding to the person's hand and indicating the position on the screen on the screen (S214), and displaying a plurality of predetermined icons on the screen according to the first movement of the detected person. a step (S311), a step (S312) of moving a plurality of icons displayed on the screen in accordance with a second motion of the detected person that is different from the first motion; A step (S215) of controlling the pointer to move according to the person's hand motion extracted from the person's hand motion (S215), and detecting the pointer motion and moving the pointer and the plurality of icons. An information processing apparatus that executes a step (S314) of causing a function associated with one of the icons to be executed according to a position with respect to one of the icons.

(Additional Note 7) A method for generating an image to be displayed on a screen, which is executed by a computer including a processor 29 and a memory 25, and includes a method in which the processor 29 photographs a moving person with a camera. A step of acquiring a photographed image (S211), a step of analyzing the acquired photographic image and identifying skeletal information of the person based on the analysis result (S212), and a step of detecting the motion of the person from the identified skeletal information (S212). S213), a step of displaying a pointer on the screen that corresponds to the person's hand and indicates the position on the screen (S214), and a plurality of predetermined points are displayed on the screen in accordance with the first movement of the detected person. A step of displaying an icon (S311), and a step of moving a plurality of icons displayed on the screen on the screen according to a second motion of the detected person that is different from the first motion (S312). and a step (S215) of controlling the pointer to move in accordance with the person's hand motion in accordance with the person's hand motion extracted from the detected person's motion; detecting the pointer motion; A method for performing a step (S314) of causing a function associated with one icon to be executed depending on a position with respect to one of the plurality of icons.

1: Avatar service providing system 10: Terminal device 10A: Terminal device 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage unit 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 30: Display screen 31L: Pointer 31R: Pointer 32: Icon 80: Network 81: Wireless base station 82: Wireless LAN router 111: Antenna 112: Antenna 121: First wireless communication unit 122: Second wireless communication unit 130: Operation reception unit 131 : Device 132 : Display 140 : Audio processing part 141 : Microphone 142 : Speaker 150 : Position information sensor 160 : Camera 170 : Storage part 171 : User information 180 : Control part 181 : Input operation reception part 182 : Transmission/reception part 183 : Notification control Unit 184: Data processing unit 201: Communication unit 202: Storage unit 203: Control unit 2021: User database 2031: Reception control module 2032: Transmission control module 2033: User authentication module

Claims (8)

A program for generating an image to be executed by a computer including a processor and a memory and displayed on a screen, the program comprising:
The program causes the processor to:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
detecting the motion of the person from the identified skeletal information;
displaying a pointer on the screen that corresponds to the person's hand and indicates a position on the screen;
Displaying a plurality of predetermined icons on the screen according to the detected first movement of the person;
moving the plurality of icons displayed on the screen in response to a second motion of the detected person that is different from the first motion;
controlling the pointer to move in accordance with the person's hand movement in accordance with the person's hand movement extracted from the detected person's movement;
detecting a movement of the pointer and executing a function associated with the one icon according to the position of the pointer and one of the plurality of icons ;
The first action and the second action include an action using both hands by the person and an action using the upper body of the person,
In the step of identifying skeletal information of the person, analyzing the captured image of the whole body of the person, and identifying skeletal information of the person based on the analysis result,
The program detects a motion using both hands of the person and a motion using the upper body of the person in the step of detecting the motion of the person. In the step of displaying a plurality of icons on the screen, displaying only one icon among the plurality of icons on the screen in a selectable manner;
According to claim 1, in the step of moving the plurality of icons, the first selectable icon is changed by moving the plurality of icons displayed on the screen according to the second operation. Programs listed. In the step of displaying a plurality of icons on the screen, a virtual space including the person is displayed on the screen, and the plurality of icons are displayed in the virtual space,
3. The program according to claim 2, wherein in the step of moving the plurality of icons, the plurality of icons in the virtual space are moved on the virtual space according to the second operation. In the step of displaying the plurality of icons on the screen, the plurality of icons are displayed in a ring shape around the person in the virtual space,
3. In the step of moving the plurality of icons, the plurality of icons in the virtual space are moved along the annular shape in response to the second operation, and the first selectable icon is changed. The program described in. 5. The program according to claim 1, wherein the first action includes an action by the person of raising both hands and an action of clapping hands. 5. The program according to claim 1, wherein the second action includes an action of twisting the upper body of the person and an action of moving both hands simultaneously in a predetermined direction. An information processing device that includes a control unit and a storage unit and generates an image to be displayed on a screen,
The control unit includes:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
detecting the motion of the person from the identified skeletal information;
displaying a pointer on the screen that corresponds to the person's hand and indicates a position on the screen;
Displaying a plurality of predetermined icons on the screen according to the detected first movement of the person;
moving the plurality of icons displayed on the screen in response to a second motion of the detected person that is different from the first motion;
controlling the pointer to move in accordance with the person's hand movement in accordance with the person's hand movement extracted from the detected person's movement;
detecting a movement of the pointer and executing a function associated with the one icon according to the position of the pointer and one of the plurality of icons ;
The first action and the second action include an action using both hands by the person and an action using the upper body of the person,
In the step of identifying skeletal information of the person, analyzing the captured image of the whole body of the person, and identifying skeletal information of the person based on the analysis result,
In the step of detecting a motion of the person, an information processing device detects a motion using both hands of the person and a motion using the upper body of the person. A method for generating an image to be displayed on a screen executed by a computer comprising a processor and a memory, the method comprising:
The method includes the processor:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
detecting the motion of the person from the identified skeletal information;
displaying a pointer on the screen that corresponds to the person's hand and indicates a position on the screen;
Displaying a plurality of predetermined icons on the screen according to the detected first movement of the person;
moving the plurality of icons displayed on the screen in response to a second motion of the detected person that is different from the first motion;
controlling the pointer to move in accordance with the person's hand movement in accordance with the person's hand movement extracted from the detected person's movement;
detecting a movement of the pointer and executing a function associated with the one icon according to the position of the pointer and one of the plurality of icons ;
The first action and the second action include an action using both hands by the person and an action using the upper body of the person,
In the step of identifying skeletal information of the person, analyzing the captured image of the whole body of the person, and identifying skeletal information of the person based on the analysis result,
The method includes detecting a motion of the person using both hands and a motion of the person using the upper body.