JP2023133885A - Program, information processing device and method - Google Patents

Abstract

To provide a technique facilitating operations for receiving provision of various services in a virtual space.SOLUTION: A terminal device 10 of an avatar service provision system 1 executes, as a function, the steps of: acquiring a captured image obtained by capturing a moving person with a camera; analyzing the acquired capture image and specifying skeleton information of the person based on an analysis result; extracting skeleton information of the hands of the person from the specified skeleton information; displaying hand images imitating the hands, on a screen; controlling the hand images to move according to movements of the person's hands based on the extracted skeleton information of the person's hands; and detecting a prescribed movement of the hand images to execute a function associated with the prescribed movement.SELECTED DRAWING: Figure 6

Description

The present disclosure relates to a program, an information processing device, and a method.

Recently, it has become possible to take a picture of a person using a camera equipped with a mobile terminal such as a smartphone, display an avatar, which is a character associated with the photographed person, in a virtual space, and provide various services in the virtual space. It is being said. Such services provide various services without displaying the appearance of the actual person, so it is possible to receive various services without exposing the appearance of the person to unspecified users, so it is possible to receive various services without displaying the appearance of the person. of users.

Patent Document 1 discloses a technique for generating a composite image in which a human object is replaced with an avatar based on avatar information and skeleton information related to a user account.

Japanese Patent Application Publication No. 2021-157800

By the way, when an avatar of a person photographed with a mobile terminal is displayed in real time in a virtual space and various services are provided in the virtual space, another person may operate the mobile terminal to start providing the various services. The person in question had to perform the operations themselves, such as starting or ending the program. Therefore, in order to receive such services, troublesome operations are required.

Therefore, in the present disclosure, a technology that facilitates operations for receiving various services in a virtual space will be described.

According to one embodiment of the present disclosure, a program is provided that causes a computer including a processor and a memory to execute and generate an image to be displayed on a screen. The program causes the processor to acquire an image of a moving person taken with a camera, analyze the acquired image, identify the person's skeletal information based on the analysis result, and use the identified skeletal information. A step of extracting skeletal information of a person's hand, a step of displaying a hand image imitating the hand on the screen, and a step of displaying a hand image that matches the movement of the person's hand based on the extracted skeletal information of the person's hand. A step of controlling the hand image to move, and a step of detecting a predetermined motion of the hand image and executing a function associated with the predetermined motion are executed.

According to the present disclosure, a hand image is controlled to be moved in accordance with the hand motion of the photographed person, a predetermined motion of the hand image is detected, and a function associated with the predetermined motion is executed. This makes it possible to facilitate operations for receiving various services in the virtual space.

1 is a configuration diagram schematically showing an avatar service providing system according to the present disclosure. 1 is a configuration diagram showing the overall configuration of an avatar service providing system 1 according to a first embodiment; FIG. 1 is a block diagram showing a functional configuration of a terminal device 10 that constitutes an avatar service providing system 1 according to a first embodiment. FIG. FIG. 2 is a block diagram showing a functional configuration of a server 20 that constitutes the avatar service providing system 1 of the first embodiment. 2 is a flowchart illustrating an example of a flow of user authentication processing performed by the avatar service providing system 1 of the first embodiment. 2 is a flowchart illustrating an example of a flow of hand image movement processing performed by the avatar service providing system 1 according to the first embodiment. 2 is a flowchart illustrating an example of a flow of icon control processing performed by the avatar service providing system 1 according to the first embodiment. 3 is a diagram showing an example of a screen in which hand images 31L and 31R are displayed on a display screen 30. FIG. 3 is a diagram showing an example of a screen in which an icon 32 displayed on a display screen 30 is selected. FIG. 3 is a diagram illustrating an example of a screen in which an icon 32 displayed on a display screen 30 is executed. FIG. 3 is a diagram illustrating an example screen of screen scrolling using hand images 31L and 31R displayed on the display screen 30. FIG.

Embodiments of the present disclosure will be described below with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<Summary>
The avatar service providing system according to the present disclosure will be described below. This avatar service provision system takes a picture of a person using a camera equipped with a mobile terminal such as a smartphone, displays an avatar, which is a character associated with the photographed person, in a virtual space, and provides various services in the virtual space. It is a system for providing. Various services provided in virtual space include, for example, entertainment such as games in which avatars are manipulated by the user's movements, online fitness services in which multiple avatars are displayed according to the movements of each user, and multiple services. Examples include communication services using avatars. The avatar service providing system according to the present disclosure is a system that provides various services performed through an avatar that is a projection of such a user, for example, as a web service, and provides so-called SaaS (Software as a Service) as a web service using a cloud server or the like. This is a system provided by Alternatively, an avatar service providing system is a system in which various services are provided via an avatar that is a projection of such a user, for example, by a mobile terminal such as a smartphone (which may or may not be via a web server). It is.

FIG. 1 is a configuration diagram showing an outline of an avatar service providing system according to the present disclosure. As shown in FIG. 1, the avatar service providing system includes a terminal device 10 such as a smartphone, and a display screen 30. The terminal device 10 is a mobile terminal such as a smartphone or a tablet. The display screen 30 is an image display device such as a display, a television, or a projector, and is configured to have a size that can be viewed from a certain distance. The terminal device 10 is communicably connected to the display screen 30 via wired or wireless communication means. Examples of communication means at this time include communication cables based on standards such as USB (Universal Serial Bus) and HDMI (registered trademark) (High Definition Multimedia Interface), Miracast (registered trademark), WiDi, AirPlay (registered trademark), etc. Examples include, but are not limited to, wireless communication based on the standards of . A virtual space is displayed on the display 132 of the terminal device 10, and when a person H is photographed with the terminal device 10, a character associated with the person H is displayed on the virtual space displayed on the display 132 of the terminal device 10. A certain avatar A is displayed. At this time, for example, the terminal device 10 may be photographed in landscape orientation so that the aspect ratio (length-to-width ratio) of the display screen 30 is approximated. The virtual space and avatar A are also displayed on the display screen 30. Avatar A is configured to move in response to the body movements of person H, and various services are provided based on the movements of avatar A. Furthermore, the information on the avatars A may be shared by a web server or the like, and a plurality of shared avatars A may be displayed on the display screen 30 to provide various services.

Here, when trying to photograph the body movements of the person H using the terminal device 10, it is necessary to photograph the body at a certain distance from the person H. In this state, it is necessary to operate the terminal device 10 in order to start providing various services, and if the person H is doing it alone, the person H must move to the location of the terminal device 10 and operate the terminal device 10. , to start providing various services, and then to return to a position distant from the terminal device 10, which is a cumbersome operation for the person H.

Therefore, in the avatar service providing system according to the present disclosure, before the provision of various services is started, a hand image imitating a hand is displayed and the hand image is controlled to move in accordance with the hand movements of the person H. It is possible to start providing various services by moving the hand image. For example, icons for starting the provision of various services are displayed on the display screen 30, and the icon can be selected and executed by moving the hand image. Therefore, the person H does not need to return to the terminal device 10 and operate it, thereby eliminating the need for troublesome operations. This allows the avatar service providing system to smoothly provide various services.

Further, in the avatar service providing system according to the present disclosure, a menu screen for providing various services can be operated by the movement of a hand image. Specifically, multiple icons for starting the provision of various services are displayed on the display screen 30, and if they cannot be displayed on one screen, they may be displayed on two or more screens. It allows you to scroll between multiple screens. This eliminates the need for troublesome operations and allows the avatar service providing system to smoothly provide various services.

<First embodiment>
The avatar service providing system 1 will be explained below. In the following description, for example, when the terminal device 10 accesses the server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

<1 Overall configuration of avatar service provision system 1>
FIG. 2 is a configuration diagram showing the overall configuration of the avatar service providing system 1. As shown in FIG. As shown in FIG. 2, the avatar service providing system 1 includes a plurality of terminal devices (in FIG. 2, a terminal device 10A and a terminal device 10B are shown. Hereinafter, they may be collectively referred to as "terminal devices 10"). and a server 20. The terminal device 10 and the server 20 are connected via a network 80 so that they can communicate with each other. The network 80 is configured by a wired or wireless network. Further, the terminal device 10 is communicably connected to the display screen 30 as shown in FIG. 1, but is not shown in FIG. 2.

The terminal device 10 is a device operated by each user. The terminal device 10 is realized by a mobile terminal such as a smartphone or a tablet that is compatible with a mobile communication system. In addition, the terminal device 10 may be, for example, a portable notebook PC (Personal Computer), a laptop PC, a desktop PC, or a smart TV. The user who operates the terminal device 10 is a person who displays an avatar in a virtual space in the avatar service providing system 1 and receives various services, and is, for example, a person who is registered as a member of the avatar service providing system 1. be.

The terminal device 10 is communicably connected to the server 20 via the network 80. The terminal device 10 is a wireless base station 81 compatible with communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN (Local Area Network) standard such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. It is connected to the network 80 by communicating with a corresponding communication device such as a wireless LAN router 82. As shown as a terminal device 10B in FIG. 2, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage section 16, and a processor 19.

The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 communicates with an external device. The input device 13 is an input device (for example, a touch panel, a touch pad, a keyboard, a pointing device such as a mouse, etc.) for receiving input operations from a user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 19 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The server 20 is a device that receives input from a user for providing various services, displays an avatar in a virtual space, and distributes and provides various services in the virtual space as a web service. Note that the avatar service providing system 1 may provide various services only with the terminal device 10, and in this case, the server 20 is not an essential configuration.

Server 20 is a computer connected to network 80. The server 20 includes a communication IF 22 , an input/output IF 23 , a memory 25 , a storage 26 , and a processor 29 .

The communication IF 22 is an interface for inputting and outputting signals so that the server 20 communicates with external devices. The input/output IF 23 functions as an interface with an input device for receiving input operations from a user and an output device for presenting information to the user. The memory 25 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 29 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal device 10>
FIG. 3 is a block diagram showing the functional configuration of the terminal device 10 that constitutes the avatar service providing system 1 of the first embodiment. As shown in FIG. 3, the terminal device 10 includes a plurality of antennas (an antenna 111, an antenna 112), a wireless communication section (a first wireless communication section 121, a second wireless communication section 122) corresponding to each antenna, and an operation Reception unit 130 (including touch sensitive device 131 and display 132), audio processing unit 140, microphone 141, speaker 142, position information sensor 150, camera 160, storage unit 170, and control unit 180 including. The terminal device 10 also has functions and configurations that are not particularly illustrated in FIG. 3 (for example, a battery for holding power, a power supply circuit that controls the supply of power from the battery to each circuit, etc.). As shown in FIG. 3, each block included in the terminal device 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 111 receives radio waves from space and provides a received signal to the first wireless communication unit 121.

The antenna 112 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 112 receives radio waves from space and provides a received signal to the second wireless communication unit 122.

The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the terminal device 10 communicates with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the terminal device 10 communicates with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation/demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10 and provide received signals to the control unit 180.

The operation reception unit 130 has a mechanism for accepting user input operations. Specifically, the operation reception unit 130 is configured as a touch screen and includes a touch sensitive device 131 and a display 132. Note that the operation reception unit 130 may be configured by, for example, a keyboard, a mouse, or the like.

Touch sensitive device 131 receives input operations from the user of terminal device 10 . The touch sensitive device 131 uses, for example, a capacitive touch panel to detect the position of the user's touch on the touch panel. The touch sensitive device 131 outputs a signal indicating the user's touch position detected by the touch panel to the control unit 180 as an input operation.

The display 132 displays data such as images, videos, and text under the control of the control unit 180. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

The audio processing unit 140 modulates and demodulates the audio signal. The audio processing unit 140 modulates the signal provided from the microphone 141 and provides the modulated signal to the control unit 180. The audio processing unit 140 also provides an audio signal to the speaker 142. The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 140 . The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10 .

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current position of the terminal device 10 equipped with a GPS module based on the received signals.

The camera 160 is a device that receives light with a light receiving element and outputs it as a photographed image. The camera 160 is, for example, a device that can detect the distance from the camera 160 to the object to be photographed.

The storage unit 170 is configured of, for example, a flash memory, and stores data and programs used by the terminal device 10. In one aspect, storage unit 170 stores user information 171.

The user information 171 is information about a user who uses the terminal device 10 to display an avatar in a virtual space and receive various services, which is a function of the avatar service providing system 1 . The user information includes information for identifying the user (user ID), password, information such as the user's name, pseudonym, and avatar image information.

The control unit 180 controls the operation of the terminal device 10 by reading a program stored in the storage unit 170 and executing instructions included in the program. The control unit 180 is, for example, an application installed in the terminal device 10 in advance. The control unit 180 functions as an input operation reception unit 181, a transmission/reception unit 182, a notification control unit 183, and a data processing unit 184 by operating according to a program.

The input operation reception unit 181 performs a process of accepting a user's input operation on an input device such as the touch sensitive device 131. The input operation reception unit 181 determines whether the user's operation is a flick operation, a tap operation, or a drag ( Determine the type of operation, such as whether it is a swipe operation. In addition, the input operation receiving unit 181 performs a process of receiving the user's voice from the microphone 141.

The transmitting/receiving unit 182 performs processing for the terminal device 10 to transmit and receive data with an external device such as the server 20 according to a communication protocol.

The notification control unit 183 performs a process of presenting information to the user. The notification control unit 183 performs processes such as displaying a display image on the display 132 and outputting audio to the speaker 142.

The data processing unit 184 performs calculations according to a program on the data input by the terminal device 10, and outputs the calculation results to a memory or the like.

In addition, the data processing unit 184 acquires a captured image of a moving person captured by a camera as various modules by the processor 19 of the terminal device 10 performing processing according to a program, analyzes the captured captured image, and analyzes the captured image. The person's skeletal information is identified based on the analysis results, the person's hand skeletal information is extracted from the identified skeletal information, a hand image imitating a hand is displayed on the screen, and based on the extracted person's hand skeletal information, The hand image is controlled to move in accordance with the movement of a person's hand, a predetermined movement of the hand image is detected, and a function associated with the predetermined movement is executed. Furthermore, the data processing unit 184 may be configured to display a predetermined icon, and when a predetermined action on the hand image icon is detected, to select the icon or to perform a function associated with the icon. .

<1.2 Functional configuration of server 20>
FIG. 4 is a block diagram showing the functional configuration of the server 20 that constitutes the avatar service providing system 1 of the first embodiment. As shown in FIG. 4, the server 20 functions as a communication section 201, a storage section 202, and a control section 203.

The communication unit 201 performs processing for the server 20 to communicate with external devices.

The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores a user database 2021 and the like.

The user database 2021 is a database for holding information regarding users who display avatars in a virtual space and receive various services in the avatar service providing system 1. The information regarding the user includes information for identifying the user (user ID), password, information such as the user's name, name, and pseudonym, avatar image information, and the like.

The control unit 203 performs functions shown in a reception control module 2031, a transmission control module 2032, and a user authentication module 2033 as various modules by the processor of the server 20 performing processing according to a program.

The reception control module 2031 controls the process by which the server 20 receives signals from an external device according to a communication protocol.

The transmission control module 2032 controls the process by which the server 20 transmits a signal to an external device according to a communication protocol.

The user authentication module 2033 controls the process of authenticating that the user accessing the server 20 is a valid user. User authentication by the user authentication module 2033 is performed based on, for example, that the user ID is registered, that the same password as the password registered for the user ID is input, and so on.

<2 Operation>
Hereinafter, the user authentication process, hand image movement process, and icon control process by the avatar service providing system 1 in the first embodiment will be described with reference to FIGS. 5 to 7.

FIG. 5 is a flowchart showing an example of the flow of user authentication processing performed by the avatar service providing system 1 of the first embodiment. Note that the avatar service providing system 1 may provide various services only using the terminal device 10, and in this case, the user authentication process shown in FIG. 5 does not need to be performed.

In step S111, the input operation reception unit 181 of the terminal device 10 starts the dedicated application of the avatar service providing system 1 according to the user's operation. The transmitter/receiver 182 transmits user information such as a user ID and password to the server 20 . The transmitting/receiving unit 182 receives various information transmitted from the server 20 for displaying an avatar in a virtual space and receiving various services. The notification control unit 183 causes the display 132 to display the received various information.

In step S121, the user authentication module 2033 of the server 20 receives user information transmitted from the terminal device 10 via the communication unit 201. The user authentication module 2033 of the server 20 authenticates that the received user information is from a valid user. If the server 20 determines that the received user information is valid, it displays various information on the display 132 of the user's terminal device 10 to display an avatar in the virtual space and provide various services. In order to do so, the information is transmitted to the terminal device 10 via the communication unit 201.

FIG. 6 is a flowchart illustrating an example of the flow of hand image movement processing performed by the avatar service providing system 1 of the first embodiment.

In step S211, the data processing unit 184 of the terminal device 10 acquires an image captured by the camera 160 of a person who is moving according to the user's operation. For example, as shown in FIG. Acquire a photographed image, for example a video. Note that at this time, the device for photographing the person H is not limited to the camera 160 of the terminal device 10, but may be a camera as an external device, for example, a LIDAR (Light Detection and Ranging) camera (sensor), an optical or A mechanical (Inertial Measurement Unit) motion capture system or the like may be used. The data processing unit 184 of the terminal device 10 stores the captured image in the storage unit 170 as image data.

In step S212, the data processing unit 184 of the terminal device 10 analyzes the image data based on the photographed image acquired in step S211, and identifies skeletal information of the photographed person based on the analysis result. For example, by analyzing the image data, the data processing unit 184 identifies the body parts of the person in the photographed image using known image recognition technology, and connects the points indicating the respective body parts with straight lines. Information obtained from points and lines is assumed to be skeleton information. The data processing unit 184 uses the position of a point in the photographed image as coordinate information in the photographed image. This point is arranged, for example, at a joint part of the body, and is configured to change the angle with the straight part according to the movement of the body. This configuration makes it possible to express body movements using points and straight lines. The data processing unit 184 of the terminal device 10 causes the storage unit 170 to store the identified skeleton information.

In step S213, the data processing unit 184 of the terminal device 10 extracts skeletal information of the photographed person's hand from the skeletal information identified in step S212. For example, the data processing unit 184 identifies the hand of the person in the photographed image by analyzing the image data, and extracts information about points placed on the hand, for example, information about the coordinates of the points.

In step S214, the data processing unit 184 of the terminal device 10 displays a hand image imitating a hand on the screen. The hand image is, for example, an image imitating the shape of a human hand, but may have any shape as long as it is possible to recognize where on the screen the hand is pointing, and may be in the shape of an arrow. Further, the image data of the hand image is stored in the storage unit 170. The data processing unit 184 of the terminal device 10 causes the hand image to be displayed on the display 132 and on the display screen 30 as shown in FIG. The hand images to be displayed may be those of both hands, or may be those of one hand, either the left or the right.

In step S215, the data processing unit 184 of the terminal device 10 controls the hand image displayed in step S214 to move in accordance with the movement of the person's hand, based on the extracted person's hand skeleton information.

As described above, in the avatar service providing system 1, a hand image imitating a hand is displayed on the screen, and the hand image is controlled to move in accordance with the movement of a person's hand.

FIG. 7 is a flowchart illustrating an example of the flow of icon control processing by the avatar service providing system 1 of the first embodiment.

In step S311, the data processing unit 184 of the terminal device 10 displays a predetermined icon on the screen. This icon is an icon for starting the provision of various services provided by the avatar service providing system 1 on the screen, and is selected by the hand image to execute the function (various services) associated with the icon. .

In step S312, the data processing unit 184 of the terminal device 10 detects the position of the hand image and the position of the icon on the screen. For example, the data processing unit 184 detects the position of the hand image on the screen as information on the coordinate position on the screen at any time in accordance with the movement of the person's hand. Furthermore, the data processing unit 184 detects the position of the icon on the screen as information on the coordinate position on the screen.

In step S313, when the data processing unit 184 of the terminal device 10 performs a predetermined operation on the hand image icon, for example, when detecting that the position of the hand image and the icon position on the screen match, the data processing unit 184 of the terminal device 10 puts the icon in a selected state, or perform the function associated with the icon. For example, when the data processing unit 184 detects that the position of the hand image of one hand among the hand images of both hands on the screen matches the position of the icon, the data processing unit 184 puts the icon in the selected state (the selected state, and the function is is not executed). Further, when the data processing unit 184 detects that the positions of both hand images on the screen match the positions of the icons, the data processing unit 184 places the icons in an execution state (a state in which a function associated with the icons is executed). Make it.

As described above, the avatar service providing system 1 is configured to display icons for starting the provision of various services, and to select and execute the icons by moving the hand image. This eliminates the need for the user to return to the terminal device 10 and operate it, thereby eliminating the need for troublesome operations.

As another example of the operation using the hand image, the image displayed on the screen may be scrolled when an action of moving the hand image in the up, down, left and right directions is detected. This eliminates the need for the user to return to the terminal device 10 and operate it, thereby eliminating the need for troublesome operations.

<3 Screen example>
Hereinafter, examples of screens for operations using hand images in the avatar service providing system 1 will be described with reference to FIGS. 8 to 11.

FIG. 8 is a diagram showing an example of a screen in which hand images 31L and 31R are displayed on the display screen 30. The screen example in FIG. 8 shows a screen example in which a hand image that is moved in accordance with the hand movements of the person H is displayed on the display screen 30 shown in FIG. This corresponds to step S214 in FIG.

As shown in FIG. 8, the display 132 of the terminal device 10 displays hand images 132L and 132R that are associated with the person H and are moved in accordance with the movements of the person H's hands. Also displayed on the display screen 30 are hand images 31L and 31R that are moved in accordance with the hand movements of the person H. The hand images 31L and 31R are moved in accordance with the movements of the left hand and the right hand of the person H, respectively. Note that at this time, a virtual space may be displayed in the background of the display 132 and display screen 30 of the terminal device 10, but illustration thereof is omitted.

When the person H moves his right hand, the hand image 31R moves vertically and horizontally on the display screen 30 in accordance with the movement. Similarly, when the person H moves his left hand, the hand image 31L moves vertically and horizontally on the display screen 30 in accordance with the movement. Thereby, it is possible to move the hand image in accordance with the movement of the person H's hand.

FIG. 9 is a diagram showing an example of a screen in which the icon 32 displayed on the display screen 30 is selected. Further, FIG. 10 is a diagram showing an example of a screen in which the icon 32 displayed on the display screen 30 is executed. In the screen examples shown in FIGS. 9 and 10, a hand image is displayed on the display screen 30 shown in FIG. 1 to be moved in accordance with the hand movements of person H, and icons for starting the provision of various services are displayed. An example screen is shown. This corresponds to step S311 in FIG.

As shown in FIGS. 9 and 10, the display screen 30 displays hand images 31L and 31R that are moved in accordance with the hand movements of the person H. Further, an icon 32 is displayed on the display screen 30.

Hand images 31L and 31R move in accordance with the movement of the hand of person H, and their positions on the screen are detected. In the state shown in FIG. 9, it is detected that the position of one of the hand images 31L, 31R on the screen matches the position of the icon 32. At this time, the icon 32 is in a selected state. Furthermore, in the state shown in FIG. 10, it is detected that the positions of both hand images 31L and 31R on the screen match the position of the icon 32. At this time, the icon 32 is in the execution state. This allows the user to select the icon 32 and execute the function of the icon 32 without returning to the terminal device 10.

Note that the operation of selecting the icon 32 and executing the function of the icon 32 is not limited to the sequence shown in FIGS. 9 and 10. For example, if the position of one of the hand images 31L, 31R matches the position of the icon 32 for a certain period of time or more, the icon 32 is brought into a selected state, and the position of one of the hand images 31L, 31R matches the position of the icon 32 for a certain period of time. If the position of the icon 32 coincides with the position of the icon 32 (so-called double click), the function of the icon 32 may be executed.

FIG. 11 is a diagram showing a screen example of screen scrolling using hand images 31L and 31R displayed on the display screen 30. The screen example in FIG. 11 shows a screen example in which a hand image that is moved in accordance with the hand movements of the person H is displayed on the display screen 30 shown in FIG. This corresponds to step S311 in FIG.

As shown in FIG. 11, the display screen 30 displays hand images 31L and 31R that are moved in accordance with the hand movements of the person H.

Hand images 31L and 31R move in accordance with the movement of the hand of person H, and their positions on the screen are detected. In the state shown in FIG. 11, it is detected that hand images 31L and 31R have slid in the direction of arrow X. At this time, the image displayed on the display screen 30 is scrolled in the direction of arrow X. This allows the user to perform an operation to scroll the screen without returning to the terminal device 10.

<Summary>
As described above, according to the present embodiment, a hand image is displayed and controlled to be moved in accordance with the user's hand movements. The configuration is such that, by the movement of the hand image, the hand image can select and execute an icon for starting provision of various services. Therefore, the user does not have to return to the terminal device and operate it, thereby eliminating the need for troublesome operations. This allows the avatar service providing system to smoothly provide various services.

In addition, although it is possible to start providing various services and scroll images by the movement of hand images, it is also configured to accept input from mobile terminals such as smartphones and voice input using voice recognition technology (multimodal). It's okay. By adopting such a configuration, it is possible to prevent malfunctions and to select a means for simplifying operation depending on the situation.

Note that the avatar service providing system according to the present disclosure is configured so that the image can be scrolled by the movement of the hand image. Therefore, the user does not have to return to the terminal device and operate it, thereby eliminating the need for troublesome operations. This allows the avatar service providing system to smoothly provide various services.

Although the disclosed embodiments have been described above, they can be implemented in various other forms, and can be implemented with various omissions, substitutions, and changes. These embodiments and modifications, as well as omissions, substitutions, and changes, are included within the technical scope of the claims and their equivalents.

<Additional notes>
The matters described in each of the above embodiments are additionally described below.

(Additional Note 1) A program for generating an image to be executed by a computer equipped with a processor 29 and a memory 25 and displayed on a screen, the program having the processor 29 take a picture of a moving person with a camera. A step of acquiring a photographed image (S211), a step of analyzing the acquired photographic image and identifying skeletal information of the person based on the analysis result (S212), and extracting skeletal information of the person's hand from the identified skeletal information. a step (S213), a step (S214) of displaying a hand image imitating a hand on the screen, and a step (S214) of displaying a hand image imitating the hand on the screen, and moving the hand image in accordance with the movement of the person's hand based on the extracted person's hand skeleton information. A program that executes a step of controlling (S215) and a step of detecting a predetermined motion of a hand image and executing a function associated with the predetermined motion (S313).

(Additional Note 2) The program further causes a step of displaying a predetermined icon on the screen (S311), and in the step of executing a function, when a predetermined motion of the hand image with respect to the icon is detected, the program selects the icon. The program according to (Appendix 1), which causes the program to enter a state or execute a function associated with an icon.

(Additional note 3) In the step of displaying a hand image, in the step of displaying a hand image imitating one or both hands and executing a function, a predetermined action of the hand image of one hand and a predetermined action of the hand image of both hands are performed. The program described in (Appendix 2) that causes the program to perform different functions.

(Additional note 4) When the action of one hand image touching an icon is detected, the icon is set to a selected state, and when the action of both hand images touching the icon is detected, the function associated with the icon is executed. The program described in 3).

(Additional note 5) In the step of executing the function, if a movement of the hand image in the up, down, left and right directions is detected, the image displayed on the screen is scrolled. Programs listed.

(Additional Note 6) An information processing device that includes a control unit 180 and a storage unit 170 and generates an image to be displayed on a screen, wherein the control unit 180 acquires a captured image of a moving person captured by a camera. (S211), a step of analyzing the acquired photographed image and identifying the skeletal information of the person based on the analysis result (S212), and a step of extracting the skeletal information of the person's hand from the identified skeletal information (S213). , a step of displaying a hand image imitating a hand on the screen (S214), and a step of controlling the hand image to move in accordance with the movement of the person's hand based on the extracted person's hand skeleton information (S214). S215); and a step (S313) of detecting a predetermined motion of a hand image and executing a function associated with the predetermined motion.

(Additional Note 7) A method for generating an image to be displayed on a screen, which is executed by a computer including a processor 29 and a memory 25, and includes a method in which the processor 29 photographs a moving person with a camera. A step of acquiring a photographed image (S211), a step of analyzing the acquired photographic image and identifying skeletal information of the person based on the analysis result (S212), and extracting skeletal information of the person's hand from the identified skeletal information. a step (S213), a step (S214) of displaying a hand image imitating a hand on the screen, and a step (S214) of displaying a hand image imitating the hand on the screen, and moving the hand image in accordance with the movement of the person's hand based on the extracted person's hand skeleton information. A method comprising: controlling (S215); and detecting a predetermined motion of a hand image and causing a function associated with the predetermined motion to be executed (S313).

1: Avatar service providing system 10: Terminal device 10A: Terminal device 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage unit 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 30: Display screen 31L: Hand image 31R: Hand image 32: Icon 80: Network 81: Wireless base station 82: Wireless LAN router 111: Antenna 112: Antenna 121: First wireless communication unit 122: Second wireless communication unit 130: Operation reception Section 131: Device 132: Display 140: Audio processing section 141: Microphone 142: Speaker 150: Position information sensor 160: Camera 170: Storage section 171: User information 180: Control section 181: Input operation reception section 182: Transmission/reception section 183: Notification control unit 184: Data processing unit 201: Communication unit 202: Storage unit 203: Control unit 2021: User database 2031: Reception control module 2032: Transmission control module 2033: User authentication module

Claims (7)

A program for generating an image to be executed by a computer including a processor and a memory and displayed on a screen, the program comprising:
The program causes the processor to:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
extracting hand skeletal information of the person from the identified skeletal information;
displaying a hand image imitating a hand on the screen;
controlling the hand image to move in accordance with the hand movements of the person based on the extracted hand skeleton information of the person;
A program that causes the program to execute the steps of detecting a predetermined motion of the hand image and executing a function associated with the predetermined motion. The program further includes:
executing a step of displaying a predetermined icon on the screen;
2. The program according to claim 1, wherein in the step of executing the function, when a predetermined motion of the hand image with respect to the icon is detected, the program selects the icon or causes the function associated with the icon to be executed. In the step of displaying the hand image, displaying the hand image imitating one or both hands,
3. The program according to claim 2, wherein in the step of executing the function, the predetermined action of the hand image of one hand and the predetermined action of the hand images of both hands are made to perform different functions. When detecting an action in which the hand image of one hand contacts the icon, the icon is set to a selected state; and when detecting an action in which the hand images of both hands contact the icon, executing a function associated with the icon; The program according to claim 3. According to any one of claims 1 to 4, in the step of executing the function, when detecting a movement of the hand image in up, down, left and right directions, the image displayed on the screen is scrolled. Programs listed. An information processing device that includes a control unit and a storage unit and generates an image to be displayed on a screen,
The control unit includes:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
extracting hand skeletal information of the person from the identified skeletal information;
displaying a hand image imitating a hand on the screen;
controlling the hand image to move in accordance with the hand movements of the person based on the extracted hand skeleton information of the person;
An information processing apparatus configured to detect a predetermined motion of the hand image and execute a function associated with the predetermined motion. A method for generating an image to be displayed on a screen executed by a computer comprising a processor and a memory, the method comprising:
The method includes the processor:
obtaining an image of a moving person taken with a camera;
analyzing the acquired photographed image and identifying skeletal information of the person based on the analysis result;
extracting hand skeletal information of the person from the identified skeletal information;
displaying a hand image imitating a hand on the screen;
controlling the hand image to move in accordance with the hand movements of the person based on the extracted hand skeleton information of the person;
Detecting a predetermined motion of the hand image and performing a function associated with the predetermined motion.