JP2023125517A - Information processing apparatus and program - Google Patents

Abstract

To increase the variety of motion of at least one of an avatar and a robot based on a motion of a person.SOLUTION: A skeleton information generation system H1 generates skeleton information, from a detection target image KTG represented by a target video ST1, and outputs the generated skeleton information as a data sequence ST5, and the detection target image KTG as an image ST6 of a detection target, to an attitude evaluation system H2. The attitude evaluation system H2 generates, in evaluating an attitude of the detection target image KTG using example data TD, text information to be transmitted to the detection target KT by differential AI analysis SH2 regarding a difference from an example, using a difference between skeleton information included in the example data TD and the input skeleton information. The text information is output as voice from a voice output system TO. The detection target image KTG and an example image OG included in the example data TD are displayed by an image display system GH.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device and a program.

A technique for estimating skeletal information of a person included in an image captured by a camera is known (for example, see Patent Document 1). There is also a known technology for remotely controlling a robot to perform a specific operation based on the movement of a person (for example, see Patent Document 2).

JP2021-189946A Japanese Unexamined Patent Publication No. 7-160310

The above-mentioned robots are controlled to operate in accordance with a person's physical movements or operations. In such motion control, the robot's motion not only becomes what the person who performs the physical movement including manipulation has assumed in advance, but also depends solely on that person. Depending on the purpose of the robot, it may be desirable to improve the variety of robot movements, taking into account convenience for humans or the morale it provides to humans. This also applies when an image such as an avatar is moved by a person's movements.

An object of the present invention is to provide an information processing device and a program that can improve the diversity of at least one of the motions of an avatar and a robot based on human motions.

An information processing device according to an aspect of the present disclosure includes an image data acquisition unit that acquires image data obtained by imaging a detection target, and a position of a joint in the detection target image based on the detection target image represented by the image data. skeletal information generating means for generating skeletal information including information and relationship information indicating relationships between joints; an analyzing means for analyzing motion content in the detection target image based on the skeletal information; a text information generation means for generating text information based on the content analysis result; and an operation for causing at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, to operate based on the text information. and a control means.

A program according to an aspect of the present disclosure causes an information processing device to obtain position information of joints in the detection target image and relationships between joints based on a detection target image represented by image data obtained by imaging assuming a detection target. generates skeleton information including relationship information indicating the text, analyzes the motion content in the detection target image based on the skeleton information, generates text information based on the analysis result of the motion content, and generates text information including relationship information indicating the text Based on the information, a process is executed to cause at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, to operate as an operation target.

According to the present invention, it is possible to improve the diversity of at least one of the motions of an avatar and a robot based on human motions.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating an example of a service providing system constructed by applying the present invention and the contents of services provided by the system. FIG. 6 is a diagram illustrating an example of a motion including a posture for causing the motion target to perform another motion. FIG. 1 is a diagram illustrating an example of a network environment connected to an AP server according to an embodiment of an information processing apparatus of the present invention. FIG. 2 is a diagram illustrating an example of a place for games and the like. FIG. 2 is a block diagram showing an example of the hardware configuration of an AP server according to an embodiment of the information processing device of the present invention. FIG. 2 is a functional block diagram showing an example of a functional configuration implemented on an AP server according to an embodiment of the information processing device of the present invention. 2 is a flowchart illustrating an example of a video display process executed by a CPU installed in an AP server, which is an information processing apparatus according to the present embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that the described embodiment is merely an example, and the technical scope of the present invention is not limited thereto. The technical scope of the present invention also includes various modifications.

FIG. 1 is a diagram illustrating an example of a service providing system constructed by applying the present invention and the contents of services provided by the system. Here, the service provided by this service providing system SY will be referred to as "this service" and will be explained.

In the example shown in FIG. 1, it is assumed that images (video shooting) of camera C are taken of animals, and the animals are mainly people. In this example, it is assumed that a person is the detection target KT whose skeleton is to be detected, and the information to be transmitted is timely transmitted to the detection target KT by audio output according to the operation content of the detection target KT. There is.

In order to confirm the operation details of the detection target KT, skeleton information of the detection target KT is generated. This skeletal information is information including position information of a person's joints and relationship information indicating relationships between joints. The relationship information is, for example, information representing the distance and direction between joints that are adjacent to each other on the skeleton. Generation of such skeleton information is performed by analysis using the target image ST1 obtained by imaging with the camera C. Thereby, the skeleton information of the detection target KT is generated from the target video ST1. The skeleton image SG shown in FIG. 1 represents an example of joint positions and joints for which relationship information is generated. The generation of skeleton information itself is performed using a well-known technique.

Note that a plurality of cameras C may be used for imaging. When a plurality of cameras C are installed at different positions, the target image ST1 obtained from each camera C makes it possible to confirm a portion of the detection target KT that is a blind spot for one or more cameras C. Therefore, skeleton information can be generated with higher accuracy. For example, this is particularly useful when there are two or more people KT to be detected. Here, for the sake of explanation and convenience, the explanation will be based on the assumption that there is only one camera C.

As shown in FIG. 1, the service providing system SY includes a skeletal information generation system H1, a posture evaluation system H2, and a posture management system H3.
Although each of the systems H1 to H3 is shown separately in FIG. 1, these systems (the functions thereof) may be realized on the same information processing device. Two of them may be implemented on the same information processing device. Here, as shown in Figure 1, it is assumed that the systems H1 to H3 are implemented on different information processing devices, that is, the service providing system SY is constructed using three or more information processing devices. Let's assume that there is one and explain.

The skeleton information generation system H1 is realized on an information processing device equipped with an OS (Operating System) H11. The OSH 11 is equipped with a camera device image acquisition/management switch (SW) H111. This camera device video acquisition/management switch H111 has a function of switching the destination to which various information including the target video ST1 received by the skeleton information generation system H1 is delivered. Thereby, the target video ST1 received by the skeleton information generation system H1 is passed to the processing engine H112 as the target video ST2 by the camera device video acquisition/management SWH111. The processing engine H112 and the skeleton detection unit H113 are functions realized by, for example, an application program (hereinafter abbreviated as "application") running on the OSH 11.

The processing engine H112 is a function that performs overall control for providing this service. The processing engine H112 causes the skeleton detection unit H113 to execute the skeleton detection process and generate skeleton information using the target video ST2 by the skeleton detection process call ST3. For this purpose, the processing engine H112 passes, for example, the target video ST2 to the skeleton detection unit H113 by a skeleton detection process call ST3. What is passed from the processing engine H112 to the skeleton detection unit H113 may be not the target video ST2 but the video information of only the portion including the detection target image. Here, it is assumed that the target video ST2 is passed from the processing engine H112 to the skeleton detection unit H113.

The skeleton detection unit H113 executes skeleton detection processing for generating skeleton information, and returns the processing result ST4 of the skeleton detection processing to the processing engine H112. This processing result ST4 is skeleton information generated in the form of a data string representing the skeleton.

The processing engine H112 can output the skeletal information, which is the processing result ST4 returned from the skeletal detection unit H113, to the posture evaluation system H2 and the posture management system H3, respectively, as a data string ST5 representing the skeleton. A detection target image KTG, which is an image of the detection target KT, is also output to the posture evaluation system H2 as a detection target image ST6.

The posture evaluation system H2 is an image display system that uses the detection target image ST6 inputted from the skeleton information generation system H1 or the character image generated using the skeleton representation data string ST5 as part of the image ST8 to be displayed. It can be output to GH. The image ST8 is for display on one screen, for example, and the character image is arranged and displayed within the screen. The character image is assumed to be an alter ego of the detection target KT.

Even when displaying another character image, the character may include an alter ego of another detection target KT or an alter ego of a virtual personality. The virtual persona is specifically envisioned for the provision of the Service. Hereinafter, among the characters, a character that is positioned as such an alter ego will be referred to as an "avatar." Unless otherwise specified, "character" is used to refer to characters other than avatars.

The image display system GH displays a screen on the display D based on the image ST8 input from the posture evaluation system H2. Therefore, the posture evaluation system H2 can arbitrarily change the content of the screen displayed on the display D through the image ST8 output to the image display system GH. FIG. 1 shows a state in which the detection target image KTG represented by the image ST8 is displayed on the display D. Instead of the detection target image KTG, an avatar image generated from skeleton information may be displayed.

The display D is, for example, a large display device or a screen. When the display D is a screen, the image display system GH also includes a projector capable of projecting onto the screen.
On the other hand, the posture evaluation system H2 generates text information to be outputted as audio by analyzing the skeleton representation data string ST5. For this purpose, the posture evaluation system H2 performs a difference AI (Artificial Intelligence) analysis SH2 with respect to the model.
As will be described in detail later, this text information also functions as control information for controlling various movements of an avatar image or a robot. The various actions include not only actions that visually change the object of action, but also displays that do not involve visual changes, actions that make speeches by emitting sounds, and the like.

The difference AI analysis SH2 with respect to the model is an analysis performed on the premise that the posture (pose) that the detection target KT should take at any timing is known in advance. For this analysis, model data TD representing a model posture is prepared. Movements where the posture to be taken is known in advance include dance, games that require movements that imitate a model movement, or movements that require movements that take one or more predetermined postures, and ideal movements. Exercises (for example, sports) that can be considered include sports. Hereinafter, such requests or a series of motions such as exercise will be collectively referred to as "posture-required motions."

The model data TD includes skeletal information in addition to the model image OG, which is an image in a model posture. Thereby, in the difference AI analysis SH2 with respect to the model, an AI analysis using the difference is performed, including calculation of the difference between the data string ST5 of the skeleton representation and the skeleton information included in the model data TD. Through the analysis, the posture represented by the detection target image KTG is evaluated, and text information is generated according to the evaluation result. In the example shown in FIG. 1, the difference correction instruction ST9 output from the posture evaluation system H2 to the audio output system TO is for instructing to emit text information as audio. Specifically, this difference correction instruction ST9 is an audio signal generated for sound emission, or a command containing information necessary for generating the audio signal.

For analysis using differences, AI performs deep learning using learning data representing the relationship between the differences and the text information to be generated. Through this deep learning, in the difference AI analysis SH2 with respect to the model, appropriate text information can be generated by generating (calculating) the difference. Since the ratio between the size of the detection target image KTG and the size of the model image OG represented by the model data TD is not necessarily within an appropriate range, the difference is, for example, one of the two pieces of skeleton information. It is generated after enlarging or reducing the image.

FIG. 1 shows that the model image OG is displayed on the display D in addition to the detection target image KTG.
The detection target image KTG is particularly different from the model image OG in the right hand. The left hand is also clearly different, but the degree of difference is small compared to the right hand. Therefore, in this case, text information representing a character string such as "raise your right hand slightly" is generated by differential AI analysis SH2 with respect to the model. As a result, this character string is output as audio from the audio output system TO, that is, is emitted. This audio output is performed assuming that the detection target image KTG or the model image OG is the avatar image.

Audio output by the audio output system TO is performed assuming a detection target KT. Therefore, the detection target KT can obtain useful information in a timely manner from the detection target image KTG displayed as an avatar image on the display D or from the model image OG through audio output to help him take the appropriate posture. It will be in the form of As a result, the displayed detection target image KTG or the model image OG is set as an operation target, and by generating text information from the detection target image KTG, a speaking operation is virtually performed to make the operation target speak.

Note that the action to be performed does not have to be a speaking action, that is, a voice output. For example, the motion target may be caused to perform a movement necessary for the detection target KT to approach the posture of the model image OG. Furthermore, a character string may be displayed as the content of the avatar image's statement. Audio output (output of messages, etc.) is performed only when the posture of the detection target image KTG is evaluated as inappropriate, and when the posture is evaluated as appropriate, so that the detection target KT can recognize this. Sound effects or performance sounds may be emitted. For this reason, the text information may represent sound effects or production sounds. The speaking action may be expressed by either sound emission or display.

As described above, the posture evaluation system H2 uses the data string ST5 of the skeleton representation inputted from the skeleton information generation system H1 to evaluate the posture of the detection target KT, and generates text information and creates an avatar image according to the text information. Make the action happen. In this way, the motion target is made to perform a different motion that is not present in the motion of the detection target KT, and information useful to the detection target KT is provided in the form of passing through the different motion. Therefore, not only the variety of motions in the motion target is improved, but also the convenience for the detection target KT is improved. The detection target KT can more easily perform a more appropriate posture requesting operation.

The detection target KT is required to recognize the voice output in order to move his or her body appropriately. Therefore, higher concentration is required. Since the motion must be performed in such a state, the detection target KT can also more easily obtain a higher sense of immersion. As a result, it becomes possible to give better confidence or a higher sense of satisfaction to the detection target KT.

On the other hand, the posture management system H3 uses the data string ST5 of the skeletal representation input from the skeletal information generation system H1 to control the motion of at least one of the robot, which is a physical machine, and the avatar image displayed on the screen. I am using it. Thereby, the motion target moves along the movement of the detection target image KTG. Therefore, the detection target KT can control the motion of the motion target through its own motion. Therefore, the detection target KT is provided with an environment in which he can enjoy moving the motion target as entertainment, or in which he can check his own posture or movement. The physical machine here is one that is equipped with a power source such as one or more motors. Robots, which are physical machines, are capable of changing their overall appearance using power transmitted from a power source.

The motion control of the robot is performed, for example, by transmitting control information for causing the robot to take the next posture as a motion/display instruction ST7 from the posture management system H3. This control information is mainly information for operating the power source. To control the movement of the avatar image, for example, the posture management system H3 transmits image information representing the screen to be displayed as a movement/display instruction ST7 to an image/audio generation system capable of displaying a screen including the avatar image as a moving image. It is done by

In both the motion target robot and the avatar image, the overall ratio of each part is usually different from the ratio in the detection target image KTG. For example, if the detection target image KTG is 8 heads and bodies, the motion targets are often less than 8 heads and bodies, and their ratios are usually different. For this reason, the posture management system H3 performs a scale conversion process SH3 to match the input skeletal information to the ratio of the motion target, and operates the skeletal information. This operation is performed with reference to parameters P prepared according to the difference in ratio. By generating and transmitting the motion/display instruction ST7 using the skeletal information after performing such an operation, the motion target can create an unnatural impression in accordance with the motion represented by the detection target image KTG. It will work like it doesn't.

Note that the posture management system H3 may also prepare the model data TD, and generate and output text information. Thereby, for example, information regarding the quality of the posture at important points or the posture to be taken next may be transmitted to the detection target KT in a timely manner. When transmitting information about the next posture to be taken, the model data TD may be used to determine whether the detection target image KTG has reached a posture for which text information should be generated.
Another motion in the above-mentioned motion target is performed by comparison with one posture represented by one model data TD. However, other motions may be performed in comparison with motions involving posture.

FIG. 2 is a diagram illustrating an example of a motion including a posture for causing the motion target to perform another motion. Each circle drawn in FIG. 2 represents a different joint.
The example shown in FIG. 2 assumes that a ninja throws a shuriken. The action of throwing a shuriken is when throwing a shuriken with the left hand. The diagram on the left side of FIG. 2 shows an example of the posture taken when throwing a shuriken. In this posture, the elbow of the right arm is bent, the right hand holding the shuriken is positioned near the waist at waist height, and the left hand is grasping the shuriken. Hereinafter, this attitude will be referred to as an "initial assumed attitude" for convenience.

On the other hand, the diagram on the right side of FIG. 2 shows an example of the posture after throwing a shuriken. This posture is an example of the posture after moving the left hand toward the right and throwing the shuriken from the initial assumed posture shown in the diagram on the left side of FIG. To throw a shuriken, the body is rotated counterclockwise when viewed from above.

In order to throw a shuriken held in the left hand, it is necessary to make a relatively large movement with the left hand, as shown in Figure 2. Therefore, by checking the movement of the hand assumed to be holding a shuriken from the initial assumed posture as the starting point, it is possible to determine whether or not the detection target KT has performed a motion of throwing a shuriken. Since such a determination is possible, if it is determined that the action of throwing a shuriken has been performed, text information can be generated to emit, for example, the wind noise of the thrown shuriken or a sound effect. becomes possible. Emitting such sounds makes it possible to provide an environment that provides a higher sense of realism, especially in games that require physical movement. The emission of such a sound itself can be positioned as an action different from the action of throwing a shuriken, which was a part of the detection target image KTG. In addition to determining whether or not the above-described action of throwing a shuriken has been performed, it is also possible to further determine the position at which the shuriken is released and the direction in which it is thrown.

When it is possible to detect a movement as shown in FIG. 2, the model data TD may include, for example, the following data. In other words, the first condition to be considered as the initial assumed posture, the second condition to consider that the specified movement has been performed from the initial assumed posture, and the time from when the first condition is no longer satisfied until the second condition is satisfied. Model data TD may be prepared that includes the range, etc. as data.

In a game or the like, when it is possible to detect the action of throwing a shuriken, the action target may be an avatar image of an opponent or the like. In such an action target, by confirming the action of throwing a shuriken, the user may make a comment regarding the throwing of the shuriken, and may also be made to perform an action in response to the throwing of the shuriken. In other words, by generating one piece of text information, one or more pieces of text information may also be generated.

Because of this, various considerations have been made regarding the movement of the detection target image KTG that generates text information, the content of the text information generated by that movement, and the operation control after outputting the generated text information. It will be done. For example, specifications may be designed by focusing on movements that assume the use of weapons other than shuriken. However, regardless of the specifications, not only the range of expression by the motion of the motion target becomes wider, but also the amount of information provided to the detection target KT becomes larger. As a result, it becomes possible to further increase the satisfaction level of the detection target KT.

Hereinafter, a specific implementation method for providing this service illustrated in FIG. 1 will be explained in detail with reference to FIGS. 3 to 7.
FIG. 3 is a diagram illustrating an example of a network environment to which an AP (application) server according to an embodiment of the information processing apparatus of the present invention is connected.

The AP server 1 is an information processing device installed by a service provider SK that provides this service. Service provider SK contracts with organizations such as companies that provide entertainment such as games, and provides this service to contracted organizations to help increase the satisfaction level of visitors who use those organizations. do.
In FIG. 3, the AP server 1 is shown as being installed within the service providing company SK, but it may be installed in another location. For example, it may be provided by a cloud service.

The AP server 1 is actually connected to the network N via another information processing device such as a proxy. However, here, for the sake of explanation and convenience, the information processing device interposed between the AP server 1 and the network N will be ignored. In other words, it is assumed that the AP server 1 is directly connected to the network N. The network N includes, for example, the Internet.

Customer company KK is an organization that has a contract with service provider SK. This customer company KK provides a place where users can play games using their bodies. This service makes it possible to provide a more satisfying environment to users who use the site. Hereinafter, customer company KK will be used as a general term for the contracted organizations. There are multiple customer companies KK.

In addition to the camera C, the customer company KK is equipped with a server 2, a projector 3, a sound system 4, and a microphone (abbreviated as "microphone" in FIG. 3. This abbreviation will be used hereinafter) 5. All except server 2 are installed in one place where games and the like can be played. Therefore, it is common for all servers except server 2 to exist in plural numbers.

FIG. 4 is a diagram illustrating an example of a place for games and the like.
A place for games and the like (hereinafter referred to as a "game place") is a space where a screen SC is installed or can be installed, as shown in FIG. In that space, a projector 3 capable of projecting onto a screen SC, a sound system 4 capable of emitting various sounds, a camera C, and a microphone 5 are installed. The camera C is installed so that it can image the user from the screen SC side.

The server 2 is connected to a camera C and a microphone 5. The projector 3 and sound system 4 are connected to the AP server 1 via a network N.
The sound system 4 is a system that includes a speaker that is a sound emitting device, and also includes an information processing device that functions as a terminal that can communicate via the network N. Thereby, the sound system 4 causes the speaker to emit sound based on the audio signal transmitted from the AP server 1.

The projector 3 is an information processing device equipped with a communication function via the network N, and generates a screen to be projected based on a video signal transmitted from the AP server 1, and projects the generated screen on the screen SC. I can do it.
Camera C is used to take video of users of the game hall. This photographic result is transmitted from the server 2 to the AP server 1 as image data (video information). All or at least one of these users is the detection target KT. The camera C is installed on the screen SC side because it is assumed that the user who is the detection target KT faces the screen SC and plays a game in which he moves his body.

The microphone 5 is installed to pick up sounds emitted by users of the game hall. The audio signal output from the microphone 5 is converted into audio information by the server 2 and transmitted to the AP server 1.

FIG. 5 is a block diagram showing an example of the hardware configuration of the AP server 1 according to an embodiment of the information processing apparatus of the present invention. Next, with reference to FIG. 5, an example of the hardware configuration of the AP server 1 will be specifically described. Note that this configuration example is just an example, and the hardware configuration of the AP server 1 is not limited to this.

As shown in FIG. 5, the AP server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input/output interface 15, and an output It includes a section 16, an input section 17, a storage section 18, a communication section 19, and a drive 20.

The CPU 11 executes various processes according to programs recorded in the ROM 12 and programs loaded into the RAM 13 from the storage unit 18 . The programs loaded from the storage unit 18 into the RAM 13 include, for example, an OS and various application programs that run on the OS. The various application programs include one or more programs developed for providing this service.

The RAM 13 also appropriately stores data necessary for the CPU 11 to execute various processes. The data also includes various programs executed by the CPU 11.
The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input/output interface 15 is also connected to this bus 14 . An output section 16 , an input section 17 , a storage section 18 , a communication section 19 , and a drive 20 are connected to the input/output interface 15 .

The output unit 16 has a configuration including, for example, a display such as a liquid crystal display. The output unit 16 displays various images or various screens under the control of the CPU 11. The output unit 16 may be installed in the AP server 1, or may be connected as necessary. In other words, the output unit 16 is not an essential component.

The input unit 17 includes various hardware buttons such as a keyboard. The configuration may include one or more pointing devices, such as a mouse. The operator can input various information via the input section 17. This input unit 17 may also be installed in the AP server 1, or may be connected as necessary. In other words, the input section 17 is also not an essential component.

The storage unit 18 is, for example, a hard disk device or an auxiliary storage device such as an SSD (Solid State Drive). Data with a large amount of data is stored in this storage section 18.
The communication unit 19 enables communication with other information processing devices via the network N. The server 2, projector 3, and information processing devices that constitute the sound system shown in FIG. 3 all correspond to other information processing devices.

The drive 20 is a device into which a removable medium 25 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory card can be attached and detached. The drive 20 is capable of reading information from and writing information to the removable medium 25 mounted thereon, for example. Thereby, the program recorded on the removable medium 25 can be stored in the storage unit 18 via the drive 20. Furthermore, the removable medium 25 attached to the drive 20 can be used as a copy destination or a movement destination of various data stored in the storage unit 18.

Application programs developed for this service may be recorded on the removable media 25 and distributed. It may also be possible to distribute it via network N or the like. For this reason, the recording medium that records the application program may be one that is installed or installed in an information processing device that is directly or indirectly connected to the network N, or one that is installed in an externally accessible device. Alternatively, it may be attached.

The hardware resources included in the AP server 1 are controlled by various programs including application programs. As a result, the AP server 1 can provide this service to the customer company KK.

FIG. 6 is a functional block diagram showing an example of a functional configuration implemented on the AP server 1 according to an embodiment of the information processing apparatus of the present invention. Next, an example of the functional configuration implemented on the AP server 1 will be described in detail with reference to FIG. 6.
Here, to avoid confusion, it is assumed that there is only one game field. Further, it is assumed that a game is played in a game arena. It is assumed that both the avatar image and the robot, which are operation targets, are operated using skeleton information generated from a detection target image (hereinafter also referred to as a detection target image KTG).

On the CPU 11 of the AP server 1, as shown in FIG. 6, the functional configuration includes a setting section 111, an image extraction section 112, a skeleton detection section 113, a posture evaluation section 114, a text conversion section 115, a video generation section 116, A posture control section 117, a voice recognition section 118, a conversation processing section 119, an image recognition section 120, and a face recognition section 121 are realized.

These are realized by the CPU 11 executing various programs including application programs developed for providing this service. As a result of this, or in order to provide this service, the storage unit 18 includes a reference image storage unit 181, a character image storage unit 182, a background image storage unit 183, a dictionary storage unit 184, and a control information storage unit 185. be done.

The setting unit 111 is a function for setting a game to be played at the game venue, starting or ending the set game, various settings for playing the set game, and the like. Various requests for these settings are transmitted from the server 2, for example.

In FIGS. 3 and 4, a device that enables such settings is omitted. This device is connected to a server 2, and the server 2 processes requests from this device, generates and sends requests to be sent to the AP server 1.
For example, from before the game starts until the game ends, the server 2 causes the camera C to shoot a video, and continues to send image data (video information) obtained by shooting the video to the AP server 1. . During this time, audio information obtained by converting the audio signal from the microphone 5 is also transmitted from the server 2 to the AP server 1.

The image extraction unit 112 specifies the range of the detection target image KTG in the image represented by the image data received from the server 2, and extracts the specified range. Extraction of this range is performed, for example, by generating still image data from image data at predetermined time intervals, and using the generated still image data. The image data extracted in this manner will hereinafter be referred to as "detection target image data."

The skeleton detection unit 113 generates skeleton information from the detection target image KTG represented by the detection target image data, for example, for each detection target image data (see, for example, Patent Document 1).
The posture evaluation section 114 uses the skeleton information generated by the skeleton detection section 113 to evaluate the posture of the detection target image KTG or the movement including the posture. This evaluation is basically the same as the difference AI analysis SH2 with respect to the model in FIG. 1, for example, or it is based on the initially assumed posture and subsequent movements as shown in FIG. In order to perform this evaluation, reference image information corresponding to the model data TD is stored in the reference image storage section 181. Thereby, the corresponding reference image information is also used for the evaluation.

The text conversion unit 115 generates text information by converting the evaluation result by the posture evaluation unit 114. As explained with reference to FIGS. 1 and 2, the generated text information includes the reference image information actually used for evaluation, the action content represented by the reference image information, and the action represented by the detection target image KTG. Depends on the difference between the content. The dictionary storage unit 184 stores dictionaries for generating text information and the like in accordance with the assumed game. Thereby, for example, a sentence registered in a dictionary, a character string such as a phrase, or a combination of two or more character strings is generated as text information.

The moving image generation unit 116 generates an image (screen) to be projected on the projector 3, and causes the communication unit 19 to transmit the generated image as image data.
The video generation unit 116 also generates information for emitting music, sound effects, performance sounds, and audio that are expected to be heard by the detection target KT as necessary, and transmits the generated information to the communication unit 19. let This provides an environment in which the detection target KT can play games, etc. while various sounds are being emitted. Hereinafter, information for emitting various sounds will be collectively referred to as "audio information."

The character image storage unit 182 stores various image information for generating various character images including avatar images. The image information is, for example, image information of a three-dimensional representation of each part constituting the avatar image. The background image storage unit 183 stores image information for generating images other than various character images as background image information. Thereby, the video generation unit 116 generates a background image from various image information stored in the background image storage unit 183, and also generates a background image using the skeleton information and various image information stored in the character image storage unit 182. Generate each avatar image. After generating these, the moving image generation unit 116 generates one screen worth of images by arranging each avatar image on the background image.

The posture control unit 117 uses the skeleton information to generate posture control information for operating the robot. The posture control unit 117 allows the robot to move in accordance with the movement of the detection target image KTG instead of or together with the avatar image. The attitude control information is generated by referring to other control information stored in the control information storage section 185. This control information is for generating attitude control information, and the parameter P shown in FIG. 1 is stored in the control information storage section 185 as one of the control information.

The voice recognition unit 118 performs voice recognition by natural language processing using voice information received from the server 2, for example. Through this voice recognition, the voice recognition unit 118 generates a character string that is text information representing the content of the statement made by the detection target KT. The character string is generated by pattern matching to identify each character represented by the audio information. This character string corresponds to the first character string in this embodiment.
The conversation processing unit 119 regards the generated character string as the content of the utterance of the detection target KT, and generates a character string to be returned to the utterance content as a response sentence. To this end, the conversation processing unit 119 sequentially performs morphological analysis, syntactic analysis, semantic analysis, context analysis, and anaphoric analysis on the generated character string to identify the content represented by the character string. The conversation processing unit 119 uses this identification result to generate, for example, a character string associated with the identification result as a response sentence. The character string generated as the response sentence corresponds to the second character string in this embodiment.

The response sentence generated by the conversation processing unit 119 and the audio information for making the text information generated by the text conversion unit 115 sound are generated by the video generation unit 116 and transmitted from the communication unit 19, for example. Thereby, the detection target KT can not only see the image (video) projected on the screen SC by the projector 3, but also hear the sound for conversation etc. emitted by the sound system 4.
The emitted sounds include, in addition to voices, music, sound effects, performance sounds, and the like. Here, unless otherwise specified, audio information is used as a general term for information for emitting these sounds and various sounds including voices.
If the object to be operated is a robot, the posture control unit 117 generates the audio information. Thereby, even when the robot is operated, the detection target KT can hear sounds such as conversations.

The voice recognition unit 118 and the conversation processing unit 119 enable a conversation between the detection target KT and the character whose image is projected on the screen SC. Thereby, the detection target KT can proceed with the game while having a conversation with the character. Through this conversation, the detection target KT can select, for example, an unexpected action or a way to proceed with the game. As a result, for the detection target KT, not only is the sense of immersion in the game enhanced, but the range of ways to enjoy the game is expanded, and a higher degree of satisfaction can be obtained. The actions that the detection target KT did not think about are the appearance of other avatar images or character images (and actions after appearance) due to his own utterances, actions or changes in response to the utterances of his own avatar image, and the background. Changes in images, etc.
Note that both the speech recognition section 118 and the conversation processing section 119 employ well-known techniques.

The image recognition unit 120 recognizes an object included in the detection target image KTG or existing as an image in the vicinity. This object should be considered separate from the detection target KT. The object is mainly something that the detection target KT is wearing or holding in his hand. After recognizing this object image, the image recognition unit 120 identifies the action content of the object image by focusing on changes in the position, orientation, size, etc. of the object image. The content of the operation can also be specified by employing a well-known technique in the image recognition unit 120.
By specifying the content of the operation, the object image can be changed, for example, depending on how the object to be detected KT is handled. Specifically, if the object is a sword-shaped object that is considered to be a sword, for example, the expression of the object image is changed depending on the positional relationship with the detection target KT, and the object image is changed depending on the speed of swinging the sword-shaped object. You may also add effects to the image.

The face recognition unit 121 extracts a face part from the detection target image KTG, and recognizes the gender, age, etc. of the detection target KT. A well-known technique can also be adopted for this face recognition unit 121.
Note that the object may be a dedicated object equipped with an acceleration sensor, a communication device, and the like. When the detection results of such a sensor are used for image recognition by the image recognition unit 120, the operation of the object image can be controlled with higher precision. By transmitting information representing the type of object etc. from the communication device via the server 2, for example, the AP server 1 side can recognize the type of object held or worn by the detection target KT.

The object recognition result by the image recognition unit 120 can be used to generate an avatar image holding the object. Thereby, for example, if the detection target KT is holding a sword-shaped object in his hand, an avatar image holding a sword may be displayed. Alternatively, if there is an object thrown by the detection target KT, the motion of the thrown object may be expressed in a moving image. The image representing the object may be stored in the character image storage section 182, for example, and selected from the recognition result of the object or the motion content of the detection target image KTG specified by skeleton information. This selection may be made by the image recognition unit 120, for example. The determination of whether the detection target KT has an object is made if, for example, there are parts where the detection target image KTG and the object image are considered to overlap, and the degree of change in their size is greater than the setting. This can be done depending on whether it is occurring or not.
For example, when throwing a shuriken as shown in FIG. 2, the direction in which the shuriken is thrown and the position of the target can be estimated based on the direction in which the left hand is moved. Such estimation results may be reflected in the movement of the shuriken.

Such screen control can be realized by causing the video generation unit 116 to perform processing according to the response sentence generated by the conversation processing unit 119. The posture control of the robot can be realized by causing the posture control unit 117 to perform processing according to the response sentence. What is operated in response to the response text is basically an avatar image or a robot that is supposed to make the response statement.

The face recognition result by the face recognition unit 121 may be used to determine the type, including the gender, of the avatar to be displayed as an image, the facial expression, etc. In that case, it becomes possible to select a more desirable avatar for the detection target KT, and it also becomes possible to change the facial expression of the avatar according to the facial expression of the detection target KT.
All of the above-mentioned controls work so that the detection target KT can further enhance the sense of immersion in the game, and can also obtain a higher degree of satisfaction.

In the above functional configuration, the skeleton detection unit 113 corresponds to the skeleton information generation means in this embodiment. Similarly, the posture evaluation section 114 is an analysis means, the text conversion section is a text information generation means, the video generation section 116 and the posture control section 117 are operation control means, the voice recognition section 118 is a voice recognition means, and the conversation processing section 119 is a conversation Each corresponds to a processing means. The CPU 11 itself corresponds to image data acquisition means and audio information acquisition means.

FIG. 7 is a flowchart illustrating an example of a video display process executed by the CPU installed in the AP server 1, which is the information processing device according to the present embodiment. This video display process is a process that is executed in order to cause the detection target KT to play the game, for example, when the server 2 or the like requests the start of the game. This processing is realized by the CPU 11 executing an application program developed for providing this service. Finally, this moving image display process will be described in detail with reference to FIG.

In order to avoid confusion, the video display process whose flowchart example is shown in FIG. 7 is based on the assumption that one game field is shown in FIG. 4. The explanation will be based on this assumption. The following description assumes that moving image information (image data) captured by the camera C and audio information captured by the microphone 5 are transmitted from the server 2 to the AP server 1. It is assumed that the avatar image moves along with the movement of the detection target KT. The CPU 11 is assumed to be the entity that executes the processing.

First, in step S1, the CPU 11 determines whether or not video information has been received from the server 2. When video information is received, the determination in step S1 is YES and the process moves to step S2. If no video information has been received, the determination in step S1 is NO and the process moves to step S6.
In step S2, the CPU 11 generates a still image from the received video information, for example, and extracts the detection target image KTG from the still image. In subsequent step S3, the CPU 11 executes skeleton detection processing to generate skeleton information from the extracted detection target image KTG. In the next step S4, the CPU 11 evaluates the posture of the detection target image KTG from the generated skeleton information. Already generated skeletal information is also used for this posture evaluation, if necessary.

In step S5 following step S4, the CPU 11 converts the evaluation result into text by generating text information according to the posture evaluation result. After this text conversion, the process moves to step S6.
In step S6, the CPU 11 determines whether or not audio information has been received from the server 2. If audio information has been received, the determination in step S6 is YES and the process moves to step S7. If audio information has not been received, the determination in step S6 is NO and the process moves to step S10.

In step S7, the CPU 11 executes voice recognition processing using the received voice information and voice information received in the past. The voice recognition process here includes not only generating a character string from voice information but also specifying the content of the character string. In the subsequent step S8, the CPU 11 determines whether or not the content of one conversation's worth of utterances by the detection target KT has been identified. If the content of the statement for one conversation, that is, the content of the statement from the start of the statement to the end of the statement, can be specified, the determination in step S8 is YES and the process moves to step S9. If the utterance content for one conversation cannot be specified, the determination in step S8 is NO and the process moves to step S10.
Note that the determination of the end of the speech is made, for example, when the state in which the voice of the detection target KT cannot be confirmed continues for a set time. As a result, if the determination in step S8 is YES, the content finally identified in step S7 becomes the content uttered by the detection target KT.
In step S9, the CPU 11 (conversation processing unit 119) generates a character string to be returned to the detection target KT as a response sentence from the specified utterance content. After the generation, the process moves to step S10.

In step S10, the CPU 11 determines whether the game has ended. If an end event occurs, such as an instruction to end the game by the detection target KT or the like, or an event within the game that should end the game, the determination in step S10 becomes YES and the process moves to step S11. If the end event has not occurred, the determination in step S10 is NO and the process moves to step S12.

In step S11, the CPU 11 transmits to the server 2 an end screen for notifying the detection target KT of the end of the game. After transmitting this end screen, the video display process ends.
In step S12, the CPU 11 executes other processes necessary for progressing the game.
Some games present options such as items to the user during the game, and allow the user to select one of the presented options. Presentation of such options and selection from among the options are realized by executing other processing in step S12.

In subsequent step S13, the CPU 11 generates video information and audio information, and causes the generated video information and audio information to be transmitted to the projector 3 and the sound system 4, respectively. Thereafter, the process returns to step S1.
If an avatar image exists on the screen represented by the video information, the avatar image is generated using the skeleton information generated by executing the skeleton detection process in step S3. An image for one screen is generated by generating each character image and a background image in addition to each avatar image, and placing each avatar image and each character image on the background image. The text information generated in step S5 or the response sentence generated in step S9 is reflected in the generation of any avatar image or any character image as necessary. Thereby, the behavior of the avatar image can be changed by at least one of the generated text information and the response sentence. The video information generated in step S13 where such screen generation is performed is, for example, information generated by performing compression processing on the generated screen.

If text information is generated in step S5 or a response sentence is generated in step S9, the above-mentioned audio information in step S13 is generated.
The audio information to be transmitted is, for example, a digital audio signal. When transmitting a digital audio signal as an audio signal, the audio signal is generated by, for example, an application that performs text-to-speech conversion (hereinafter referred to as "speech conversion software"). In this case, in step S13, the text information to be converted or the response sentence is designated, the voice conversion software is called, and the voice signal is acquired. Sending of the acquired audio signal as audio information continues until there is no more audio signal to be transmitted. When sound is emitted by the sound system 4 via the server 2 equipped with this voice conversion software, text information and response sentences can be transmitted as voice information. For this reason, the audio information is not particularly limited.

If the object to be operated is a robot, posture control processing is executed instead of video display processing. The flow of the attitude control process is basically the same as the above-described video display process. The difference from the video display process is that posture control information is generated instead of video information. For this reason, a detailed explanation of the attitude control process will be omitted.

In the present embodiment, the explanation has been given focusing on a game played by moving the body, but as described above, the present invention allows the body to perform posture-requiring movements for which one or more postures to be taken are known in advance. It can be widely applied as long as it is carried out.
For example, sign language, which is performed by moving the hands and fingers, is used to express the content that is desired to be conveyed through a certain posture and certain movements from that posture. Therefore, the content expressed in sign language may be converted into text information using the skeleton information, and the text information may be output. Specifically, for example, the movement of a sign language may be reproduced with an action object, and the content of the sign language reproduced with the action object may be outputted as at least one of display and audio. It is desirable to have text information output for both people, taking into consideration that it will be used by both people who are hearing impaired and people who are not hearing impaired.

In sports, various types of running (sprint running, long distance running, hurdle competition, etc.), golf, etc. correspond to movements that require posture. In such sports, for example, while having the motion target perform the motion represented by the skeletal information, text information is generated to inform the detection target KT of areas to be improved or good points by comparison with the model image OG. However, the text information may be output as at least one of display and audio. Since it is difficult to convey text information in a timely manner to a running detection target KT, it is desirable to save video information representing the motion of the motion target together with the text information so that it can be played back.

Furthermore, by using the skeleton information, it becomes possible to timely specify the state of the detection target KT. Utilizing this fact, the future movement of the detection target KT may be estimated, and text information may be generated and output according to the estimation result. Thereby, for example, a warning may be given to the detection target KT who tries to touch an exhibit that should not be touched, a warning to the detection target KT about moving to a place where it should not be moved, etc. It is also conceivable that customers visiting a store are the KTs to be detected, and used to prevent criminal acts such as shoplifting by the KTs to be detected, or to catch KTs to be detected who have committed criminal acts. In either case, more information will be provided and convenience for users will be higher.

In estimating the motion, the moving speed, moving direction, etc. of the detection target image KTG may be estimated from the skeleton information, and the estimation results may be reflected in the motion of the motion target. By performing such estimation (prediction), it is expected that the response will be improved. When applied to detect dangerous behavior of the detection target KT, the detection can be performed more quickly.

This motion estimation may be used to control the motion of an avatar such as an opponent or a character to be confronted in a game such as a battle game or an RPG (Role-Playing Game). It is conceivable to have the opponent make defensive movements in accordance with the movements represented by the skeletal information. However, by estimating the movement, the movement when defending can be made more natural. By having the character to be confronted perform actions that correspond to the movements of the detection target KT, it is possible to give the detection target KT the impression that the character is confronting itself. If the movement estimation results are used to control the character's movement, the character's movement can be made more natural, giving the detection target KT a stronger impression that they are actually facing the opponent. . In either case, it is desirable to generate text information based on posture evaluation using skeletal information, to perform movements according to the generated text information, and to have natural conversation.

Further, the present invention can be applied to controlling the operation of an avatar or a robot in an unmanned store or an unmanned reception desk. Both the avatar and the robot are used to make a motion different from the motion of the detection target KT and provide a service desired by the detection target KT.
As an application, for example, it may be combined with image recognition technology to recognize the presence or absence of products, luggage, etc., and take different actions depending on the recognition result. More specifically, for example, when the detection target KT makes an action such as showing some kind of product, the type of the product may be specified, and the identification result may be reflected in the subsequent conversation. This is because there is a high possibility that the detection target KT who made an action to show the product has the intention of checking other products of a different color, other products in the same category, or inventory, etc. .

As mentioned above, the motion targets include those that operate in accordance with the motion of the detection target KT (hereinafter referred to as "first motion targets") and those that behave in a manner different from the motion of the detection target KT (hereinafter referred to as "first motion target"). It is roughly divided into two types: Therefore, the generated text information is also roughly divided into two types: that which assumes the first action object, and that which assumes the second action. This also applies to model data TD (reference image information).

Both the first operation target and the second operation target may be subjected to operation control. When controlling the two operations at the same time, text information may be generated assuming that it will be used to control the two operations. For example, when the detection target KT assumes a transformation pose represented by the model data TD, text information is generated, the first action target is output with the word "transformation", the second action target is stopped, or the set posture is output. It is also possible to cause the user to perform the following actions. When voice recognition technology and natural conversation (language) technology are also used, on the condition that the detection target KT utters "transformation", it is necessary to evaluate whether the detection target KT has taken a transformation pose and generate text information. You can also do it. In this case, the text information may be used for emitting sound effects for the first action object and for similar action control for the second action object. For this reason, various modifications can be made to the conditions for generating text information, the contents of the generated text information, and the operation control of the operation target using the text information.

1 AP server, 2 server, 3 projector, 4 sound system, 5 microphone, 11 CPU, 18 storage unit, 19 communication unit, 111 setting unit, 112 image extraction unit, 113 skeleton detection unit, 114 posture evaluation unit, 115 text conversion 116 Video generation section 116, 117 Posture control section, 118 Voice recognition section, 119 Conversation processing section 119, 120 Image recognition section, 121 Face recognition section, 181 Reference image storage section, 182 Character image storage section, 183 Background image storage Section, 184 Dictionary storage section, 185 Control information storage section, C camera, H1 Skeletal information generation system, H2 Posture evaluation system, H3 Posture management system, KK Customer company, KT Detection target, KTG Detection target image, OG Model image, SK Service provider, SY service provider system

Claims (7)

an image data acquisition means for acquiring image data obtained by imaging a detection target;
skeletal information generation means for generating skeletal information including position information of joints in the detection target image and relationship information indicating relationships between the joints, based on the detection target image represented by the image data;
analysis means for analyzing the motion content in the detection target image based on the skeleton information;
text information generation means for generating text information based on the analysis result of the operation content;
a motion control means for causing at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, to operate based on the text information;
An information processing device having: The action of the action target based on the text information by the action control means includes a speaking action by at least one of displaying and emitting sound.
The information processing device according to claim 1. The motion control means is capable of operating the motion target based on the skeleton information.
The information processing device according to claim 1 or 2. audio information acquisition means for acquiring audio information representing the audio emitted by the detection target;
voice recognition means that performs voice recognition using the voice information and generates a first character string from the voice information;
further comprising conversation processing means for generating a second character string to be a response to the first character string,
The operation control means outputs the second character string as the statement content in the statement operation of the operation target.
The information processing device according to any one of claims 1 to 3. The motion control means causes the motion target to perform a motion different from the motion represented by the skeleton information, based on the skeleton information.
The information processing device according to claim 1, 2, or 4. The text information generation means generates the text information representing the content of the sign language when the movement of the hand and finger in the detection target image is identified as a movement for sign language as a result of the analysis of the action content;
The motion control means uses the skeletal information to cause the motion target to perform a finger motion in accordance with the hand and finger motion, and output the text information.
The information processing device according to any one of claims 1 to 4. In the information processing device,
Generating skeletal information including position information of joints in the detection target image and relationship information indicating relationships between joints, based on a detection target image represented by image data obtained by imaging assuming the detection target,
Analyzing the motion content in the detection target image based on the skeleton information,
Generating text information based on the analysis result of the operation content,
Based on the text information, at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, is operated as an operation target;
A program that executes processing.

Images