JP6804781B1 - Information processing system, information processing method and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing system, an information processing method and a computer program capable of confirming an evaluation of one's own movement in real time during operation. SOLUTION: A first information processing device is an analysis that acquires a user's image captured by an imaging device and analyzes the image acquired by the acquisition unit in real time to analyze the user's movement in real time. An image display device having a unit, a comparison unit that compares a user's movement analyzed by the analysis unit with predetermined model data, and a screen generation unit that generates a result screen showing the result of comparison by the comparison unit. Is characterized in that the result screen generated by the screen generation unit is displayed in real time together with the user's image. [Selection diagram] Fig. 5

Description

The present invention relates to information processing systems, information processing methods and computer programs.

In recent years, due to growing health consciousness, more and more people are exercising at fitness clubs and at home.

The person who exercises generally imitates the movement of the instructor. In a fitness club or the like, an actual instructor stands in front of the sample, but at home, it is common to play a video recording the movement of the instructor as a sample.

Further, in recent years, a technique for comparing and evaluating the movement of an instructor and the movement of a user has been developed. Patent Document 1 describes a similarity evaluation method for evaluating the similarity of the motion to be evaluated with respect to the model motion, and discloses a technique for evaluating the similarity based on the comparison of the evaluation results.

In the above technique, the practice motion image of the practitioner is photographed by using a camera or the like attached to the mobile terminal. Then, after the practice is completed, the dance scoring result can be confirmed on the display screen of the mobile terminal.

However, in the above technique, there is a problem that the practitioner cannot confirm any evaluation during the practice.

Japanese Unexamined Patent Publication No. 2012-073935

Therefore, it is an object of the present invention to provide technical improvements that solve or alleviate at least some of the problems of the prior art described above. One of the more specific objects of the present invention is to provide an information processing system, an information processing method, and a computer program that allow a user to confirm the evaluation of his / her own movement in real time during operation.

The information processing system of the present invention is the first connected to an image pickup device for imaging a user, an image display device for displaying an image of the user captured by the image pickup device, and the image pickup device and the image device. An information processing system including an information processing device, in which the first information processing device analyzes an acquisition unit that acquires an image of a user captured by the imaging device and an image acquired by the acquisition unit in real time. , An analysis unit that analyzes user movements in real time, a comparison unit that compares the user movements analyzed by the analysis unit with predetermined model data, and a screen that generates a result screen showing the results of comparison by the comparison unit. The image display device includes a generation unit, and is characterized in that the result screen generated by the screen generation unit is displayed in real time together with the user's image.

The comparison unit compares the user's movements with predetermined model data for each part of the user's body, and the screen generation unit can generate a result screen showing the comparison result by the comparison unit for each part in real time. it can.

The screen generation unit can generate a result screen showing the result of comparison by the comparison unit in a line graph for each part in real time.

The screen generation unit can generate a result screen showing the result of comparison by the comparison unit in a score and / or radar chart in real time.

The image display device can further display the image of the model user corresponding to the model data together with the image of the user and the result screen.

The first information processing device is further connected to the second information processing device via a network, and the image of the model user can be acquired from the second information processing device.

The image pickup apparatus can include a first image pickup apparatus for imaging the front surface of the user and a second image pickup apparatus for imaging the side surface of the user.

In the information processing method of the present invention, the information processing device analyzes the acquisition step of acquiring the image of the user captured by the imaging device and the image acquired in the acquisition step in real time, and analyzes the movement of the user in real time. An analysis step, a comparison step for comparing the user's movement analyzed in the analysis step with predetermined model data, a screen generation step for generating a result screen showing the result of comparison in the comparison step, and an image display device. It is characterized in that the result screen generated in the screen generation step is executed together with an output step that outputs the result screen together with the user's image in real time.

The computer program of the present invention has an acquisition function for acquiring an image of a user captured by an imaging device in an information processing device, and an analysis that analyzes the image acquired by the acquisition function in real time and analyzes the movement of the user in real time. A function, a comparison function that compares the user's movement analyzed by the analysis function with predetermined model data, a screen generation function that generates a result screen showing the comparison result by the comparison function, and a screen on the image display device. It is characterized by realizing an output function that outputs the result screen generated by the generation function together with the user's image in real time.

According to the present invention, it is possible to provide technical improvements that solve or alleviate at least a part of the problems of the prior art described above. Further, according to the present invention, it is possible to provide an information processing system, an information processing method, and a computer program that allow a user to confirm the evaluation of his / her own movement in real time during operation.

It is a system block diagram which shows an example of the information processing system which concerns on embodiment of this invention. It is a system block diagram which shows an example of the information processing system which concerns on embodiment of this invention. It is a conceptual diagram which shows an example of the usage mode of the information processing system which concerns on embodiment of this invention. It is a block diagram which shows an example of the structure of the information processing apparatus which concerns on embodiment of this invention. It is a conceptual diagram which shows an example of the result screen displayed on the image display apparatus which concerns on embodiment of this invention. It is a conceptual diagram which shows an example of the screen displayed on the image display device which concerns on embodiment of this invention. It is a conceptual diagram which shows an example of the part of the user which concerns on embodiment of this invention. It is a system block diagram which shows an example of the information processing system which concerns on embodiment of this invention. It is a hardware block diagram which shows an example of the hardware structure of the information processing apparatus which concerns on embodiment of this invention. It is a flow figure which shows an example of the flow of the information processing method which concerns on embodiment of this invention. It is a circuit block diagram which shows an example of the circuit structure for realizing the computer program which concerns on embodiment of this invention.

First, the outline of the information processing system of the present invention will be described with reference to the drawings.

As shown in FIG. 1, the information processing system 100 according to the embodiment of the present invention includes an image pickup device 200, an image display device 300, and a first information processing device 400 connected to the image pickup device 200 and the image display device 300. To be equipped.

The image pickup device 200 is for taking an image of a user. The image pickup device 200 is not particularly limited as long as it can capture a user, but is a Web camera capable of acquiring an image captured by the first information processing device 400 via a predetermined network in real time. It is preferable to do so.

The imaging device 200 may include a first imaging device 210 and a second imaging device 220, as shown in FIG.

The first imaging device 210 is for imaging the front surface of the user, and the second imaging device 220 is for imaging one side surface of the user.

The image display device 300 is for displaying a user's image captured by the image pickup device 200. The image display device 300 is not particularly limited as long as it is for displaying the image captured by the image pickup device 200, but is a monitor such as a television device that can be connected to the first information processing device 400. It is preferable to do so. Further, as the image display device 300, an image display device 300 with an image pickup device 200 incorporating the image pickup device 200 may be adopted.

The information processing device 400, the image pickup device 200, and the image display device 300 can be connected by wire or wireless communication.

FIG. 3 shows an arrangement example of the image pickup device 200 (the first image pickup device 210 and the second image pickup device 220), the image display device 300, and the information processing device 400 in a certain space.

The space shown in FIG. 3 is a space provided in a section of the user U's home for performing exercise such as fitness realized by the information processing system 100 of the present invention.

Here, as shown in FIG. 4, the first information processing device 400 is connected to the image pickup device 200 and the image device 300, and includes the acquisition unit 410, the analysis unit 420, and the comparison unit 430. , And a screen generation unit 440.

The acquisition unit 410 acquires the image of the user U captured by the image pickup device 200.

The analysis unit 420 analyzes the image acquired by the acquisition unit 410 in real time, and analyzes the movement of the user U in real time. The method of such analysis will be described later.

The comparison unit 430 compares the movement of the user U analyzed by the analysis unit 420 with the predetermined model data. The result of the comparison by the comparison unit 430 can be shown as the degree of agreement as an example. The degree of this agreement can be indicated between 0% and 100%.

The screen generation unit 440 generates a result screen showing the result of comparison by the comparison unit 430.

FIG. 5 is an image diagram showing an example of the result screen 310. The details of the result screen 310 will be described later.

Then, as shown in FIG. 6, the image display device 300 displays the result screen 310 generated by the screen generation unit 440 together with the user's image 320 in real time.

According to the above configuration, it is possible to provide a technical improvement that solves or alleviates at least a part of the problems of the prior art described above. Specifically, the user can confirm the evaluation of his / her own movement in real time during the operation.

Subsequently, the details of the result screen 310 will be described.

The comparison unit 430 can compare the movement of the user U with the predetermined model data for each part of the body of the user U.

FIG. 7 is an image diagram showing a part of the user's body, and as an example, the head (Head), the right arm (Right arm), the right leg (Right leg), the left arm (Left arm), and the left leg. It is divided into parts (Left legs). In addition, the right arm is further divided into a right shoulder, a right elbow, and a right hand, and the right foot is further divided into a right knee and a right foot. It is divided into. Similarly, the left arm is further divided into a left shoulder, a left elbow, and a left hand, and the left leg is further divided into a left knee and a left foot. ) Is divided. In addition, these parts may be divided into finer parts.

At this time, the screen generation unit 440 generates a result screen 320 showing the result of comparison by the comparison unit 430 for each part in real time.

As shown in FIG. 5, the result screen 310 is like the comparison result 311 of the head, the comparison result 312 of the right arm, the comparison result 313 of the left arm, the comparison result 314 of the right leg, and the comparison result 315 of the left foot. The comparison results are shown for each part.

Then, as shown in FIG. 5, the screen generation unit 440 can generate a result screen 310 in which the result of the comparison by the comparison unit 430 is shown in a line graph for each part in real time.

The line graph may have a minimum value of 0 and a maximum value of 100, and the degree of coincidence may be represented as a line graph.

According to the above configuration, the user can confirm the evaluation of the movement of each part in detail in real time.

Further, as shown in FIG. 5, the screen generation unit 440 can generate a result screen 310 in which the result of the comparison by the comparison unit 430 is shown by the score 317 and / or the radar chart 318 in real time.

The score 317 can be the average value or the total value of the scores calculated for each part. The score of each part can be calculated from the average value of the degree of agreement described above.

The radar chart can be a graphic visualization of the score 317 .

According to the above configuration, the user can easily confirm the evaluation of his / her movement by scoring or visually.

As shown in FIG. 6, the image display device 300 can further display the image 330 of the model user corresponding to the model data together with the image 320 of the user and the result screen 310.

According to the above configuration, the user can simultaneously check his / her own image, the image of the model user to be modeled, and the result screen.

Further, as shown in FIG. 8, the first information processing device 400 may be further connected to the second information processing device 500 via a network.

Then, the image 330 of the model user shown in FIG. 6 may be acquired from the second information processing device 500. The acquired image 330 of the model user may be stored in the first information processing device 400 or may be streamed and reproduced.

According to the above configuration, the analysis processing of the image which has a large processing load and requires real-time performance is performed by the first local information processing apparatus 400, and the acquisition of the image 330 of the model user having a small processing load is performed on the Internet. It can be acquired in real time from the second information processing apparatus 500 via. Then, in the first information processing device 400, the result screen 310 analyzed by the own device, the user image 320 acquired from the image pickup device 200, and the model user image 330 acquired from the second information processing device 500 are displayed. It can be combined and displayed on the image display device 300.

Further, the second information processing device 500 may be directly connected to the image display device 300 without going through the first information processing device 400.

In addition, a plurality of images of the model user are stored in the second information processing device 500, a fitness menu is proposed based on information such as physical condition and mood input by the user, and images of the model user are proposed based on the menu. May be output.

Subsequently, the analysis method by the analysis unit 420 will be described in detail.

The first information processing device 400 stores in advance an image (moving image data) of a model user as teacher data. The analysis unit 410 decomposes the moving image into a still image and measures the pose (posture) of the model user for each frame by using a technique such as OpenPose (acquires the coordinate data of the joint). Then, the feature amount is extracted by LSTM and stored in the database. It is not necessary to extract such features in real time, and they may be prepared in advance.

Similarly, the analysis unit 410 extracts the feature amount in real time from the user's image (moving image data) acquired from the image pickup apparatus 200 by using the same method.

Then, the comparison unit 430 can determine the degree of matching of each joint by comparing these feature amounts. In addition, each part mentioned above can be defined based on the said joint.

<Hardware configuration>
Here, the hardware configuration of the first information processing apparatus 400 included in the information processing system 100 will be described with reference to FIG. The information processing device 400 includes a processor 401, a memory 402, a storage 403, an input / output interface (input / output I / F) 404, and a communication interface (communication I / F) 405. The components are connected to each other via bus B.

The information processing device 400 can realize the functions and methods described in the present embodiment by the cooperation of the processor 401, the memory 402, the storage 403, the input / output I / F 404, and the communication I / F 405. it can.

The processor 401 executes a function and / or a method realized by a code or an instruction contained in a program stored in the storage 403. The processor 401 is, for example, a central processing unit (CPU), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a microprocessor (microprocessor), a processor core (processor core), a multiprocessor (multiprocessor), an ASIC (Application-). Each implementation includes a specific integrated circuit), FPGA (Field Programmable Gate Array), etc., and is implemented by a logic circuit (hardware) or a dedicated circuit formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc. Each process disclosed in the form may be realized. Further, these circuits may be realized by one or a plurality of integrated circuits, and a plurality of processes shown in each embodiment may be realized by one integrated circuit. Further, the LSI may be referred to as a VLSI, a super LSI, an ultra LSI, or the like depending on the degree of integration.

The memory 402 temporarily stores the program loaded from the storage 403 and provides a work area for the processor 401. Various data generated while the processor 401 is executing the program are also temporarily stored in the memory 402. The memory 402 includes, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like.

The storage 403 stores the program. The storage 403 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, and the like.

The communication I / F 405 is implemented as hardware such as a network adapter, communication software, and a combination thereof, and transmits and receives various data via the network 600. The communication may be executed by wire or wirelessly, and any communication protocol may be used as long as mutual communication can be executed. The communication I / F 405 executes communication with another information processing device via the network 600. The communication I / F 405 transmits various data to another information processing device according to an instruction from the processor 401. Further, the communication I / F 405 receives various data transmitted from other information processing devices and transmits them to the processor 401.

The input / output I / F 404 includes an input device for inputting various operations to the information processing device 400 and an output device for outputting the processing result processed by the information processing device 400. The input / output I / F 404 may be integrated with the input device and the output device, or may be separated into the input device and the output device.

The input device is realized by any one or a combination of all kinds of devices capable of receiving an input from a user and transmitting information related to the input to the processor 401. The input device includes, for example, a hardware key such as a touch panel, a touch display, and a keyboard, a pointing device such as a mouse, a camera (operation input via an image), and a microphone (operation input by voice).

The output device outputs the processing result processed by the processor 401. The output device includes, for example, a touch panel, a speaker, and the like.

Subsequently, an example of the information processing method according to the embodiment of the present invention will be described.

In the information processing method of the present invention, as shown in FIG. 10, the information processing apparatus is made to execute the acquisition step S110, the analysis step S120, the comparison step S130, the screen generation step S140, and the output step S150.

The acquisition step S110 acquires an image of the user captured by the imaging device. The acquisition step S110 can be executed by the acquisition unit 410 described above.

The analysis step S120 analyzes the image acquired in the acquisition step in real time, and analyzes the movement of the user in real time. The analysis step S120 can be executed by the analysis unit 420 described above.

The comparison step S130 compares the movement of the user analyzed in the analysis step with the predetermined model data. The comparison step S130 can be executed by the comparison unit 430 described above.

The screen generation step S140 generates a result screen showing the result of the comparison in the comparison step. The screen generation step S140 can be executed by the screen generation unit 440 described above.

The output step S150 outputs the result screen generated in the screen generation step to the image display device in real time together with the user's image. The output step S150 can be executed by the output unit 450 (not shown) included in the information processing apparatus 400.

According to the above configuration, it is possible to provide a technical improvement that solves or alleviates at least a part of the problems of the prior art described above. Specifically, the user can confirm the evaluation of his / her own movement in real time during the operation.

Finally, the computer program of the present invention will be described.

The computer program of the present invention makes the information processing apparatus realize an acquisition function, an analysis function, a comparison function, a screen generation function, and an output function.

The acquisition function acquires the user's image captured by the imaging device.

The analysis function analyzes the image acquired by the acquisition function in real time, and analyzes the movement of the user in real time.

The comparison function compares the movement of the user analyzed by the analysis function with predetermined model data.

The screen generation function generates a result screen showing the result of comparison by the comparison function.

The output function outputs the result screen generated by the screen generation function to the image display device in real time together with the user's image.

The acquisition function, analysis function, comparison function, and screen generation function can be realized by the acquisition circuit 1410, the analysis circuit 1420, the comparison circuit 1430, and the screen generation circuit 1440 shown in FIG. It is assumed that the acquisition circuit 1410, the analysis circuit 1420, the comparison circuit 1430, and the screen generation circuit 1440 are realized by the acquisition unit 410, the analysis unit 420, the comparison unit 430, and the screen generation unit 440 described above, respectively. Details of each part are as described above.

According to the above configuration, it is possible to provide a technical improvement that solves or alleviates at least a part of the problems of the prior art described above. Specifically, the user can confirm the evaluation of his / her own movement in real time during the operation.

Further, in order to function as the information processing device according to the above-described embodiment, an information processing device such as a computer or a mobile phone can be preferably used. Such an information processing device stores a program describing processing contents that realize each function of the server device or the terminal device according to the embodiment in the storage unit of the information processing device, and the CPU of the information processing device stores the program. This can be achieved by reading and executing.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

Further, the method described in the embodiment includes, for example, a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), as a program that can be executed by a computer (computer). It can be stored in a recording medium such as a semiconductor memory (ROM, RAM, flash memory, etc.), or transmitted and distributed by a communication medium. The program stored on the medium side also includes a setting program for configuring the software means (including not only the execution program but also the table and the data structure) to be executed by the computer in the computer. A computer that realizes this device reads a program recorded on a recording medium, constructs software means by a setting program in some cases, and executes the above-mentioned processing by controlling the operation by the software means. The recording medium referred to in the present specification is not limited to distribution, and includes a storage medium such as a magnetic disk or a semiconductor memory provided in a device connected inside a computer or via a network. The storage unit may function as, for example, a main storage device, an auxiliary storage device, or a cache memory.

100 Information processing system 200 Image pickup device 210 First image pickup device 220 Second image pickup device 300 Image display device 400 First information processing device 500 Second information processing device

Claims (9)

An imaging device for imaging a user, an image display device for displaying an image of the user captured by the imaging device, and a first information processing device connected to the imaging device and the image display device. An information processing system equipped with,
The first information processing device is
An acquisition unit that acquires an image of the user captured by the image pickup device, and
An analysis unit that analyzes the image acquired by the acquisition unit in real time and analyzes the movement of the user in real time.
A comparison unit that compares the movement of the user analyzed by the analysis unit with predetermined model data.
It has a screen generation unit that generates a result screen showing the result of comparison by the comparison unit.
The image display device displays the result screen generated by the screen generation unit and the image of the model user corresponding to the predetermined model data in real time together with the image of the user.
Image of the model user, which is stored in the second information processing apparatus connectable to the first information processing apparatus,
The image display device is an information processing system that displays an image of a model user corresponding to the predetermined model data streamed and reproduced in the first information processing device . Image of the model user information according to claim 1, characterized in that is to be determined from among the images of a plurality of models users based on the proposed menu based on the information input by the user Processing system. The image pickup device
A first imaging device for imaging the front of the user, and
Including a second imaging device for imaging the side of the user.
The image captured by the first image pickup device is used for analysis in the analysis unit and display in the image display device.
The information processing system according to claim 1 or 2, wherein the image captured by the second imaging device is used for analysis in the analysis unit. The comparison unit compares the movement of the user with the predetermined model data for each part of the body of the user.
The information processing system according to claim 1, 2 or 3, wherein the screen generation unit generates a result screen showing the result of comparison by the comparison unit for each part in real time. The information processing system according to claim 4, wherein the screen generation unit generates a result screen in which the result of comparison by the comparison unit is shown in a line graph for each part in real time. The information processing system according to any one of claims 1 to 3, wherein the screen generation unit generates a result screen showing the result of comparison by the comparison unit in a score and / or a radar chart in real time. .. The information processing system according to claim 5, wherein on the result screen, a marker indicating a current time point is also displayed on the line graph. For the first information processing device,
The acquisition step of acquiring the user's image captured by the image pickup device, and
An analysis step that analyzes the image acquired in the acquisition step in real time and analyzes the movement of the user in real time.
A comparison step that compares the movement of the user analyzed in the analysis step with predetermined model data.
A screen generation step that generates a result screen showing the result of comparison in the comparison step,
The result screen generated in the screen generation step and the image of the model user corresponding to the predetermined model data are output to the image display device in real time together with the image of the user.
Image of the model user, which is stored in the second information processing apparatus connectable to the first information processing apparatus,
The image display device is an information processing method for executing an output step of displaying an image of a model user corresponding to the predetermined model data streamed and reproduced in the first information processing device . For the first information processing device,
An acquisition function that acquires the user's image captured by the image pickup device,
An analysis function that analyzes the image acquired by the acquisition function in real time and analyzes the movement of the user in real time.
A comparison function that compares the movement of the user analyzed by the analysis function with predetermined model data, and
A screen generation function that generates a result screen showing the result of comparison by the comparison function, and
The result screen generated by the screen generation function and the image of the model user corresponding to the predetermined model data are output to the image display device in real time together with the image of the user.
Image of the model user, which is stored in the second information processing apparatus connectable to the first information processing apparatus,
The image display device is a computer program that realizes an output function for displaying an image of a model user corresponding to the predetermined model data streamed and reproduced in the first information processing device .

Images