JP2023045906A - Information processing device, information processing program, and information processing method - Google Patents

Abstract

To provide an information processing device, a method, and a program capable of updating a training menu according to a change in athletic ability of a user.SOLUTION: In a menu management system, an administrative server includes: a control unit including an acquisition section 11a for acquiring a user training image, an analysis section 11b for analyzing user motion from the training image; an evaluation section 11c for evaluating, on the basis of the analysis result, proficiency degree, and an update section 11d for updating, on the basis of the evaluation result and a request, the user training menu; a transmission unit that transmits the training image to a trainer terminal; a first reception unit that receives a training scoring result and feedback to the user from the trainer terminal; and a second reception unit that receives a request for the feedback from a user terminal.SELECTED DRAWING: Figure 21

Description

The present invention relates to an information processing device and the like that updates a training menu.

Exercise facilities such as fitness gyms create training menus suitable for each user. For example, Patent Literature 1 proposes an exercise support system that presents a training menu (exercise menu) that reflects the user's health condition.

JP 2008-299492 A

However, the exercise support system described above does not assume that the training menu is updated according to changes in the user's exercise ability. The present invention has been made in view of such circumstances. The object is to provide an information processing device or the like that can update a training menu according to changes in the user's athletic ability.

An information processing apparatus according to an aspect of the present application includes an acquisition unit that acquires a training image of a user, an analysis unit that analyzes the user's motion from the training image, an evaluation unit that evaluates progress based on the analysis result, an evaluation and an updating unit that updates the user's training menu based on the results.

According to one aspect of the present application, it is possible to update the training menu according to changes in the user's athletic ability.

1 is an explanatory diagram showing a configuration example of a menu management system; FIG. It is a block diagram which shows the hardware structural example of a management server. 3 is a block diagram showing a hardware configuration example of a user terminal; FIG. 3 is a block diagram showing a hardware configuration example of a trainer terminal; FIG. It is explanatory drawing which shows the example of user DB. FIG. 4 is an explanatory diagram showing an example of a trainer DB; FIG. 4 is an explanatory diagram showing an example of a menu DB; It is explanatory drawing which shows the example of training log|history DB. FIG. 4 is an explanatory diagram showing an example of a moving image DB; It is explanatory drawing which shows the example of goods DB. FIG. 5 is an explanatory diagram showing an example of a recommendation history DB; FIG. 10 is an explanatory diagram showing an example of an analysis result DB; 7 is a flowchart illustrating an example of update processing procedures; 6 is a flow chart showing an example of a procedure of training processing; 4 is a flow chart showing an example of a diagnostic process procedure; 9 is a flow chart showing an example of the procedure of menu update processing; 10 is a flow chart showing an example of a recommended process procedure; 6 is a flow chart showing an example of a procedure of evaluation processing; 7 is a flow chart showing an example of a reference process procedure; FIG. 5 is an explanatory diagram showing an example of an output screen; It is a block diagram which shows the functional structural example of a management server.

Embodiments will be described below with reference to the drawings. FIG. 1 is an explanatory diagram showing a configuration example of a menu management system. The menu management system 100 includes a management server 1 , a display device 2 , a camera 3 , a vending machine 4 , a user terminal 5 , a trainer terminal 6 and a central server 7 . A management server 1, a display device 2, a camera 3, a vending machine 4, a user terminal 5, a trainer terminal 6, and a central server 7 are connected by a network N so as to be able to communicate with each other.

The management server 1 is composed of a server computer, a workstation, a PC (Personal Computer), and the like. The management server 1 is installed in a fitness gym. A plurality of management servers 1 may be installed in a fitness gym. Instead of the management server 1 , a thin client may be installed in the fitness gym, and the processing performed by the management server 1 may be performed by the central server 7 . Also, the management server 1 may be composed of a multicomputer consisting of a plurality of computers, a virtual machine or a quantum computer virtually constructed by software. Furthermore, the functions of the management server 1 may be realized by a cloud service.

The display device 2 is composed of a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. The display device 2 presents a training menu to the user.

The camera 3 has an imaging element such as a CCD element or a CMOS element. A camera 3 photographs a user who is training using training equipment. It is desirable to install a plurality of cameras 3 in each fitness gym. When a plurality of cameras 3 are installed, each camera is given an ID so that it can be identified. The ID is called a camera ID.

The vending machine 4 provides users with soft drinks, jelly drinks, supplements, and the like. The vending machine 4 can communicate with the management server 1, and can provide the product specified by the management server 1 to the user.

The user terminal 5 is composed of a smart phone, a tablet computer, a notebook PC, or the like. A user terminal 5 is used by a user who uses the fitness gym. The user terminal 5 communicates with the management server 1 and displays the user's training history and the user's training image captured by the camera 3 .

The trainer terminal 6 is composed of a smart phone, a tablet computer, a notebook PC, or the like. The trainer terminal 6 is used by a trainer who works at the fitness gym. The trainer uses the trainer terminal 6 to create or update a training menu for the user. In addition, training images of the user are viewed and training advice is given to the user.

The central server 7 is composed of a server computer, a workstation, a PC, and the like. The central server 7 generates, updates, etc. learning models. The central server 7 may be composed of a multicomputer consisting of a plurality of computers, a virtual machine virtually constructed by software, or a quantum computer. The functions performed by the central server 7 may be provided by a cloud service.

Instead of preparing the display device 2 and the camera 3 individually, a device in which these are integrated may be adopted. For example, a device including a mirror, a display (display device 2), a camera 3, and the like that captures the user during training and having the functions of the user terminal 5, such as a smart mirror, may be employed. Not limited to smart mirrors, a mirror with a tablet function or a digital signage with a mirror function may be adopted.

FIG. 2 is a block diagram showing a hardware configuration example of a management server. The management server 1 includes a control section 11 , a main storage section 12 , an auxiliary storage section 13 , a communication section 15 and a reading section 16 . A bus B connects the control unit 11 , the main storage unit 12 , the auxiliary storage unit 13 , the communication unit 15 and the reading unit 16 .

The control unit 11 has one or more arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The control unit 11 reads and executes a control program 1P (information processing program, program product) stored in the auxiliary storage unit 13 to perform various information processing, control processing, etc. related to the management server 1, and various Realize the function part.

The main storage unit 12 is SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, or the like. The main storage unit 12 mainly temporarily stores data necessary for the control unit 11 to execute arithmetic processing.

The auxiliary storage unit 13 is a hard disk, SSD (Solid State Drive), or the like, and stores the control program 1P and various DBs (Databases) necessary for the control unit 11 to execute processing. Auxiliary storage unit 13 stores user DB 131 , trainer DB 132 , menu DB 133 , training history DB 134 , video DB 135 , product DB 136 , recommendation history DB 137 and analysis result DB 138 . The auxiliary storage unit 13 also stores a motion analysis model 141 , a motion diagnosis model 142 and an update model 143 . The auxiliary storage unit 13 may be a storage device externally connected to the management server 1 separately. Various DBs and the like stored in the auxiliary storage unit 13 may be mirrored on the central server 7 . Also, various DBs and the like stored in the auxiliary storage unit 13 may be stored in a database server or cloud storage different from the management server 1 .

The communication unit 15 communicates with the user terminal 5, the trainer terminal 6 and the like via the network N. FIG. Alternatively, the control unit 11 may use the communication unit 15 to download the control program 1P from another computer via the network N or the like and store it in the auxiliary storage unit 13 .

The reader 16 reads portable storage media 1a including CD (Compact Disc)-ROM and DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the auxiliary storage unit 13 . Alternatively, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

FIG. 3 is a block diagram showing a hardware configuration example of a user terminal. The user terminal 5 is composed of a smart phone, a tablet computer, a notebook computer, or the like. The user terminal 5 includes a control section 51 , a main storage section 52 , an auxiliary storage section 53 , a communication section 54 , an input section 55 and a display section 56 . Each configuration is connected by a bus B.

The control unit 51 has one or more arithmetic processing units such as CPU, MPU, and GPU. The control unit 51 provides various functions by reading and executing the control program 5P (program product) stored in the auxiliary storage unit 53 .

The main storage unit 52 is SRAM, DRAM, flash memory, or the like. The main storage unit 52 mainly temporarily stores data necessary for the control unit 51 to execute arithmetic processing.

The auxiliary storage unit 53 is a hard disk, SSD, or the like, and stores various data necessary for the control unit 51 to execute processing. The auxiliary storage unit 53 may be an external storage device that is separate from the user terminal 5 and externally connected. Various DBs and the like stored in the auxiliary storage unit 53 may be stored in a database server or cloud storage.

The communication unit 54 communicates with the management server 1 via the network N. FIG. Alternatively, the control unit 51 may use the communication unit 54 to download the control program 5P from another computer via the network N or the like and store it in the auxiliary storage unit 53 .

The input unit 55 is a keyboard or mouse. The display unit 56 includes a liquid crystal display panel and the like. The display unit 56 displays the training menu output by the management server 1 and the like. Also, the input unit 55 and the display unit 56 may be integrated to form a touch panel display. Note that the user terminal 5 may display on an external display device.

FIG. 4 is a block diagram showing a hardware configuration example of the trainer terminal. The trainer terminal 6 is composed of a smart phone, a tablet computer, a notebook computer, or the like. The trainer terminal 6 includes a control section 61 , a main storage section 62 , an auxiliary storage section 63 , a communication section 64 , an input section 65 and a display section 66 . Each configuration is connected by a bus B. The control unit 61, the main storage unit 62, the auxiliary storage unit 63, the communication unit 64, the input unit 65, and the display unit 66 are the control unit 51, the main storage unit 52, the auxiliary storage unit 53, the communication unit 54, the input Since it has the same configuration as each of the unit 55 and the display unit 56, its description is omitted.

Next, a database (DB) used by the menu management system 100 will be described. FIG. 5 is an explanatory diagram showing an example of the user DB. The user DB 131 stores information on users who use fitness gyms. The user DB 131 includes a user ID column, a name column, an age column, a gender column, a height column, a weight column, and a body fat percentage column. The user ID column stores user IDs that uniquely identify users. The name column stores the user's name. The age column stores the user's age. The gender column stores the gender of the user. The height column stores the user's height. The weight column stores the user's weight. Body fat percentage column Stores the user's body fat percentage.

FIG. 6 is an explanatory diagram showing an example of the trainer DB. The trainer DB 132 stores information about trainers who give training advice and the like to fitness gym users. The trainer DB 132 includes a trainer ID column, name column, age column, and gender column. The trainer ID column stores trainer IDs that uniquely identify trainers. The name column stores the name of the trainer. The age column stores the age of the trainer. The gender column stores the gender of the trainer.

FIG. 7 is an explanatory diagram showing an example of the menu DB. The menu DB 133 stores training menus created for each user. The menu DB 133 includes a menu ID column, a user ID column, a created date/time column, an execution count column, a trainer ID column, an order number column, a type column, and a training amount column. The menu ID column stores menu IDs that uniquely identify training menus. The user ID column stores the user ID of the user who is the object of the menu. The created date and time column stores the date and time when the menu was created. The execution count column stores the number of times the user has executed training according to the menu. The Trainer ID column stores the Trainer ID of the trainer involved in the menu. Participating in the menu means, for example, creating or updating the menu, or providing comments or advice regarding the menu to the user. The order number column stores order numbers indicating the execution order of a plurality of trainings included in the training menu. The type column stores the type of each training. The training amount column stores the amount of each training to be performed, such as execution time, number of executions, load to be applied, and the like.

FIG. 8 is an explanatory diagram showing an example of a training history DB. The training history DB 134 stores a history of training performed by the user. The training history DB 134 includes a history ID column, a training date column, a start time column, an end time column, a menu ID column, a user ID column, a video ID column, and an evaluation column. The history ID column stores a history ID that uniquely identifies a history. The training date column stores the date and time when the user performed the training. The start time column stores the time when the user started training. The end time column stores the time when the user finished training. The menu ID column stores the menu ID of the training menu performed by the user. The user ID column stores the user ID of the user who performed the training menu. The moving picture ID column stores a moving picture ID specifying a moving picture obtained by photographing a pattern of executing the training menu with the camera 3 . The evaluation column stores the evaluation of training menu execution results. The evaluation is, for example, a 5-step evaluation. The evaluation value is A, B, C, D or E, with A being the highest evaluation and E being the lowest evaluation.

FIG. 9 is an explanatory diagram showing an example of a moving image DB. The moving image DB 135 stores moving images obtained by photographing the pattern of execution of the training menu with the camera 3. - 特許庁The moving picture DB 135 includes a moving picture ID column, a sequence number column, a camera ID column and a moving picture column. The movie ID column stores movie IDs that specify movies. Here, one moving image ID is assigned to one execution of the training menu. The order number column stores the order numbers of moving images. The order number corresponds to the order number of each training included in the training menu. The camera ID column stores the camera ID that identifies the camera that shot each workout. When the training pattern is shot with three cameras, the camera IDs for the three cameras are stored in the camera ID column. The animation sequence stores the entity of the animation data. Since the moving image data is binary data, it is shown as Binary in FIG.

FIG. 10 is an explanatory diagram showing an example of the product DB. The product DB 136 stores information on products recommended for users who have completed training. The product is available at vending machine 4 . The product DB 136 includes a product ID column, a product name column, and a fatigue level column. The product ID column stores product IDs that uniquely identify products. The product ID may be uniquely assigned by the menu management system, or a versatile GTIN (Global Trade Item Number) or JAN code (Japanese Article Number) may be used. The product name column stores product names. The degree-of-fatigue column stores the degree of fatigue of users to whom the product should be recommended. The degree of fatigue is, for example, 10 levels, with 10 indicating the most fatigued and 1 indicating the least fatigued. In addition, although the fatigue degree is set to a single value, it is not limited to that. A plurality of scales may be used to determine the degree of fatigue of the user. For example, recommended products may be selected from a plurality of fatigue levels, such as the fatigue level determined from the user's facial expression, the user's fatigue level determined from the user's pulse wave, and the user's fatigue level determined from the user's heartbeat.

FIG. 11 is an explanatory diagram showing an example of a recommendation history DB. The recommendation history DB 137 stores a history of recommending products to users. The recommendation history DB 137 includes a recommendation ID column, a history ID column, an estimated fatigue level column, a product ID column, and a result column. The recommendation ID column stores a recommendation ID that uniquely identifies a recommendation history. The history ID column stores the history ID of the corresponding training history. The estimated fatigue level column stores the user's estimated fatigue level when a product is recommended to the user. The product ID column stores product IDs of recommended products. The result column stores whether or not the user purchased the recommended product.

FIG. 12 is an explanatory diagram showing an example of the analysis result DB. The analysis result DB 138 stores the result of analyzing the product purchase tendency from the product recommendation history. In the example shown in FIG. 12, the analysis result DB 138 stores user groups with a high possibility of purchasing each product. The analysis result DB 138 includes a product ID column, an age column, a sex column and a fatigue level column. The product ID column stores product IDs. The age column stores the age of the user who purchased the product most. The gender column stores the gender of the user who purchased the item most often. The degree-of-fatigue column stores the degree of fatigue after training of the user who purchased the product the most.

Next, the learning model will be explained. The motion analysis model 141 (first learning model) is a learning model that, when a human motion image is input, outputs an analysis result of the human motion. For example, the motion analysis model 141 is constructed using a Pose Proposal Network. The motion analysis model 141 outputs a skeletal model representing the motion of a human when a motion image of a human is input. Pose Proposal Network is a model that detects a person using an object detection algorithm, detects the joint points of each person, and estimates the pose by connecting the joint points. A moving image of the user's training obtained by shooting with the camera 3 is input to the motion analysis model 141, and the moving image of the skeletal model obtained by outputting is used to determine the range of motion of the user's predetermined parts during training, Operation speed can be obtained as an analysis result.

The motion diagnosis model 142 (second learning model) outputs the degree of progress of the motion based on the reference data when the analysis result of the human motion and the reference data are input. In the motion diagnosis model 142, if the reference data indicates an ideal motion, the degree of proficiency indicates the deviation from the ideal motion. If the reference data is past data of the same person, the amount of change in the movement range and movement speed in comparison with the past is the degree of progress. In this specification, the latter is assumed, and the degree of progress is determined based on the movement of the user's body during training and the movement of the body during previous training. The motion diagnosis model 142 is composed of, for example, a CNN (Convolutional Neural Network).

As training data for the motion diagnosis model 142, a plurality of videos of the same user during training are prepared. A trainer or the like selects two moving images from which the user's progress can be confirmed by comparing them in chronological order, and labels the degree of progress. The two moving images of the skeletal model obtained by inputting the two moving images into the motion analysis model 141 and the label indicating the degree of proficiency become the training data. Parameters used for arithmetic processing in the intermediate layer so that, when two moving images constituting training data are input to the motion diagnosis model 142, the degree of proficiency output by the motion diagnosis model 142 approaches the value of the correct label. to optimize. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions used in each neuron, and the like. Although the parameter optimization method is not particularly limited, for example, error backpropagation is used to optimize various parameters. A motion diagnosis model 142 is generated by performing learning using a plurality of training data.

As training data for the motion diagnosis model 142, a moving image of a user during training and a moving image of an expert such as a trainer during training may be prepared. In this case, the trainer or the like compares the user's degree of proficiency with the movement of the expert as a reference, and uses the label as a label.

The update model 143 outputs the updated content of the training menu when the training menu and the level of progress are input. For example, if "leg press, 40 km, 10 reps, 2 sets" and "improvement level 5" are input, "leg 45 km, 12 reps, 2 sets" is output. A menu update model is configured by CNN, for example.

As training data for the updated model 143, a training menu to be proposed next is prepared based on the training menu and the level of progress. In the intermediate layer, the training menu output by the updating model 143 is closer to the correct training menu to be proposed next when the training menu and the degree of proficiency constituting the training data are input to the updating model 143. Optimize the parameters used in arithmetic processing. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions used in each neuron, and the like. Although the parameter optimization method is not particularly limited, for example, error backpropagation is used to optimize various parameters. An updated model 143 is generated by performing learning using a plurality of training data. It is also possible to create a table that defines the correspondence relationship between the training menu, the level of progress, and the next training menu to be proposed, and use this table instead of the update model 143 to update the training menu.

Next, information processing performed by the menu management system 100 will be described. FIG. 13 is a flow chart showing an example of the update process procedure. The update process is a process of collecting the user's training status and updating the user's training menu after the training. The control unit 11 of the management server 1 acquires a user ID (step S1). Acquisition of the user ID is performed, for example, by the user causing a card reader (ID acquisition device) to read the membership card. Alternatively, the user may enter the user ID using a keyboard or touch panel. Also, the user may be identified by face authentication. The control unit 11 acquires a training menu for the user from the menu DB 133 using the user ID as a search key, and displays it on the display device 2 (step S2). The control unit 11 performs training processing (step S3). Training processing will be described later. The control unit 11 performs diagnostic processing (step S4). Diagnosis processing will be described later. The control unit 11 updates the training menu based on the result of the diagnostic processing (step S5). The control unit 11 inputs the training menu executed by the user and the degree of progress of each training obtained in the diagnosis process to the update model 143, and updates the training menu based on the content obtained as output. The control unit 11 ends the processing.

FIG. 14 is a flow chart showing an example of a training process procedure. The training process is a process corresponding to step S3 in FIG. The control unit 11 of the management server 1 starts photographing with the camera 3 (step S11). The camera 3 used for photographing is appropriately selected according to the contents of the training. For example, if the training is a chest press, the camera 3 installed around the chest press machine takes pictures. The control unit 11 determines whether or not training has ended (step S12). If the controller 11 determines that the training has not ended (NO in step S12), it repeats step S12. When the control unit 11 determines that the training has ended (YES in step S12), it stops shooting (step S13). The control unit 11 stores the captured moving image in the moving image DB 135 (step S14). The control unit 11 determines whether or not the training menu has ended (step S15). If the control unit 11 determines that the training menu has not ended (NO in step S15), it resumes shooting (step S16). The camera 3 used for photographing is selected according to the contents of the following training. The control unit 11 returns the process to step S12. When the control unit 11 determines that the training menu has ended (YES in step S15), it ends the training process and returns the process to the caller.

FIG. 15 is a flow chart showing an example of diagnostic processing procedures. Diagnosis processing is processing corresponding to step S4 in FIG. The control unit 11 of the management server 1 selects one training included in the training menu, and acquires the moving image of the user who is performing the selected training from the moving image DB 135 (step S21). The control unit 11 analyzes the motion of the user from the acquired moving image (step S22). The control unit 11 inputs the moving image to the motion analysis model 141, and obtains the motion range and motion speed of the user's predetermined parts during training from the moving image of the skeleton model output by the motion analysis model 141 as an analysis result. The control unit 11 performs operation diagnosis (step S23). The control unit 11 stores the result of the operation diagnosis in the auxiliary storage unit 13 or the like (step S24). The control unit 11 inputs the analysis result of the user's previous training and the analysis result of this time to the motion diagnosis model 142 and acquires the degree of progress output by the motion diagnosis model 142 . The control unit 11 stores the degree of progress in a temporary storage area such as the auxiliary storage unit 13 . For example, there are 5 levels of progress, where 3 indicates the status quo, 4 indicates a slight improvement, 5 indicates a significant improvement, 2 indicates a slight decline, and 1 indicates a significant decline. The control unit 11 determines whether or not there is unprocessed training (step S25). When the control unit 11 determines that there is unprocessed training (YES in step S25), the process returns to step S21. When the control unit 11 determines that there is no unprocessed training (NO in step S25), the training performed by the user this time is evaluated based on the degree of progress of each training stored in the temporary storage area. The value is stored in the evaluation column of the training history DB 134 (step S26). Evaluation values are the mean, median, mode, total, etc. of the progress of each training. Also, the evaluation value may be a matrix in which the degree of progress of each training is arranged. The control unit 11 notifies the user of the evaluation result (step S27). The control unit 11 displays the evaluation result on the display device 2 . Alternatively, the control unit 11 transmits the evaluation result to the user terminal 5 . The control unit 11 terminates the diagnostic processing and returns the processing to the calling source.

FIG. 20 is an explanatory diagram showing an example of an output screen. The output screen is transmitted to the user terminal 5 or displayed on the display device 2 after the training is finished. The output screen shows the training evaluation and the next training menu based on it.

In the present embodiment, it is possible to update the training menu based on the result of execution of the training menu.

The basic functions of the menu management system 100 are as described above. Next, additional functions of the menu management system 100 will be described.

(Update function by trainer)
If the user does not want the training menu to be updated by the menu management system 100, or if the user disagrees with the content of the training menu to be updated by the menu management system 100, the trainer can update the training menu. FIG. 16 is a flow chart showing a procedure example of menu update processing. When the user does not want the training menu to be updated by the menu management system 100, after step S3 in FIG. 13, steps S4 and S5 are replaced and menu update processing is executed. Also, if the user does not agree with the content of the training menu updated by the menu management system 100, after the update processing of FIG. 13, the menu update processing is executed at the request of the user who has seen the updated training menu. The control unit 11 of the management server 1 transmits a training evaluation request to the trainer terminal 6 (step S31). The control unit 61 of the trainer terminal 6 receives the evaluation request and displays it on the display unit 66 (step S32). In order to perform evaluation, the trainer instructs the trainer terminal 6 to reproduce the user's training video. Upon receipt of the instruction, the control unit 61 transmits a video request to the management server 1 (step S33). The control unit 11 of the management server 1 receives the video request (step S34). The control unit 11 transmits the moving image to the trainer terminal 6 (step S35). The controller 61 of the trainer terminal 6 receives and reproduces the moving image (step S36). Streaming reproduction may be used for reproduction of moving images. After watching the video, the trainer inputs training scoring results and feedback (such as comments to the user) to the trainer terminal 6 . The control unit 61 transmits the input scoring results and feedback to the management server 1 (step S37). The control unit 11 of the management server 1 receives the scoring results and feedback, and transmits the received scoring results and feedback to the user terminal 5 (step S38). The control unit 51 of the user terminal 5 receives the scoring results and feedback, and displays them on the display unit 56 (step S39). The user refers to the scoring results and feedback, and then inputs a request to the trainer. The request is a training menu update request to the trainer. The request may include questions for the trainer or requests for updates. The control unit 51 transmits the input request to the management server 1 (step S40). The control unit 11 of the management server 1 receives the request and transmits the received request to the trainer terminal 6 (step S41). The control unit 11 also transmits the training menu together with the request. The control unit 61 of the trainer terminal 6 receives the request and the training menu, and displays the request and the training menu to be updated on the display unit 66 (step S42). The trainer refers to the request and updates the training menu. The control unit 61 receives the updated content (step S43). The control unit 61 transmits the received update contents to the management server 1 (step S44). The control unit 11 of the management server 1 receives the updated content, reflects it in the menu DB 133 (step S45), and terminates the process.

(Product recommendation function)
A user who has finished training often replenishes water and nutrients at a fitness gym. The product recommendation function is a function of recommending products for replenishing water and nutrition to the user who has finished training. FIG. 17 is a flow chart showing an example of the procedure of recommendation processing. The recommendation process is performed after the user finishes training, for example, after step S3 in FIG. 13 is performed. In this case, the control unit 11 of the management server 1 performs the update process and the recommendation process in parallel. The control unit 11 outputs a message addressed to the user to the display device 2 (step S61). An audio output unit including a speaker or the like may be provided in the display device 2 to output the message as audio. In order to acquire the user's face image necessary for determining the degree of fatigue, the message should be the content that prompts the user to take action. For example, the camera is installed at a position where a face image can be taken when the user faces the display device 2 . The control unit 11 takes a face image of the user with the camera (step S62). The control unit 11 analyzes the expression of the photographed face image (step S63). The control unit 11 diagnoses the degree of fatigue of the user from the result of facial expression analysis (step S64). Facial expression analysis and fatigue level diagnosis are well-known techniques, so description thereof will be omitted. Based on the degree of fatigue, the control unit 11 searches the product DB 136 and selects products recommended to the user (step S65). The control unit 11 outputs products recommended to the user (step S66). The control unit 11 displays recommended products on the display device 2 . The control unit 11 may display recommended products on a display provided in the vending machine 4 . The control unit 11 communicates with the vending machine 4 and determines whether or not the user has purchased the recommended product (step S67). The control unit 11 stores the history including the determination result in the recommendation history DB 137 (step S68), and terminates the process. Note that the recommended product may be provided to the user free of charge as a sample. In this case, the control unit 11 instructs the vending machine 4 to carry out the recommended product to the receiving port without paying the price when the user performs a predetermined operation.

(analysis function)
The product recommendation function analyzes the recommendation history accumulated in the recommendation history DB 137 and analyzes what kind of user selected each product. For example, the control unit 11 analyzes the age, gender, and post-training fatigue level of the user who most selected each product. The control unit 11 stores the analysis result in the analysis result DB 138 .

(Group lesson diagnosis function)
In the embodiments described above, the training is machine-based and assumes that only one user is visible in the captured images. In the following, diagnostic functions in group lessons, such as aerobics and jazz dance, will be described. FIG. 18 is a flow chart showing an example of the evaluation process procedure. The evaluation process is executed after the group lesson ends. The control unit 11 of the management server 1 acquires the video of the group lesson (step S81). The control unit 11 analyzes the motion of each person appearing in the moving image (step S82). For example, the above Pose Proposal Network is used. Pose Proposal Network detects the joint points of each person when inputting a video in which multiple people are shown, estimates the pose of each person by connecting the joint points, and outputs a skeletal model. The control unit 11 determines the skeleton model of the instructor from among a plurality of skeleton models (step S83). From the position and angle of view of the camera 3 that shot the input moving image, the standing position of the instructor, etc., the instructor is determined from among the persons appearing in the moving image. The control unit 11 selects a user to be evaluated (step S84). The control unit 11 evaluates the user (step S85). Using the skeletal model, the control unit 11 compares the movement of the instructor and the movement of the target user, and determines the degree to which the target user can follow the movement of the instructor. The degree is the evaluation value. The degree of follow-up may be determined for each frame, and the degree of all frames may be averaged as an evaluation value. The control unit 11 stores the evaluation value (step S86). The control unit 11 determines whether or not there is an unprocessed user (step S87). If the control unit 11 determines that there is an unprocessed user (YES in step S87), the process returns to step S84. When the control unit 11 determines that there is no unprocessed user (NO in step S87), it notifies the end of the evaluation (step S88). For example, the end of evaluation is displayed on the display device 2 . Alternatively, the user terminal 5 may be notified. Before the group lesson, the user causes a card reader or the like to read the membership card, and the management server 1 stores the users who participated in the group lesson. The control unit 11 terminates the processing.

FIG. 19 is a flow chart showing an example of a reference process procedure. The referencing process is performed when the user refers to the evaluation after completing the evaluation. The control unit 51 of the user terminal 5 issues a login request to the management server 1 (step S101). The control unit 11 of the management server 1 receives the login request (step S102). The control unit 11 performs login authentication and transmits the authentication result to the user terminal 5 (step S103). The control unit 51 of the user terminal 5 receives the authentication result (step S104). The control unit 51 determines whether or not the authentication result is login permission (step S105). When the control unit 51 determines that the authentication result indicates that login is not permitted (NO in step S105), the process ends. When the control unit 51 determines that the authentication result is login permission (YES in step S105), it transmits an evaluation display request to the management server 1 (step S106). The control unit 11 of the management server 1 receives the display request (step S107). The control unit 11 acquires the video of the lesson (step S108). The control unit 11 processes the other user appearing in the moving image into an avatar display (step S109). At this time, a numerical value indicating the degree to which the user can follow the instructor may be added to each frame. The control unit 11 transmits the processed video and the user's evaluation value in the lesson to the user terminal 5 (step S110). The control unit 51 of the user terminal 5 receives the moving image and the evaluation value (step S111). The control unit 51 displays the moving image and the evaluation value (step S112), and ends the process.

A smart mirror may be used as the display device 2 in the above-described embodiments. By providing a half mirror at a position facing the display surface of the display, the smart mirror reflects the user's appearance and allows the user to visually recognize the display content of the display.

FIG. 21 is a block diagram showing a functional configuration example of a management server. The management server 1 includes an acquisition unit 11a, an analysis unit 11b, an evaluation unit 11c, and an update unit 11d. By the control unit 11 executing the control program 1P, the management server 1 operates as follows.

The acquisition unit 11a acquires a training image of the user. The analysis unit 11b analyzes the user's motion from the training image. The evaluation unit 11c evaluates the degree of progress based on the analysis result. The updating unit 11d updates the user's training menu based on the evaluation result.

The technical features (components) described in each embodiment can be combined with each other, and new technical features can be formed by combining them.
The embodiments disclosed this time are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

100 Menu Management System 1 Management Server 11 Control Part 11a Acquisition Part 11b Analysis Part 11c Evaluation Part 11d Update Part 12 Main Storage Part 13 Auxiliary Storage Part 131 User DB
132 Trainer DB
133 Menu DB
134 Training history DB
135 Video database
136 product database
137 Recommendation History DB
138 Analysis result DB
141 motion analysis model 142 motion diagnosis model 143 update model 15 communication unit 16 reading unit 1P control program 1a portable storage medium 1b semiconductor memory 2 display device 3 camera 4 vending machine 5 user terminal 6 trainer terminal 7 central server B bus N network

Claims (13)

an acquisition unit for acquiring training images of a user;
an analysis unit that analyzes a user's motion from the training image;
an evaluation department that evaluates progress based on analysis results;
An information processing apparatus, comprising: an updating unit that updates the user's training menu based on an evaluation result. a transmission unit that transmits the training image to the trainer terminal;
a first receiving unit that receives training scoring results from the trainer terminal and feedback to the user;
a second receiving unit that receives a request for the feedback from a user terminal;
The information processing apparatus according to claim 1, wherein the update unit updates the menu based on the request. a detection unit that detects completion of training of the user from the training image;
an audio output unit that outputs a predetermined audio when the end of training is detected,
The acquisition unit acquires an image of the user after outputting the predetermined sound,
a diagnostic unit that diagnoses the degree of fatigue of the user based on the image acquired by the acquisition unit;
a selection unit that selects a recommended product based on the degree of fatigue;
3. The information processing apparatus according to claim 1, further comprising an output unit that outputs the selected product. accumulating a history of whether or not the user has purchased the product output by the output unit;
4. The information processing apparatus according to claim 3, wherein the user's preferences are analyzed based on the accumulated history. obtaining a user ID of the user from an ID obtaining device;
The information processing apparatus according to any one of claims 1 to 4, wherein the updating unit updates the menu stored in association with the user ID. a display unit that displays the menu;
The information processing apparatus according to any one of claims 1 to 5, further comprising a half mirror facing the display surface of the display unit. 2. The analysis unit, when inputting a human action image, inputs the training image to a first learning model that outputs an analysis result of the human action, and obtains the analysis result. The information processing apparatus according to any one of claims 6 to 6. The evaluation unit inputs the analysis result of the user to a second learning model that outputs the progress level of the motion when the analysis result of the human motion is input, and the progress level output by the second learning model. 8. The information processing apparatus according to any one of claims 1 to 7, wherein the evaluation is performed based on. When a plurality of users are shown in the training image, the degree of progress is evaluated for each user, and the users other than a predetermined user are converted into avatars in the training image, and the converted image and the degree of progress are identifiable by the user. The information processing apparatus according to any one of claims 1 to 8, characterized by displaying. Get training images for the user,
analyzing user behavior from the training images;
Evaluate progress based on analysis results,
An information processing program for causing a computer to perform a process of updating a training menu for the user based on an evaluation result. the computer
Get training images for the user,
analyzing user behavior from the training images;
Evaluate progress based on analysis results,
An information processing method, comprising updating a training menu for the user based on an evaluation result. an acquisition unit for acquiring training images of a user;
a transmission unit that transmits the training image to the trainer terminal;
a first receiving unit that receives training scoring results from the trainer terminal and feedback to the user;
a second receiving unit that receives a request for the feedback from a user terminal;
and an updating unit that updates the user's training menu based on the request. an acquisition unit for acquiring images of the user, including training images;
a detection unit that detects completion of training of the user from the training image;
an audio output unit that outputs a predetermined audio when the end of training is detected;
a diagnosis unit that diagnoses the degree of fatigue of the user;
a selection unit that selects a recommended product based on the degree of fatigue;
an output unit for outputting the selected product,
The acquisition unit acquires an image of the user after outputting the predetermined sound,
The information processing apparatus, wherein the diagnosis unit diagnoses the degree of fatigue of the user based on the image acquired by the acquisition unit.

Images