JP2020080096A - Object identifying apparatus, identifying system and identifying method - Google Patents

Abstract

To provide an object identifying apparatus that accomplishes, in a real-time manner, identifications of a large number of person objects by a small number of GPUs.SOLUTION: In a frame image unifying unit 28, a dividing method deciding unit 281 decides the dividing method of a frame memory M. A resizing unit 282 resizes each camera image obtained from a user terminal 1 in accordance with the size of each partial area of the frame memory M. An image layout unit 283 lays out each resized camera image obtained at a predetermined time cycle in the partial area allocated to each user terminal 1 in sequence, thereby forming a unified frame image I at the predetermined time cycle in sequence. An attitude estimating unit 29 estimates, on the basis of a feature map created from the unified frame image, the positions of the body parts of a person object and the connectivity thereof by a single estimation through a bottom-up approach.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an object identification device, an identification system, and an identification method, and in particular, the position and connectivity of body parts are estimated by a bottom-up approach for a plurality of camera images of human objects to identify each object. Object identifying apparatus, identifying system, and identifying method.

  In fields such as crime prevention and marketing, various technologies for identifying the actions of a large number of person objects shown in a camera image have been proposed. Conventionally, a top-down method has been adopted in which a person object is extracted from a camera image, a Bounding Box of each person object is detected, and then posture estimation is performed for each box. However, in the top-down method, the calculation amount increases in proportion to the number of person objects to be identified. Moreover, if the extraction of the person object fails, the action cannot be identified.

  With respect to such a technical problem, Non-Patent Document 1 describes the position and connectivity of body parts of a human object extracted from a camera image by two sequential prediction processes using Confidence Map and Part Affinity Fields (PAFs). A technique for estimating the behavior of each object with high accuracy in real time regardless of the number of human objects is disclosed by estimating a bottom-up approach.

  On the other hand, training may be carried out at home or in a facility such as a training gym for the purpose of maintaining or promoting health. Professional training is indispensable for safe and effective training, but it is difficult to receive professional training at home training, and it becomes a self-paced training.

  The training gym is fully equipped with training equipment and professional instructors are stationed there. However, it is not convenient because I have to go to the training gym. In addition, the one-on-one instruction requires a corresponding cost burden, which increases the time constraint.

  With respect to such a technical problem, Patent Document 1 discloses a recognition unit that recognizes a motion of a person shown in an input image, and a display control that superimposes a different virtual object on the input image according to the effectiveness of the recognized motion. An image that presents feedback regarding the effectiveness of exercise to the user by calculating a score indicating the effectiveness of exercise recognized by the recognition unit and superimposing the calculation result on the input image. A processing device has been proposed.

  In Non-Patent Document 2, a skeleton information is acquired by analyzing a video image of a training user with a motion sensor system, a user's muscle training motion is detected, and the detected training motion is detected from the viewpoint of speed and angle. A strength training support system for supporting is disclosed.

JP 2013-103010 A

Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp .1302-1310. "Muscular strength training support system using Kinect", Proc. of the 77th National Convention (P437-438, 2015-03-17)

  In order for the server to acquire the camera image of each user sent from the user terminal and apply it to an appropriate analysis algorithm to identify the behavior of the human object, the camera image showing the human object is expanded on the frame memory of the GPU. Then, the posture estimation is executed on the memory.

  In the posture estimation algorithm as in Non-Patent Document 1, a camera image including a human object is developed on the frame memory of the GPU. At this time, if one camera image is expanded in the entire area of the frame memory, one GPU can only estimate the posture of a person in one camera image. Therefore, when receiving a large number of camera images in parallel from a plurality of user terminals at the same time, as many GPUs as the number of user terminals are required to realize object identification in real time.

  An object of the present invention is to solve the above technical problems and enable one GPU to simultaneously perform posture estimation for a plurality of camera images with a low processing load. It is to provide an object identification device, an identification system, and an identification method that can realize real-time identification of a large number of person objects existing in a place.

  In order to achieve the above-mentioned object, the present invention is characterized in that an object identification device that develops a camera image on a frame memory and estimates the posture of a human object has the following configuration.

  (1) A means for acquiring a time series of camera images transmitted by a plurality of user terminals, a means for virtually dividing a frame memory into a plurality of partial areas, and arranging each camera image in each partial area of the frame memory By means of a bottom-up approach, the position and connectivity of the body parts of the person object are determined based on the means for generating an integrated frame image by integrating a plurality of camera images into one frame and the feature map generated from the integrated frame image. A posture estimating means for estimating the posture by one inference is provided.

  (2) The posture estimating means estimates the posture of the human object for each partial region by excluding the connectivity of body parts extracted from different partial regions from the estimation target.

  (3) Based on the estimated likelihood output from the posture estimating means, the dividing means is configured to increase the number of partial areas to be divided as the estimated likelihood increases.

  (4) A means for requesting the user terminal to change the size of the camera image to be transmitted is provided.

  (5) A means for requesting the user terminal to change the transmission cycle of the camera image is provided.

  (6) The frame memory is made to include non-uniformly divided partial areas, and the partial area in which the camera image is arranged is determined based on the attribute information of the user who transmitted the camera image.

  (7) The integrated frame image is generated by arranging each of the 1st to n(<m)th camera images in each partial area of the frame memory for m camera images. In the next cycle, each of n+1 to mth camera images is arranged in each partial area of the frame memory to generate an integrated frame image, and this is repeated for every m subsequent camera images. I did it.

  According to the present invention, the following effects are achieved.

  (1) Since the frame memory is virtually divided into multiple partial areas and the camera image of each user is placed in each partial area to generate an integrated frame image, pose estimation for many users can be performed with a small number of GPUs. You can do it at the same time.

  (2) Since the connectivity of body parts extracted from different partial areas is excluded from the estimation target and the pose of the human object is estimated for each partial area, pose estimation for multiple users is performed while suppressing false estimation. Can be performed simultaneously and with low load for each GPU.

  (3) Based on the estimated likelihood of the pose, the number of sub-regions to be divided is increased as the estimated likelihood increases. Therefore, while maintaining the image resolution required for highly reliable pose estimation, It will be possible to estimate the user's posture for each GPU simultaneously and with low load.

  (4) Since the means for requesting the user terminal to change the size of the camera image is provided, even if the number of divisions of the frame memory is changed and the size of the camera image needs to be changed, the size change processing is performed. Since the load is distributed to each user terminal, not only the load concentration can be prevented, but if the size is requested to be reduced, the traffic volume from the user terminal to the server can be reduced.

  (5) Since the means for requesting the user terminal to change the camera image transmission cycle is provided, it is possible to prevent the camera image buffer overflow and the failure of real-time processing even if the number of users transmitting camera images increases. Like

  (6) The frame memory is configured to include non-uniformly divided partial areas, and the partial area in which the camera image is arranged is determined based on the attribute information of the user who sent the camera image. You will be able to optimize the services you provide.

  (7) The integrated frame image is generated by arranging each of the 1st to n(<m)th camera images in each partial area of the frame memory for m camera images. In the next cycle, each of n+1 to mth camera images is arranged in each partial area of the frame memory to generate an integrated frame image, and this is repeated for every m subsequent camera images. Therefore, it becomes possible to simultaneously process camera images of a number exceeding the number of divisions of the frame memory while suppressing a decrease in image resolution for each GPU.

It is a block diagram showing the composition of the training support device to which the present invention is applied. It is a functional block diagram showing the composition of the main part of training support server (2) and training DB (3). It is the figure which showed the list of the training menu. It is a figure showing an example of a training table which manages the contents of a training menu. FIG. 3 is a block diagram showing the configurations of a user terminal (1), a frame image integration unit (28), and a posture estimation unit (29). It is the figure which showed the example of the trimming method of a camera image. It is the figure which showed the example which equally divides a frame memory. It is the figure which showed the example which divides|segments a frame memory unequal. It is the figure which showed the balance diagnostic menu which turns the upper body left and right. It is the figure which showed the balance diagnostic menu which raises and lowers the left and right heels one by one. 3 is a sequence flow showing a procedure of communication and various processes among the user terminal (1), the training support server (2) and the training DB (3) in time series. It is the figure which showed the example of the screen which a user selects a training objective. It is the figure which showed the example of the operation procedure of a user terminal, and the screen which adjusts the relative position of a user terminal and a user. It is a figure showing a display example of guidance for trunk balance diagnosis. It is the figure which showed the example of a display of a diagnostic result. It is the figure which showed the example of the start screen of a training menu. It is the figure which showed the example of the list display of this training content. It is the figure which showed the example of a display of the content and cautions for every training. It is the figure which showed the example of the total score screen. It is the figure which showed the example which displays the history of the total score in time series. It is a figure showing other integration methods by a frame image integration part (28).

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a main part of a training support device to which the object identification device of the present invention is applied, and a user terminal 1, a training support server 2, and a training DB 3 are main components.

  The user terminal 1 includes a moving image capturing function, a wireless communication function, and a display, and can be replaced with, for example, a smartphone. The training support server 2 acquires a training video of a user who is training from the user terminal 1 via Wi-Fi, the base station BS and the network NW, analyzes the video, and obtains an appropriate training evaluation. To provide. The training DB 3 stores attribute information of each user and a large number of training menus and diagnosis menus.

  FIG. 2 is a functional block diagram showing a configuration of main parts of the training support server 2 and the training DB 3, and the training DB 3 includes an attribute information storage unit 31, a training level storage unit 32, a diagnostic menu storage unit 33, and a training menu. The storage unit 34 is included.

  The attribute information storage unit 31 stores, for each user ID, age, sex, height, weight, blood pressure, anamnessis, etc. as attribute information of the user. The training level storage unit 32 stores the current training level for each user ID. The diagnostic menu storage unit 33 stores a plurality of diagnostic menus.

  The training menu storage unit 34 stores a plurality of training menus. FIG. 3 is a diagram showing a list of training menus stored in the training menu storage unit 34. In the present embodiment, 32 types of training menus such as “squat”, “arm twist”, and “wood chopper” are shown. Is prepared.

  FIG. 4 is a diagram showing an example of a training table that manages the contents of the training menu, and for each training menu, its effect (mainly training area), reference tempo, left/right presence/absence, voice guide identifier, instruction point And the NG threshold and the like are registered. The NG threshold is a threshold condition that is not recognized as effective training. For example, if the training menu is “squat”, it is not recognized as effective training when the angle of the lower leg is 40° or more.

  Further, in the present embodiment, a large number of training menus are classified into a plurality of groups so that an optimal training menu is selected according to the user's training level and attribute information, and as repulsion information, for example, hypertension and heart disease. A training menu that is unsuitable for each medical condition such as low back pain is also registered.

  Returning to FIG. 2, in the training support server 2, the input/output interface 21 establishes a communication session with each user terminal 1 and transmits/receives a message using voice, video or text. The user authentication unit 22 performs user authentication by matching the user ID transmitted from the user terminal 1 with the user ID registered in the training DB 3. The training purpose receiving unit 23 receives the training purpose designated by the user from the user terminal 1.

  As shown in FIG. 5, the frame image integration unit 28 and the posture estimation unit 29 cooperate with each user terminal 1 to perform training from a plurality of camera images transmitted from a plurality of user terminals 1 at the same time. The skeleton information of each user is acquired by one inference, and the pose of each user is estimated.

  In the user terminal 1, the camera unit 101 shoots a moving image of the user and outputs a camera image. The capture unit 102 captures a camera image at a predetermined sampling period to convert the still image into a time series of still images. The sampling period by the capture unit 102 can be changed in response to a change request fed back from the training support server 2 according to the likelihood of posture estimation, as described in detail later.

  As shown in FIG. 6, the trimming unit 103 trims a vertically or horizontally long captured image into a rectangular shape at the center of the image of the user. Note that such trimming processing may be performed by the camera unit 101. The image reduction unit 104 changes the reduction ratio of the camera image trimmed into the rectangle according to the size change request fed back from the frame image integration unit 28 or the network environment of the user terminal 1.

  In the present embodiment, if the user terminal 1 is in an environment in which it is connected to the network via WiFi, the user terminal 1 is compressed in the JPEG format having a size of 160×160 and a quality of 0.65, while it is connected to the network by wire. Under the environment, it is compressed in the JPEG format with a size of 320 x 320 and a quality of 0.65. Also, in an environment in which a network is connected via a 4G line, for example, it is compressed in a JPEG format with a size of 160×160 and a quality of 0.45.

  The transmission unit 105 uses the time series of the reduced rectangular camera images (A1, A2, A3...), (B1, B2, B3...), (C1, C2, C3...), (D1, D2, D3...). Are sequentially transmitted to the training support server 2 via Web socket. The transmission cycle of the camera image by the transmission unit 105 depends on the sampling cycle by the capture unit 102.

  In the frame image integration unit 28, the division method determination unit 281 determines the division method of the frame memory M based on the number of user terminals transmitting camera images at the same time and the estimated likelihood and recognition rate obtained in posture estimation described later. To do. In the present embodiment, if camera images are transmitted from a large number of user terminals 1 and the likelihood of posture estimation is sufficient, for example, as shown in FIGS. It is virtually divided into 4 equal parts or 9 equal parts, and any partial area is allocated to each user terminal 1.

  The resizing unit 282 decodes each camera image of the acquired JPEG format, and further resizes the camera image according to the size of each partial area of the frame memory M. The image arrangement unit 283 provides the resized camera images (a1, a2, a3...), (b1, b2, b3...), (c1, c2, c3...), (d1, d2, d3...) By sequentially arranging them in the partial areas assigned to the user terminal 1, one integrated frame image I is sequentially constructed in a predetermined cycle.

  The posture estimation unit 29 first performs feature map extraction on the one integrated frame image I, as in Non-Patent Document 1. Next, the extracted feature map is sequentially applied with two sequential prediction processes using Confidence Map that encodes the positions of the body parts and Part Affinity Fields (PAFs) that encodes the connectivity between the body parts, and integrated. A skeleton model is constructed by estimating the position and connectivity of body parts of human objects (users) extracted from frame images by a single inference using a bottom-up approach.

  At this time, by implementing a process that excludes the connectivity of body parts extracted from different partial regions as an estimation target, the position and connectivity of the body parts can be calculated for each partial region, that is, for each user to obtain the skeleton model of the object. Be able to estimate.

  The estimated likelihood obtained in the estimation process is fed back to the frame image integration unit 28 as a division size change request for the frame memory M. If the estimation likelihood is sufficiently high and the number of user terminals transmitting camera images exceeds a predetermined threshold, the division method determination unit 281 determines, for example, as shown in FIG. Is virtually divided into 9 and the number of camera images to be integrated into one frame is increased. The resizing unit 282 resizes the camera image according to the changed equal division number.

  On the other hand, when the estimated likelihood decreases and falls below a predetermined lower limit value, for example, if 9 equal divisions are being performed, 4 divisions are performed, and if 4 divisions are being performed, no division is performed. A sufficient estimation likelihood is secured by reducing the number of camera images to be integrated in and maintaining high resolution.

  The posture estimation unit 29 uses the result of posture estimation including the position and connectivity of body parts extracted for each user terminal (hereinafter, sometimes collectively referred to as skeleton information) to the balance diagnosis unit 24 in the subsequent stage or the training evaluation. The data is sequentially transferred to the unit 26.

  As described above, in the present embodiment, the number of camera images to be integrated into one frame can be adaptively changed based on the estimated likelihood of the posture, so that the behavior recognition of multiple users can be performed by one GPU without lowering the estimation accuracy. You can do it at the same time.

  In the above embodiment, the frame memory M of the GPU is equally divided and each camera image is arranged in each partial area with the same size, but the present invention is not limited to this. As shown in FIG. 8, the size of each partial area may be different by, for example, unequal division for each user according to the content and fee of the contracted service.

  By doing so, a larger partial area can be assigned as a priority user to, for example, a user who subscribes to a specialized diagnostic service or a service with a higher fee. Therefore, it is possible to perform advanced diagnosis using skeleton information that cannot be estimated unless it is a high-resolution camera image, such as fingertip movements and facial expressions.

  Information regarding whether each user is a priority user is registered in advance in the attribute information storage unit 31 as one of the attribute information of the user in association with the user ID, and the user ID is stored in the user ID at the time of user authentication. You may make it based on this.

  Furthermore, in the above-described embodiment, it is described that the number of divisions of the frame memory M is reduced when the estimated likelihood in the posture estimation unit 29 is reduced, but the number of divisions of the frame memory M is only reduced to obtain from each user terminal. Processing of camera images may be delayed, and inconvenience such as buffer overflow may occur on the training support server side.

  Therefore, in this embodiment, when the processing rate of the integrated frame image I in the training support server 2 becomes lower than the transmission rate of the camera image by each user terminal 1, the transmission cycle of the camera image from the training support server 2 to each user terminal 1. Change request is sent to reduce the.

  In each user terminal 1, in response to the change request, the capture unit 102 lengthens a sampling period when capturing a still image from a moving image of the user, and a transmission period of a camera image so as to synchronize with the sampling period. Lower. As a result, even if the number of users transmitting camera images increases, it is possible to prevent the buffer overflow of camera images and the failure of real-time processing.

  Returning to FIG. 2, the balance diagnosis unit 24 includes a diagnosis menu selection unit 241, a diagnosis menu distribution unit 242, and a first action recognition unit 243, and a body based on the posture estimated from the skeleton information of the user who performs the balance diagnosis exercise. Diagnose the left-right balance of the trunk.

  The diagnosis menu selection unit 241 selects a diagnosis menu for diagnosing the trunk balance of the user from the diagnosis menu storage unit 33 based on the attribute information of the user, the training purpose, and the training level. The diagnostic menu delivery unit 242 delivers the selected diagnostic menu to the user terminal 1 and displays it on its display.

  The first action recognition unit 243 executes the action recognition of the user based on the skeletal information extracted from the diagnostic image of the user who executes the diagnostic menu. A behavior recognition method based on skeletal information is disclosed in, for example, Non-Patent Document 1 above. The balance diagnosis unit 24 can diagnose the trunk balance of the user based on the result of the action recognition.

  9 and 10 are diagrams showing an example of a diagnostic menu distributed by the diagnostic menu distribution unit 242 in order to evaluate the trunk balance of the user. As shown in FIG. 9, the upper body is turned left and right. A diagnostic menu, or as shown in FIG. 10, a diagnostic menu for raising and lowering the left and right heels one by one is delivered to each user.

  The first action recognition unit 243 compares the left turning angle and the right turning angle based on the user's posture and its change estimated from the diagnostic image of the user who executes the diagnostic menu of FIG. The trunk balance can be diagnosed based on the difference. When the diagnostic image of the user who executes the diagnostic menu of FIG. 10 is acquired, the trunk balance can be diagnosed by comparing the height of the waist position when the left heel is raised and when the right heel is raised.

  The physical condition diagnosis unit 25 diagnoses the current physical condition of the user based on the image of the user who executes the balance diagnosis menu. Such a physical condition diagnosis can be realized, for example, by measuring the tempo of the user's movement based on a diagnostic image obtained by photographing the user who executes the balance diagnosis menu.

  The training evaluation unit 26 includes a training menu selection unit 261, a training menu distribution unit 262, a second action recognition unit 263, an improvement point guidance unit 264, a training quality determination unit 265, a training amount counting unit 266, a score calculation unit 267, and an evaluation result. The training is evaluated by including the distribution unit 268, analyzing the posture estimated from the training video of the user who performs the training, and observing the transition of the specific part.

  The training menu selection unit 261 selects a training menu most suitable for the user at present based on the training purpose of the user, the diagnosis result of the trunk balance and the training level, and further, if necessary, the attribute information and the physical condition of the user. Select from the training DB3.

  The training message delivery unit 262 delivers the selected training menu to the user terminal 1. The second behavior recognition unit 263 recognizes the behavior of the user based on the change in the posture estimated from the training image of the user who executes the delivered training menu.

  The improvement point instructing unit 264 determines whether or not each “instruction point” shown in FIG. 4 has cleared the corresponding “NG threshold” based on the result of the action recognition of the user, and NG If the threshold value has not been cleared, the user is instructed to use the instruction point as an improvement point.

  In the present embodiment, for example, with respect to the training menu “squat”, “lower leg angle” is registered as one of the teaching points, and the NG threshold is set to “40° or more”. The improvement point instruction unit 264 determines the angle of the lower leg based on the result of the action recognition, and if it is 40° or more, selects the instruction message registered in the voice number “201” from the training DB 3 to select the user. Deliver to terminal 1. In the guidance message, for example, the content such as "Please bend the lower leg further" is registered. The instruction message is not limited to a voice message, and may be a video message with voice.

  The training quality determination unit 265 determines the quality of training based on the number of times of improvement point instruction given by the improvement point instruction unit 264 and the training tempo. In the present embodiment, as the number of times of improvement point teaching is increased, and as the training tempo, for example, the time required for a predetermined movement (for example, bending and stretching) is measured, the lower the tempo deviates from the standard tempo, the lower the quality of training. Is determined and the determination result is quantified and normalized to a value of 0.5 (low quality) to 1 (high quality), for example.

  The training amount counting unit 266 counts the training amount of the user by applying the result of the action recognition to an evaluation policy prepared in advance. In the present embodiment, for each delivered training menu, it is defined as a count condition that a predetermined part of the skeleton satisfies a predetermined transition condition in a predetermined order, and the count condition is determined based on the skeleton information acquired from the training video. The training amount is counted each time is satisfied.

  For example, if the provided training menu is squat, the angle θ of the knee when bending and stretching, the time (tempo) t required for bending and stretching once, the relative position P of the head and elbow when bending and stretching in side view, etc. It is registered as an evaluation item, and the number of times that the angle θ, time t, and relative position P satisfy predetermined conditions is counted.

  The score calculation unit 267 reflects the determination result regarding the quality of the training in the count value of the training amount and scores this training. In the present embodiment, it is assumed that the count value of the training amount is reduced and scored according to the quality of the training. For example, if the count value of the training amount is “100” and the quality of the training is “0.8”. For example, the score is calculated as 100×0.8=80. The evaluation result scored as described above is distributed to the user terminal 1 by the evaluation result distribution unit 268.

  The training information update unit 27 updates the training level of the user stored in the training level storage unit 32 in association with the user ID, based on the evaluation result of this time. The training level starts from level 1, and is updated to level 2, level 3... Level N (advanced) every time the cumulative amount of training exceeds a predetermined reference value.

  FIG. 11 is a sequence flow showing the procedures of communication and various processes among the user terminal 1, the training support server 2, and the training DB 3 in time series.

  At time t1, when the user starts the training support application installed in advance in the user terminal 1, user authentication is performed at time t2 and a communication session is established between the user terminal 1 and the training support server 2. ..

  At time t3, as shown in FIG. 12A, a selection screen for allowing the user to input the current training purpose is displayed on the display of the user terminal 1. When the user selects "diet", a screen for selecting a specific portion for which a diet is desired is displayed as shown in FIG. In the present embodiment, any one of “whole body”, “trunk/abdomen”, “upper body/second arm”, and “lower body/pip-up” can be selected.

  On the other hand, when "conditioning" is selected on the selection screen, a screen for selecting a symptom of concern is displayed as shown in FIG. In the present embodiment, any one of “stiff shoulder improvement”, “back pain improvement”, and “flexibility improvement” can be selected.

  At time t4, the selected training purpose and its “site” or “symptom” are transmitted from the user terminal 1 to the training support server 2. At time t5, the training support server 2 refers to the attribute information storage unit 31 and the training level storage unit 32 of the training DB 3 based on the user ID, and extracts the attribute information and training level of the user. Then, based on the extracted attribute information and training level, and the transmitted training purpose, the balance diagnosis menu shown in FIG. 9 or 10 is selected.

  At time t6, the selected balance diagnosis menu is transmitted to the user terminal 1. At time t7, the built-in camera of the user terminal 1 is activated in the moving image shooting mode. At time t8, as shown in FIG. 13, a screen for adjusting the operating procedure of the user terminal 1 and the relative position between the user terminal 1 and the user is displayed in a scroll format on the display of the user terminal 1.

  Here, when the user taps the “start measurement” button or the like, at time t9, the guidance for trunk balance diagnosis is displayed on the display of the user terminal 1 as shown in FIG. Here, when the user adjusts the display position of himself/herself on the display to the guide in response to the message “Please match the humanoid guide on the screen and stand in front of the camera”, the result shown in (b) in the figure is displayed. Thus, the balance diagnosis menu is started, and it is required to move the body according to the image and the message. When the user moves the body based on the balance diagnosis menu, the image is transmitted from the user terminal 1 to the training support server 2 as a diagnostic image after time t10.

  At time t11, the balance diagnosis unit 24 of the training support server 2 evaluates the trunk balance of the user. The physical condition diagnosis unit 25 analyzes the user's complexion and movement (tempo) in the balance diagnosis image to diagnose the current physical condition of the user. At time t12, the diagnosis result is delivered to the user terminal 1. At time t13, as shown in FIG. 15, the diagnosis result is displayed on the display of the user terminal 1.

  In the present embodiment, the “left center of gravity” is displayed as the trunk balance diagnosis result, together with the training purpose “diet” selected by the user and the region of concern “abdomen”.

  When the user taps the “OK” button on the diagnosis result screen, at time t14, the training menu selection unit 261 of the training support server 2 causes the trunk balance analysis result, the training purpose, the training level of the user and the current physical condition. Further, if necessary, the attribute information of the user is applied to a training menu selection policy that is built in advance, so that the most suitable training menu for the current user is selected from the training menu storage unit 34.

  For example, if the result of the physical condition diagnosis is “good”, a training menu suitable for the trunk balance, the training level and the training purpose is selected. On the other hand, if the result of the physical condition diagnosis is “improper”, a training menu having a lower load than the training menu suitable for the trunk balance, the training level and the training purpose is selected. It should be noted that, depending on the degree of "disorder" of the physical condition, a message indicating that this training is not recommended may be distributed.

  Further, if the attribute information of the user is further considered, the load of the training menu may be increased or decreased according to age or sex. Alternatively, a user who has registered “hypertension” as an existing condition may be allowed to select a training menu that does not easily increase blood pressure. Similarly, a user who has registered "back pain" as an existing condition may select a training menu with a low back load.

  At time t15, the training menu distribution unit 262 distributes the selected training menu from the training support server 2 to the user terminal 1. At time t16, the selected training menu is displayed as shown in FIG.

  FIG. 16 is a diagram showing an example of a training menu start screen, in which the contents and description of the selected training menu are displayed.

  In the present embodiment, it is displayed that the number of trainings is “4”, the time standard is 20 minutes, and the training content is “reverse lunge” for all four trainings. The user does not have to completely digest the proposed menu. For example, if you do not want to implement a part due to time constraints or physical condition, you can tap the training item you do not want to implement at time t17. You can also skip.

  When the modification of the training menu is completed as described above, a list of the current training contents is displayed as shown in FIG. In this case, the skipped training is grayed out.

  When the user taps the training start button, the training menu starts at time t18, and as shown in FIG. 18, the training content and cautions are displayed for each training to be performed.

  At time t19, the training video of the training user is transmitted to the training support server 2 in frame units. At time t20, the training evaluation unit 26 of the training support server 2 performs action recognition based on the change in the posture of the user estimated from the training video.

  At time t21, the evaluation result is transmitted to the user terminal 1, and is displayed on the display of the user terminal 1 at time t22. At time t23, the registered training level is updated by accumulating the current score in the history information of the past score.

  In the present embodiment, the training level is defined, for example, from the first level (beginner level) to the fifth level (advanced level), and when the cumulative score obtained at the nth level exceeds a predetermined threshold, the next nth+ Configured to promote to one level.

  FIG. 19 is a diagram showing an example of the evaluation result displayed on the user terminal after the training, in which the contents of the training carried out, the individual score of each training and the message are displayed in a time series, and the total score of the current training is displayed. "390" is displayed.

  The total score is managed as history information and can be displayed in a time series list as shown in FIG. As a result, the user can easily check the training history, and by referring to the score in particular, it is possible to objectively feel that the training level is improving, so that the motivation for the training can be improved. Can be expected to improve.

  FIG. 21 is a diagram schematically showing another image integration method of the frame image integration unit 28.

  In the above-described embodiment, for example, the frame memory M is divided into four equal parts and four camera images (a, b, c, d) are integrated [(a) in the figure], thereby simultaneously targeting four users. If the number of users increases and it becomes necessary to integrate eight camera images (a, b, c, d, e, f, g, h) while performing posture estimation, the frame memory M is adaptive. It has been described that the posture estimation can be simultaneously performed for a maximum of 9 users by dividing into 9 equal parts [(b) in the figure].

  However, the present invention is not limited to this, and as shown in FIG. 7C, the camera image set to be integrated is divided into two groups (a , B, c, d) and (e, f, g, h), and by sequentially switching the groups to be integrated in frame units, the postures of eight users can be simultaneously evaluated with one GPU. The estimation may be performed.

  That is, in the kth process, one group of camera image sets (a1, b1, c1, d1) is arranged in the frame memory M, and in the k+1st process, the other group of camera image sets (e1, f1, g1 and h1) are arranged, and in the k+2nd processing, the camera image sets (a2, b2, c2, d2) of one group are arranged again, and so on. It may be done.

  In this way, for m camera images, the first to n (<m)-th camera images are arranged in the partial areas of the frame memory to generate the integrated frame image of this time, and in the next cycle, By arranging each of the (n+1)th to mth camera images in each partial area of the frame memory to generate an integrated frame image, and repeating this for every m subsequent camera images, the image resolution for each GPU It is possible to simultaneously process a number of camera images exceeding the number of divisions of the frame memory while suppressing the deterioration of

  If the frame image processing rate in the training support server 2 is lower than the camera image transmission cycle of each user terminal 1, inconveniences such as a buffer overflow may occur on the training support server side. Therefore, also in the present embodiment, when the frame image processing rate in the training support server 2 is lower than the camera image transmission cycle of each user terminal 1, the training support server 2 reduces the camera image transmission cycle to each user terminal 1. Request is sent.

In response to the change request, each user terminal 1, in response to the change request, lengthens the sampling period when capturing a still image from the moving image of the user, and synchronizes with this. As described above, the transmission cycle of the camera image is reduced.
Furthermore, in the above embodiment, the camera image capture unit 102, the trimming unit 103, and the reduction unit 104 are described as being provided in the user terminal 1, but the present invention is not limited to this, and all of these are provided. Alternatively, a part of the training support server 2 may be provided.

  DESCRIPTION OF SYMBOLS 1... User terminal, 2... Training support server, 3... Training DB, 22... User authentication part, 23... Training purpose acceptance part, 24... Balance diagnosis part, 25... Physical condition diagnosis part, 26... Training evaluation part, 27... Training Information updating unit, 28... Frame image integrating unit, 29... Posture estimating unit, 31... Attribute information storage unit, 32... Training level storage unit, 33... Diagnostic menu storage unit, 34... Training menu storage unit

Claims (15)

In an object identification device that develops a camera image on a frame memory and performs posture estimation of a human object,
Means for acquiring a time series of camera images transmitted by a plurality of user terminals,
Means for virtually dividing the frame memory into a plurality of partial areas,
A means for generating an integrated frame image in which a plurality of camera images are integrated into one frame by arranging each camera image in each partial area of the frame memory;
Object identification, comprising: posture estimation means for estimating the position and connectivity of body parts of a human object by a single inference by a bottom-up approach based on a feature map generated from the integrated frame image. apparatus.   The object identification apparatus according to claim 1, wherein the posture estimation unit excludes connectivity of body parts extracted from different partial areas from estimation targets.   3. The object identifying apparatus according to claim 1, wherein the posture estimating unit estimates the posture of the human object for each partial area.   4. The dividing unit increases the number of partial regions to be divided as the estimated likelihood increases, based on the estimated likelihood output from the posture estimating unit. The described object identification device.   The object identifying apparatus according to claim 4, further comprising means for requesting the user terminal to change the size of a camera image to be transmitted.   6. The object identifying apparatus according to claim 4, further comprising means for requesting the user terminal to change the transmission cycle of the camera image.   7. The object identification device according to claim 1, wherein the frame memory is equally divided into a plurality of partial areas.   8. The object identifying apparatus according to claim 7, wherein the frame memory includes non-equally divided partial areas.   The means for generating the integrated frame image determines a partial area in which the camera image is arranged based on attribute information of a user who has transmitted the camera image. Object identification device.   The unit for generating the integrated frame image generates the integrated frame image of this time by arranging each of the 1st to n(<m)th camera images in each partial area of the frame memory for m camera images. Then, in the next cycle, each of the n+1 to m-th camera images is arranged in each partial area of the frame memory to generate an integrated frame image, and this is repeated for each of the subsequent m camera images. 10. The object identification device according to claim 1, wherein the object identification device is an object identification device. In an object identification system that develops camera images transmitted by a plurality of user terminals on a frame memory of an object identification device and performs posture estimation of a human object,
Each of the user terminals,
A means for capturing a camera image at a predetermined sampling period,
Means for transmitting a time series of the captured camera image to an object identification device,
The object identification device,
Means for acquiring a time series of camera images from each user terminal,
Means for virtually dividing the frame memory into a plurality of partial areas,
A means for generating an integrated frame image in which a plurality of camera images are integrated into one frame by arranging each camera image in each partial area of the frame memory;
Object identification, comprising: posture estimation means for estimating the position and connectivity of body parts of a human object by a single inference by a bottom-up approach based on a feature map generated from the integrated frame image. system. The object identification device further comprises means for requesting the user terminal to resize a camera image,
The object identification system according to claim 11, wherein the user terminal changes a size of a camera image to be transmitted in response to the size change request. The object identification device further comprises means for requesting the user terminal to change a transmission cycle of a camera image,
The object identification system according to claim 11 or 12, wherein the user terminal changes a transmission cycle of a camera image in response to the change request.   14. The object identification system according to claim 11, wherein the user terminal further comprises means for trimming a camera image. In an object identification method in which a computer develops a camera image on a frame memory and executes posture estimation of a human object,
A procedure for acquiring and storing time series of camera images transmitted by a plurality of user terminals,
A procedure of virtually dividing the frame memory into a plurality of partial areas,
A procedure for generating an integrated frame image in which a plurality of camera images are integrated into one frame by arranging each camera image in each partial area of the frame memory;
A method of estimating the position and connectivity of body parts of a human object by a single inference by a bottom-up approach based on a feature map generated from the integrated frame image.