JP2022062313A - Tactile sensation meta data generation apparatus, video tactile sensation linkage system and program - Google Patents

Abstract

To provide a tactile sensation meta data generation apparatus which automatically extracts a dynamic person object from a video and synchronously and automatically generates corresponding tactile sensation meta data, a video tactile sensation linkage system which controls drive of a tactile sensation presentation device based on the generated tactile sensation meta data and a program.SOLUTION: A tactile sensation meta data generation apparatus 12 according to the present invention comprises: a person skeleton extraction unit 122 and a person identification unit 123 which generate a skeleton coordinate aggregate of each person from a video and identify a person within a variable search range; a skeleton trajectory feature image generation unit 124 which generates a skeleton trajectory feature image (STI) indicating only the direction of the movement for each skeleton from which the person is identified; a person motion recognition unit 126 and a meta data generation unit 127 which recognize a person motion with deep learning (CNN) with the STI as the input, detect information for operating tactile sensation presentation devices 14L, 14R and generate the tactile sensation meta data. A video tactile sensation linkage system 1 comprises: the tactile sensation meta data generation apparatus 12; and a control unit 13 for tactile sensation presentation device control.SELECTED DRAWING: Figure 1

Description

The present invention is a tactile metadata generator that extracts a person object from an image and generates tactile metadata corresponding to a dynamic person object, and an image tactile interlocking that drives and controls a tactile presentation device based on the generated tactile metadata. Regarding systems and programs.

The video content of a general camera image such as a broadcast image is a medium that provides information that appeals to the two senses of sight and hearing. However, audiovisual information alone is not sufficient for the visually impaired and the hearing impaired, and it is not possible to accurately convey the status of the program content. Therefore, many people with disabilities do not have a TV, or even if they do, they do not watch. Therefore, it is desired to construct a system that allows visually or hearing impaired persons to understand television broadcasting by presenting information that can be felt by "tactile sense" other than visual and auditory senses to video contents.

Further, even for a healthy person having a sense of sight and hearing, by presenting a tactile stimulus, it can be expected to improve the sense of presence and immersiveness when watching a broadcast program. In particular, the movement of a person in sports content is important information, and by presenting this with a tactile stimulus, the sense of presence in viewing the content is enhanced.

For example, when watching a baseball image, the batter's hitting sensation can be simulated by stimulating the viewer via a tactile presentation device at the timing when the ball hits the bat. In addition, by providing tactile stimuli to people with visual impairments, it is thought that it will lead to understanding the game situation of sports. As described above, the tactile sensation is expected as a third sensation in video viewing.

In particular, since real-time video viewing is important in sports, the presentation of tactile stimuli for video needs to be performed automatically and in real time. Therefore, the presentation of tactile stimuli synchronized with the player's movements regarding the type, timing, situation, etc. of play is often effective for viewing video content that also uses tactile sensation. Then, it becomes possible to convey the situation of sports to people with visual or hearing disabilities.

Therefore, in order to realize video viewing of video content that also uses tactile sensation, it is necessary to extract the movement of a person object from the video content and generate tactile information corresponding to the movement of the extracted person object as tactile metadata. You will need it.

However, even if the conventional method of generating tactile metadata realizes video viewing using tactile sensation, tactile metadata indicating at what timing and what kind of stimulus is presented to the user by the tactile presentation device. It was necessary to manually edit the data in a manner synchronized with the video.

In the case of recorded programs, it is possible to manually edit the tactile metadata over time. However, in order to link the stimulus presentation by the tactile presentation device to the live broadcast video, it is not possible to edit the tactile information in advance, so it is necessary to perform video analysis of the video content in real time and generate tactile metadata. Required.

In recent years, sports video analysis technology has achieved remarkable growth. The tennis Hawkeye system, which is also used in Wimbledon, tracks a tennis ball three-dimensionally using a plurality of fixed camera images as sensors, and determines IN / OUT related to the judge. In addition, at the 2014 FIFA World Cup, called goal line technology, images from several fixed cameras are analyzed to automate goal determination. Furthermore, the sophistication of real-time video analysis technology in sports, such as the TRACAB system that sets a large number of stereo cameras in a soccer stadium and tracks all players in the field in real time, is advancing.

On the other hand, in order to measure the posture of a player as a dynamic person object, the measurement using a marker-type motion capture method is generally used. However, this method requires a large number of markers to be attached to the player's body and cannot be applied to actual games. Therefore, in recent years, it has become possible to measure a person's posture without a marker by using a depth sensor that reads an infrared pattern projected on a player's body and obtains depth information from the distortion of the infrared pattern. Further, various techniques applying an optical motion capture method instead of a marker method are disclosed (see, for example, Patent Documents 1, 2 and 3).

For example, in Patent Document 1, when teaching an operation to a user by displaying a model operation image of another person in a virtual reality system using stereoscopic vision, a measurement target person uses an optical motion capture method. A device for measuring the three-dimensional position of the skeleton of the above is disclosed. Further, Patent Document 2 discloses a technique for performing motion recognition by using information obtained from images such as gymnastics and motion capture data, and uses a hidden Markov model to limit the time length of motion. It has the feature of being removed. Further, Patent Document 3 discloses a training evaluation device that measures a player's motion by using an optical motion capture method and evaluates the player's form based on the measured data and data related to the model's form. Has been done. However, since these techniques use a motion capture method, they cannot be applied to an actual game, and it is difficult to measure a person's playing motion from a general-purpose camera image.

Further, regardless of the motion capture method, a device that simulates the movement of a person is disclosed only from a camera image of a badminton game or badminton practice taken by one or two people as a group (for example, a patent). See Document 4). The technique of Patent Document 4 detects actions such as shots from the captured camera image, but it is premised on processing from the captured image by a dedicatedly set camera, and is a general-purpose broadcast camera image. It is difficult to measure a person's playing behavior from.

By the way, with the development of deep learning technology in recent years, it has become possible to estimate the skeleton position of a person from a normal still image that does not include depth information, which was difficult in the past, without using a depth sensor. By using this deep learning technique, it is possible to extract a still image from a normal camera image and automatically measure the posture of the athlete included in the still image. That is, by measuring the posture of the athlete from a normal camera image, it is possible to acquire information on the tactile stimulus without affecting the competition.

Although it is possible to measure the posture of a person by acquiring skeletal information, recognition processing is required to give meaning to that posture. For example, when a judo image is input, it is necessary to determine whether the movement content performed in the frame is "combination", "throwing technique", or "grounding technique" from the image features and skeletal features. The Convolutional Neural Network (CNN) is widely used in recognition processing in image processing. CNN is a neural network with many deep layers, and is a network that demonstrates excellent performance especially in the field of image recognition. This network is composed of layers with some characteristic functions such as "convolution layer" and "pooling layer", and is currently used in a wide range of fields.

In a general neural network, neurons are arranged in layers, and the neurons contained in the layers before and after are usually connected comprehensively, but in this convolutional neural network, the connection between these neurons is well restricted, and moreover. By using a technique called weight sharing, processing equivalent to image convolution is expressed within the framework of a neural network. This layer is called the "convolution layer" and is the greatest feature of CNN. Another major feature of this convolutional neural network is the "pooling layer". In CNN, if the "convolution layer" plays a role of feature extraction such as edge extraction from an image, the "pooling layer" has robustness so that such extracted features are not affected by translation. Giving.

On the other hand, in addition to using skeletal information, an image called Motion History Image (MHI) has been conventionally used as a method of recognizing motion from an image (see, for example, Non-Patent Document 1 and Patent Document 5). MHI is an image drawn by filling the area where the brightness difference occurs for each frame with high brightness and gradually lowering the brightness in the subsequent frames, and is a single image having information on the direction of movement of the moving object. It has become.

Patent Document 5 discloses a technique for detecting a pitching motion from a baseball image using an image recognition technique, and creates a Motion History Image (MHI) for the baseball image to detect the pitching motion. There is. However, the MHI of the technique disclosed in Patent Document 5 does not detect the skeleton, and it is difficult to recognize the detailed operation.

Therefore, Skeleton motion history Image (Skeleton motion history Image), which generates MHI for an image (bone image) showing the connection line connecting the human skeleton and each skeleton obtained by performing skeleton detection, and measures the posture of the person from the camera image by deep learning technology. A technique called Skl MHI) is also disclosed (see, for example, Non-Patent Document 2).

Japanese Unexamined Patent Publication No. 2002-8063 Japanese Patent Application Laid-Open No. 2002-253718 Japanese Unexamined Patent Publication No. 2020-38440 Japanese Unexamined Patent Publication No. 2018-187383 Japanese Unexamined Patent Publication No. 2008-22142

"Motion History Image", [online], [Search on September 15, 2nd year of Reiwa], Internet <https://web.cse.ohio-state.edu/~davis.1719/CVL/Research/MHI/mhi .html> C. N. Ohyo, T. T. Zin, P. Tin., “Skeleton motion history based human action recognition using deep learning”, [online], [Search on September 15, 2nd year of Reiwa], Internet <https://ieeexplore.ieee. org / document / 8229448>

As described above, conventionally, in general, when adding tactile information to video content, it has been necessary to manually edit the type and timing of the stimulus. Therefore, it was impossible to present tactile information in a live broadcast program. If tactile information extraction can be automated by real-time video analysis, tactile information can be provided even in live broadcast programs. Then, in order to realize video viewing of video content using tactile sensation, it is necessary to extract the movement of a person object from the video content and generate tactile information corresponding to the extracted movement of the person object as tactile metadata. become.

In particular, sports broadcasts are content that emphasizes real-time performance. Therefore, tactile information about the competition must be given in real time and presented at the same time as the video. In many cases, tactile stimuli synchronized with the movement of the athlete are effective, and when extracting tactile metadata from the video, it is necessary to analyze the movement of the athlete in real time from the camera image. Since it does not affect the competition, it is desirable to extract tactile metadata from normal broadcast camera images without using motion capture by attaching markers or depth sensors with a limited shooting distance.

That is, a technique for automatically and in real time generating tactile metadata regarding the movement of a person object (player, etc.) is desired from only a normal camera image for shooting sports.

Also, in order to detect the movement of a person object with high accuracy, a technique of referring to a motion object other than the person (for example, shuttle, racket in the case of badminton competition) can be considered, but the motion object other than the reference person is Even in non-existent competitions (for example, judo and wrestling), a technique for detecting the movement of a person object with high accuracy is desired.

With the recent development of deep learning technology, it has become possible to estimate the skeleton position of a person from a normal still image that does not include depth information, which was difficult in the past, without using a depth sensor. The skeleton detection algorithm represented by this basically detects the skeleton position in units of still images. For this reason, further ingenuity is required to automatically and in real time generate tactile metadata related to the movement of a person object (player, etc.) from only a normal camera image for shooting sports.

By the way, as machine learning for motion recognition, although motion recognition can be performed at high speed by using a conventional supervised learning method such as SVM, further improvement in accuracy can be expected by using deep learning, which is desirable to be developed in recent years. CNN is often used for motion recognition based on video analysis. However, CNN is a still image-based identification algorithm and does not consider the time axis. In order to understand the movement content of a video scene, it is necessary to handle features related to the movement of a person, but since the still image does not contain time axis information, it is not possible to expect highly accurate identification of the movement content of CNN. ..

Therefore, it is conceivable to use an image called Motion History Image (MHI) as a method of recognizing the motion from the image by CNN. By analyzing this MHI, it becomes possible to recognize and judge the basic movements of a person such as "spreading arms", "squatting", and "raising hands". However, since MHI does not measure each part of a person's joints, it is limited to recognizing large movements using the whole body. For example, in order to create a Motion History Image (MHI) for a baseball image and detect a pitching motion as disclosed in Patent Document 5, the pitcher's area is set in order to suppress the influence of noise contained in the background. It is necessary to detect with high accuracy, and it is difficult to recognize detailed motion because it does not detect the skeleton.

Therefore, as disclosed in Non-Patent Document 2, a Motion History Image (MHI) is generated for an image (bone image) showing a connection line connecting a human skeleton obtained by performing skeleton detection and each skeleton, and a deep learning technique is used. The technology called Skeleton motion history Image (Skl MHI), which measures the posture of a person from a camera image, improves the accuracy of motion recognition, but further improvement of motion recognition accuracy is required.

In view of the above problems, an object of the present invention is a tactile metadata generator that automatically extracts a person object from a video and synchronizes and automatically generates tactile metadata corresponding to a dynamic person object, and a generated tactile meta. It is an object of the present invention to provide a video-tactile interlocking system and a program for driving and controlling a tactile presentation device based on data.

The tactile metadata generation device of the present invention is a tactile metadata generation device that extracts a person object from an image and generates tactile metadata corresponding to a dynamic person object, and is a current frame image of the input image. A multi-frame extraction means for extracting a multi-frame image including a predetermined number of past frame images, and a person who generates a first skeletal coordinate set of each person object based on a skeletal detection algorithm for each of the multiple frame images. By variably setting the search range based on the first skeleton coordinate set for each of the skeleton extraction means and the plurality of frame images, the position and size of the skeleton of each person object and the peripheral image information thereof are extracted. Based on the person identification means that identifies the person object and generates the second skeletal coordinate set to which the person ID is given, and the second skeletal coordinate set in each of the plurality of frame images based on the current frame image. , A person's identification motion is performed by a skeletal locus feature image generation means that generates one skeletal locus feature image showing only the direction of movement for each identified person skeleton, and a convolutional neural network that inputs the skeletal locus feature image. A person motion recognition means that recognizes and detects impact information for activating a predetermined tactile presentation device, and a tactile metadata including the impact presentation information corresponding to the current frame image is generated and a frame is generated. It is characterized by comprising a metadata generation means for externally outputting in units.

Further, in the tactile metadata generation device of the present invention, the skeleton locus feature image generation means draws a locus connected for each person's skeleton coordinates in the plurality of frame images as the skeleton locus feature image, and draws the locus. At that time, it is characterized in that the brightness is lowered or raised toward the past to obtain one image generated by drawing.

Further, in the tactile metadata generation device of the present invention, the skeleton locus feature image generation means uses the skeleton locus feature image as the skeleton locus feature image, and the skeleton coordinates of each person in the plurality of frame images are common to or distinguished from each person. It is characterized by color-coding each person's skeleton coordinates and drawing the movement of each person's skeleton coordinates in a time-series manner on a frame-by-frame basis to form a single image.

Further, in the tactile metadata generation device of the present invention, the person identification means is limited to a person search range that surrounds the entire human skeleton at the maximum as the search range, and a predetermined area of the human skeleton is the attention search range at the minimum. The search range is determined so as to include at least the attention search range based on the state transition estimated value of the skeleton of the person obtained by the state estimation algorithm, and the person object is identified. It is characterized by having a means for performing the processing.

Further, in the tactile metadata generation device of the present invention, a moving object is detected based on a difference image between adjacent frames using each of the plurality of frame images, and the identified person among the moving objects detected from each difference image. A motion object other than a person is selected in comparison with a skeletal trajectory feature image showing only the direction of movement for each skeleton, and the coordinate position, size, and movement direction obtained from each difference image are elements for the motion object other than the person. Further provided with a moving object detecting means for generating a moving object locus image connected to the above, the person motion recognizing means is a moving object locus image on a skeleton locus feature image showing only the direction of movement for each identified human skeleton. It is characterized by recognizing a specific motion of a person by a convolutional neural network whose input is a composite of images.

Further, the image tactile interlocking system of the present invention is predetermined based on the tactile metadata generation device of the present invention, the tactile presentation device for presenting the tactile stimulus, and the tactile metadata obtained from the tactile metadata generation device. It is characterized by comprising a control unit that refers to the drive reference data and controls the tactile presentation device to be driven.

Further, the program of the present invention is configured as a program for making the computer function as the tactile metadata generator of the present invention.

According to the present invention, a person object can be automatically extracted from a video, and tactile metadata corresponding to a dynamic person object can be automatically generated in synchronization. In particular, it is possible to present tactile stimuli during real-time viewing of sports images. By presenting information not only to the eyes and hearing but also to the sense of touch, it is possible to convey the situation of sports to people with visual and hearing disabilities in an easy-to-understand manner. In addition, even for general sighted people, it is possible to provide a sense of presence and immersiveness that cannot be conveyed by conventional video viewing.

It is a block diagram which shows the schematic structure of the image tactile interlocking system provided with the tactile metadata generation apparatus of one Embodiment by this invention. It is a flowchart which shows the processing example of the tactile metadata generation apparatus of one Embodiment by this invention. It is explanatory drawing about the person skeleton extraction processing in the tactile metadata generation apparatus of one Embodiment by this invention. (A) is a diagram illustrating a one-frame image, and (b) is a diagram showing an example of extracting a human skeleton in a one-frame image in the tactile metadata generator of one embodiment according to the present invention. (A) and (b) are diagrams showing a processing example of a search range of a person object related to a person skeleton extraction process in the tactile metadata generation device of one embodiment according to the present invention, respectively. (A) is a diagram showing an image example of a skeleton trajectory image (STI: Skeleton Trajectory Image) according to the present invention, and (b) is an explanatory diagram of the locus feature image (STI). (A) is a diagram simulating a one-frame image example, (b) is a conventional bone image example, and (c) is a conventional Skeleton Motion History Image (Skl MHI) image example, (d). ) Is a diagram showing an image example of the skeleton locus feature image (STI) according to the present invention. It is a figure which shows the comparative evaluation of the detection accuracy of the person movement of the bone image of the prior art, the Skl MHI of the prior art, and the skeletal locus feature image (STI) according to the present invention. It is a block diagram which shows the schematic structure of the control unit in the image tactile interlocking system of one Embodiment by this invention.

(System configuration)
Hereinafter, the video tactile interlocking system 1 provided with the tactile metadata generation device 12 according to the embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a video tactile interlocking system 1 including the tactile metadata generation device 12 of the embodiment according to the present invention.

The video tactile interlocking system 1 shown in FIG. 1 inputs video from a video output device 10 such as a camera or a recording device, automatically extracts a person object from the input video, and tactile metadata corresponding to the dynamic person object. Based on the tactile metadata generator 12 that automatically generates (two types of first tactile metadata and the second tactile metadata) and the generated tactile metadata, two tactile presentations are presented in this example. The devices 14L and 14R and a control unit 13 for individually driving and controlling the tactile presentation devices 14L and 14R are provided.

First, it is assumed that the video output by the video output device 10 is displayed on the display 11 as a real-time shot of the judo competition as an example, and is viewed by the user U.

Judo competition is a sport in which two athletes combine to compete for techniques such as "holding down" and "throwing". By presenting it to the user U as a tactile stimulus by 14R, it is possible to further enhance the sense of presence and to inform the audiovisually impaired person of the game situation.

In particular, in judo competitions, there are many overlaps and occlusions between players on the video, so in addition to the timing and speed according to the type of impact that occurs on each player, it is also possible to combine and throw each player's push and pull. By making it possible to continuously present the tactile sensation, it is possible to convey the sense of urgency of the game to the visually and hearing impaired, and to enhance the sense of presence.

Therefore, it is assumed that the user U grips the tactile presentation device 14L with the left hand HL and the tactile presentation device 14R with the right hand HR, and in this example, the vibration stimulus synchronized with the image analysis is presented. The control unit 13 is based on tactile metadata including impact presentation information indicating the timing and speed according to the type of impact generated in each person object Op1 and Op2 obtained from the tactile metadata generation device 12. The tactile presentations of the two tactile presentation devices 14L and 14R associated with the objects Op1 and Op2 are individually controlled. However, the control unit 13 may be driven and controlled only for one tactile presenting device, or may be individually driven and controlled for three or more tactile presenting devices. Further, although not limited to this, in the control unit 13 of this example, the vibration stimulus corresponding to the movement of the person object Op1 (player) in the video is the tactile presentation device 14L, and corresponds to the movement of the person object Op2 (player). The vibration stimulus generated shall be classified and controlled so as to be presented by the tactile presentation device 14R.

In the tactile presentation devices 14L and 14R, a vibration actuator 142 capable of presenting a vibration stimulus under the control of the control unit 13 is housed in a spherical case 141. The tactile presentation devices 14L and 14R may present electromagnetic pulse stimulation in addition to vibration stimulation. In this example, a mode in which the control unit 13 and the tactile presentation devices 14L and 14R are connected by wire and the tactile metadata generator 12 and the control unit 13 are also connected by wire will be described. Each may be wirelessly connected by short-range wireless communication.

The tactile metadata generation device 12 includes a plurality of frame extraction unit 121, a person skeleton extraction unit 122, a person identification unit 123, a skeleton trajectory feature image generation unit 124, a moving object detection unit 125, a person motion recognition unit 126, and a metadata generation unit. 127 is provided.

The multi-frame extraction unit 121 extracts a plurality of frame images including the current frame image and the past frame image for T (T is an integer of 1 or more) frames from the video input from the video output device 10, and extracts the human skeleton. It is output to the unit 122 and the moving object detection unit 125.

The human skeleton extraction unit 122 has a skeleton coordinate set P ⁿ _b (n: number of people to be detected) of each person object (hereinafter, also simply referred to as “person”) Op1 and Op2 based on the skeleton detection algorithm for each of the plurality of frame images. , B: Skeleton ID) and output to the person identification unit 123 together with the plurality of frame images including the current frame image.

The person identification unit 123 variably sets a search range (details will be described later) based on the skeleton coordinate set P ⁿ _b for each of the plurality of frame images, and sets the position and size of the skeleton of each person and the peripheral image thereof. The person is identified by extracting the information, a skeleton coordinate set Pi _b ( ⁱ : person ID, b: skeleton ID) to which the person ID is given is generated, and is output to the skeleton trajectory feature image generation unit 124.

The skeleton locus feature image generation unit 124 is a single skeleton locus feature image showing only the direction of movement for each identified person skeleton based on the skeleton coordinate set ^Pi _b in the plurality of frame images with reference to the current frame image. Is generated and output to the person motion recognition unit 126. Here, the details of the skeletal trajectory feature image will be described later, but in the present specification, it is named STI (Skeleton Trajectory Image).

The motion object detection unit 125 is not necessarily provided for recognizing the motion of the judo competition as in this example, but uses each of the plurality of frame images to generate a motion object based on the difference image between adjacent frames. Among the moving objects detected from each difference image, a moving object other than the person is selected by comparing with the skeletal locus feature image (STI) obtained from the skeletal locus feature image generation unit 124, and the moving object other than the person is selected. , A moving object locus image connected with the coordinate position, size, and movement direction obtained from each difference image as elements is generated, and is output to the skeleton locus feature image generation unit 124. In this case, the skeleton locus feature image generation unit 124 adds a moving object locus image on the skeleton locus feature image (STI) and draws (synthesizes) the image, and outputs the image to the person motion recognition unit 126.

The human motion recognition unit 126 recognizes a specific motion of a person by a CNN (convolutional neural network) that inputs a skeletal locus feature image (STI), and operates predetermined impact presentation devices 14L and 14R. That is, the identification of each person in the current frame image, the position coordinates (and, in this example, it is excluded because it is a judo competition, but if it is a team competition, the team classification), and the timing to activate the tactile presentation device. And the information for impact presentation indicating the speed is detected and output to the metadata generation unit 127.

The metadata generation unit 127 corresponds to the current frame image and is obtained from the person motion recognition unit 126 for information for presenting a predetermined impact, that is, identification of each person in the current frame image, position coordinates (and in this example,). It is out of scope because it is a judo competition, but if it is a team competition, its team classification), and tactile metadata (for impact presentation) that includes information for impact presentation indicating the timing and speed at which the tactile presentation device is activated. Is generated and output to the control unit 13 in frame units.

Hereinafter, the tactile metadata generation process in the tactile metadata generation device 12 will be described more specifically with reference to FIGS. 3 to 6 with reference to FIG. 2.

(Tactile metadata generation process)
FIG. 2 is a flowchart showing a processing example of the tactile metadata generation device 12 according to the embodiment of the present invention. FIG. 3 is an explanatory diagram relating to the human skeleton extraction process in the tactile metadata generation device 12. Further, FIG. 4A is a diagram illustrating a one-frame image, and FIG. 4B is a diagram showing an example of extracting a human skeleton in a one-frame image in the tactile metadata generation device 12. 5 (a) and 5 (b) are diagrams showing a processing example of a search range of a human object related to a human skeleton extraction process in the tactile metadata generation device 12 of the embodiment according to the present invention, respectively. FIG. 6A is a diagram showing an image example of the skeleton locus feature image (STI) according to the present invention, and FIG. 6B is an explanatory diagram of the locus feature image (STI).

As shown in FIG. 2, the tactile metadata generation device 12 first receives the current frame image and T (T is an integer of 1 or more) frames of the video input from the video output device 10 by the plurality of frame extraction units 121. A plurality of frame images including the past frame images of the above are extracted (step S1).

Subsequently, the tactile metadata generation device 12 uses the person skeleton extraction unit 122 to obtain the skeleton coordinate set P ⁿ _b (n: number of people to be detected) of each person object Op1 and Op2 based on the skeleton detection algorithm for each of the plurality of frame images. , B: Skeleton ID) (step S2).

Recent developments in deep learning techniques have made it possible to estimate the skeleton position of a person from ordinary images. As represented by OpenPose and VisionPose (NextSystem), there are some open source skeleton detection algorithms. Therefore, the human skeleton extraction unit 122 of this example detects 30 points of the human skeleton for each frame image using VisionPose, and generates a skeleton coordinate set P ⁿ _b indicating the position coordinates thereof. do.

In VisionPose, in FIG. 3, P ⁿ ₁ : "head", P ⁿ ₂ : "nose", P ⁿ ₃ : "left eye", P ⁿ ₄ : "right eye", P ⁿ ₅ : "left ear", P ⁿ . ₆ : "Right ear", P ⁿ ₇ : "Neck", P ⁿ ₈ : "Spine (shoulder)", P ⁿ ₉ : "Left shoulder", P ⁿ ₁₀ : "Right shoulder", P ⁿ ₁₁ : "Left elbow" , P ⁿ ₁₂ : "Right elbow", P ⁿ ₁₃ : "Left wrist", P ⁿ ₁₄ : "Right wrist", P ⁿ ₁₅ : "Left hand", P ⁿ ₁₆ : "Right hand", P ⁿ ₁₇ : " Left thumb ”, P ⁿ ₁₈ :“ Right thumb ”, P ⁿ ₁₉ :“ Left fingertip ”, P ⁿ ₂₀ :“ Right fingertip ”, P ⁿ ₂₁ :“ Spine (center) ”, P ⁿ ₂₂ :“ Spine (base) End) ”, P ⁿ ₂₃ :“ Left butt ”, P ⁿ ₂₄ :“ Right butt ”, P ⁿ ₂₅ :“ Left knee ”, P ⁿ ₂₆ :“ Right knee ”, P ⁿ ₂₇ :“ Left ankle , P ⁿ ₂₈ : "Right ankle", P ⁿ ₂₉ : "Left foot", and P ⁿ ₃₀ : "Right foot", and each coordinate position can be drawn by connecting them with a line as shown in the figure. Is.

Based on this Vision Pose skeleton detection algorithm, the frame image Fa obtained by extracting the skeleton of a person from the one-frame image F of the judo competition shown in FIG. 4 (a) is shown in FIG. 4 (b). The frame image F shown in FIG. 4A shows that only each person object Op1 and Op2 (players) are reflected, but other person objects such as the referee's moving object are reflected. In another sports competition, a moving object other than a person (such as a racket or shuttle in a badminton competition) or an object such as a spectator (substantially a static object) may be reflected. However, by applying VisionPose's skeleton detection algorithm, it is possible to extract the skeleton extraction of a person only for the person of the player and the referee's person object. In this example, as shown in FIG. 4 (b), the skeleton coordinate sets P ¹ _b and P ² _b corresponding to the person objects Op 1 and Op 2 can be estimated and generated. As can be understood from FIG. 4B, the skeleton of each person can be estimated relatively accurately even in judo competition. Since the skeleton detection algorithm is limited to estimation in units of still images, the tactile metadata generation device 12 identifies a person as a subsequent process, and changes the skeleton position of each person as one skeleton trajectory feature. It is drawn on an image (STI), and high-precision motion recognition is performed by CNN in consideration of the time axis.

Subsequently, the tactile metadata generation device 12 variably sets the search range for each of the plurality of frame images by the person identification unit 123 based on the skeleton coordinate set P ⁿ _b , and sets the position and size of the skeleton of each person. , A person is identified by extracting the peripheral image information thereof, and a skeleton coordinate set Pi _b ( ⁱ : person ID, b: skeleton ID) to which the person ID is given is generated (step S3).

The human skeleton extraction unit 122 described above makes it possible to detect the skeleton of one or more people as the skeleton coordinate set P ⁿ _b for each of the plurality of frame images. However, since the information of "who" does not exist in the skeleton coordinate set P ⁿ _b of each frame image, it is necessary to identify the skeleton of each person. For this identification, image information near the coordinates of each skeleton coordinate set P ⁿ _b in each frame image is used. That is, the person identification unit 123 extracts the position and size of the skeleton of each person and the peripheral image information (color information and texture information near the face or back) based on the skeleton coordinate set P ⁿ _b . , A person is identified, and a skeleton coordinate set Pi _b ( ⁱ : person ID, b: skeleton ID) to which a person ID is given is generated.

For example, in judo, a match is played in white and blue dogi, but by referring to the color information in the frame image F as the image information near the position of the skeleton of each skeleton coordinate set P ⁿ _b , the player can be identified. It will be possible. In the badminton competition, when the court is held vertically at an angle of view, if the position of the skeleton of each skeleton coordinate set P ⁿ _b is on the upper side of the screen in the frame image F, the player in the back and the lower side of the screen. If there is, it can be identified as the player in the foreground.

Therefore, although the skeleton detection algorithm in the human skeleton extraction unit 122 is limited to estimation in units of still images, it is possible to recognize a person as a moving object based on the skeleton coordinate set P ⁿ _b .

In addition to the athletes, the above-mentioned human skeleton extraction unit 122 often detects the skeletons of other persons such as referees and spectators who are not the targets of the presentation of tactile stimuli. Referees often wear different clothing than athletes, so they can be identified by color information. Also, since the spectators are often farther than the athletes, they can be identified by the size of the skeleton. In this way, by setting the peripheral image information (color information and texture information near the face or back) appropriate for person identification in consideration of the rules and shooting conditions of each competition, the athlete to be presented with the tactile stimulus. Can be identified.

By the way, the person identification unit 123 of the present embodiment variably sets the search range (person search range R ⁱ and attention search range R ^bi ) for each frame image in order to deal with the overlap and occlusion of each person. For example, when the person skeleton extraction unit 122 extracts each skeleton coordinate set P ⁿ _b (not shown) for the person objects Op1 and Op2 (players) and the person object Op3 (referee) shown in FIG. 5 (a). The person identification unit 123 can variably set the person search range R ⁱ and the attention search range R ^bi for each frame image. This search range R ⁱ is set for each person ID (i) in FIG. 5A, and has the position coordinates of the person on the image coordinates of the frame image and the size (width and height) of the person. It is represented by an circumscribing rectangle. Further, the region surrounding the waist region (P ⁿ ₂₂ , P ⁿ ₂₃ , P ⁿ ₂₄ ) of each person is represented as the attention search range ^Rbi .

More specifically, the person identification unit 123 of the present embodiment limits the search range of the person in each frame image to the person search range ^Ri that surrounds the entire person skeleton at the maximum, and determines the person skeleton at the minimum. A variable setting is made by narrowing down the area (the area surrounding the waist area (P ⁿ ₂₂ , P ⁿ ₂₃ , P ⁿ ₂₄ ) in this example) as the attention search range ^Rbi , and the skeleton of the person obtained by the state estimation algorithm is set. Based on the state transition estimated value, the search range is determined so as to include at least the attention search range ^Rbi , and the process of identifying the person object is performed. This prevents misrecognition of other people and the processing speed even when the movement of each person changes or the size of the person relative to the frame image changes, for example, as shown in FIG. 5 (b). Can also be improved. In particular, it is effective to use the search range in order to accurately identify a player from an image in which the persons to be identified are heavily overlapped and the background is complicated as in judo.

That is, the person identification unit 123 of the present embodiment has a predetermined range (in this example, the waist region) that enables color identification among the skeleton coordinate sets ^Pn _b in Op1, Op2, and Op3 of the person objects of each player and the referee. Since the colors (blue, white, brown) of (P ⁿ ₂₂ , P ⁿ ₂₃ , P ⁿ ₂₄ ) are predetermined as the attention search range ^Rbi , the skeleton coordinate sets P ⁿ _b of a plurality of detected persons overlap. In the case, by narrowing down the search to the attention search range ^Rbi , it is possible to accurately extract and track a person in each frame image. Since a skeleton other than the analysis target may be detected in the background, the skeleton of the person to be analyzed is determined by assigning a person ID ( ⁱ ) to the skeleton of the person to be _tracked . Can be identified.

The width and shape of the search range (person search range R ⁱ and attention search range R ^bi ) are determined at least based on the state transition estimates of the human skeleton obtained by a state estimation algorithm such as a Kalman filter or a particle filter. It is determined to include the attention search range ^Rbi (in this example, the waist region of each person).

Then, the range can be narrowed when the search range (person search range R ⁱ and attention search range R ^bi ) is stably detected, and the range can be expanded when the detection is unstable. For example, each person ID (i) can be expanded. A search range determined based on the state transition estimated value of the skeleton of is set, and if the state transition estimated value is within a predetermined value from the immediately preceding frame, it is made stable, otherwise it is made unstable, or a state estimation algorithm. Based on the state transition estimation value of the skeleton of the person obtained in, the ratio of successful detection is calculated during the time window for T frames, and if the ratio is equal to or more than the predetermined value, it is stabilized and falls below the predetermined value. In this case, the search range can be variably set by making it unstable.

Subsequently, the tactile metadata generation device 12 uses the skeleton trajectory feature image generation unit 124 to identify the movement of each person's skeleton based on the skeleton coordinate set ^Pi _b in the plurality of frame images with reference to the current frame image. A single skeletal locus feature image (STI) showing only the direction is generated (step S4).

Here, in generating the drawing of the skeleton locus feature image (STI), first, the skeleton coordinate set Pi _{ib in an arbitrary frame image is set to Pi b} ⁽ _t ), the current frame image is set to t ⁼ 0, and the skeleton in the current frame image is set. The coordinate set P ⁱ _b is represented by P ⁱ _b (0), and the skeleton coordinate set P ⁱ _b in the frame image of the past T frame is represented by P ⁱ _b (T). That is, when the skeleton locus feature image generation unit 124 expresses the frame number up to the past T frame by t = T, where the frame number of the current frame image is t = 0, t = 0,1 with respect to the current frame image. Using each frame image F of, ..., T, it is possible to generate one skeleton locus feature image (STI) showing only the direction of movement for each identified human skeleton. The skeleton locus feature image (STI) is, so to speak, concatenated past optical flows with reference to the current frame image, and includes information on the time axis as one image.

The T, which is the duration of the locus feature amount in the skeleton locus feature image (STI), can be arbitrarily set. Further, it is not always necessary to use all 30 points shown in FIG. 3 as the skeleton coordinates used for generating one skeleton locus feature image (STI), and the processing speed is determined by using only a predetermined specific skeleton locus. It can also be configured to improve.

The skeleton locus feature image (STI) uses the skeleton coordinates of each person in the frame image for the past T frames from the current frame image, and draws a locus connected for each skeleton coordinate of each person, and draws this drawing. At that time, it is made into one image generated by drawing with the brightness lowered or increased toward the past. Preferably, the skeleton locus feature image (STI) is color-coded for each person's skeleton coordinates in the past frame image for T frames from the current frame, and the movement (transition) for each person's skeleton coordinates is frame-by-frame. It is assumed that the image is drawn so as to be gradation in time series.

For example, when drawing a locus connected for each skeleton coordinate of each person from the current frame to the past T frame, the brightness b is b = 255 × (Tt) / T.
It shall be determined as.

In addition, you may draw so that the brightness increases as you go back in time. In this case,
b = 255 × t / T
Can be.

Here, when t = 0 is the current frame image and the past T frames are the processing targets (t = 0 to T), b is set to 0 to 255, and the values are color-coded for each person's skeleton coordinates. It is preferable to express it. For example, FIG. 6A is a diagram showing an image example of a skeleton locus feature image (STI) according to the present invention. In FIG. 6A, it is assumed that the grayscale display is used for the recognition process, but preferably, as shown in the explanatory diagram of the locus feature image (STI) shown in FIG. 6B, for example, the background has the lowest luminance value. For _{any of the person objects Op1 and Op2, for example, the color of "head" (P1 1) and (P2 1} ^{) is} _" blue". In addition, the color of the "left fingertip" (P ¹ ₁₉ ) and (P ² ₁₉ ) is set to "red", and the drawing is performed by color-coding with a color that can be distinguished in advance. Further, in this embodiment, as shown in FIG. 6B, the person objects Op1 and Op2 are not color-coded to distinguish them, but each person object Op1 and Op2 may also be color-coded, for example, two persons. In order to color-code the skeleton coordinates of up to 30 points, 60 colors may be defined. The skeleton locus feature image (STI) according to the present invention is drawn darker (or brighter) so that only the brightness goes back to the past from the original frame image while the color is fixed for each skeleton coordinate of each person.

Therefore, the skeleton locus feature image generation unit 124, as the skeleton locus feature image (STI), common or distinguishes the skeleton coordinates of each person in the plurality of frame images for each person, and color-codes each person's skeleton coordinates. , It is assumed that the movement (transition) of each person's skeleton coordinates is drawn so as to be gradation in time series in frame units.

Further, the skeleton locus feature image generation unit 124 can draw a locus other than the human skeleton, such as a ball in the case of a ball game, on the skeleton locus feature image (STI) by the function of the moving object detection unit 125. In this case, since the moving direction of the ball is added to the feature amount, the determination accuracy of motion recognition is improved.

That is, the tactile metadata generation device 12 detects a moving object based on the difference image between adjacent frames by using each of the plurality of frame images by the moving object detection unit 125, and the moving object detected from each difference image. Among them, a moving object other than a person is selected in comparison with the skeletal locus feature image (STI) obtained from the skeletal locus feature image generation unit 124, and the coordinate position and size of the moving object other than the person are obtained from each difference image. , A moving object locus image connected with the movement direction as an element is generated, and is output to the skeleton locus feature image generation unit 124. In this case, the skeleton locus feature image generation unit 124 adds a moving object locus image on the skeleton locus feature image (STI) and draws (synthesizes) the image, and outputs the image to the person motion recognition unit 126 (step S5).

That is, the moving object detection unit 125 is not required in a case such as a judo competition in which there is no moving object other than the person involved in the competition (there is no harmful effect even if it is provided as a process), but the person other than the person involved in the competition. When there is a moving object of (for example, shuttle or racket of badminton competition, ball or racket of table tennis or tennis competition), the movement trajectory of the moving object other than that person is detected, generated as a moving object trajectory image, and the skeleton A moving object locus image is added to the locus feature image generation unit 124 on the skeleton locus feature image (STI) and drawn (combined). As a result, for example, when a moving object other than a person involved in the competition exists, information related to the movement of the person involved in the competition increases, so that the recognition accuracy of the person movement in the subsequent person movement recognition unit 126 is improved. Therefore, by providing the moving object detection unit 125, the same processing can be applied to any competition, so that a versatile tactile metadata generation device 12 can be configured.

Subsequently, the tactile metadata generation device 12 recognizes a specific motion of a person by a CNN (convolutional neural network) inputting a skeletal locus feature image (STI) by a person motion recognition unit 126, and a tactile presentation device 14L, The information for presenting a predetermined impact that activates the 14R is detected (step S6). The information for impact presentation includes identification of each person in the current frame image, position coordinates (and, in this example, it is excluded because it is a judo competition, but if it is a team competition, its team classification), and tactile sensation. It contains information indicating when and how fast the presentation device is activated.

At the time of machine learning by CNN, a skeletal trajectory feature image (STI) for learning is created and trained in advance. As described above, CNN (convolutional neural network), which is one of deep learning, is used for the recognition process in the person motion recognition unit 126. CNN is a neural network with many deep layers, and is a network that exhibits excellent performance especially in the field of image recognition. This network is composed of layers with some characteristic functions such as "convolution layer" and "pooling layer", and is currently used in a wide range of fields. The processing of the "convolution layer" realizes high accuracy, and the processing of the "pooling layer" realizes versatility that does not depend on the shooting angle of view.

By analyzing the skeletal trajectory feature image (STI) using this CNN, motion events such as "combination", "throwing", and "ground fighting" can be identified with high accuracy regardless of the shooting size and position of the athlete. By generating tactile metadata for controlling the tactile devices 14L and 14R based on this information, it is possible to present tactile stimuli even during real-time viewing of sports images.

Finally, the tactile metadata generation device 12 is provided with the information for presenting a predetermined impact obtained from the person motion recognition unit 126 corresponding to the current frame image by the metadata generation unit 127, that is, each of the information in the current frame image. For impact presentation that indicates the identification of a person, position coordinates (and, in this example, it is not applicable because it is a judo competition, but if it is a team competition, its team classification), and the timing and speed at which the tactile presentation device is activated. Tactile metadata (for impact presentation) including the above information is generated and output to the control unit 13 in frame units (step S7).

(Experimental verification)
An evaluation experiment was conducted to show the effectiveness of the tactile metadata generator 12 according to the present invention.
FIG. 7 (a) is a diagram simulating a one-frame image example, FIG. 7 (b) is a conventional bone image example, and FIG. 7 (c) is a conventional Skeleton Motion History Image (Skeleton Motion History Image). 7 (d) is an image example of the skeleton locus feature image (STI) according to the present invention. Further, FIG. 8 is a diagram showing a comparative evaluation of the detection accuracy of the human movement of the bone image of the prior art, the Skl MHI of the prior art, and the skeletal locus feature image (STI) according to the present invention.

First, before comparative evaluation, from the judo match video (see FIG. 7 (a)), the bone image of the prior art (see FIG. 7 (b)), the Skeleton MHI of the prior art (see FIG. 7 (c)), For the skeletal locus feature image (STI) (see FIG. 7 (d)) according to the present invention, about 2,000 images were created for each of the positive and negative examples, and pre-learning was performed by CNN.

The results identified in another match video are shown in FIG. In FIG. 8, as a comparison between the identification results of the four game situations (scene classification) of "attendance", "throwing", "ground fighting", and "waiting" and the detection results of the "throwing" operation, the matching rate and the recall rate are shown. And the value of F value (F-Measure) which is an integrated index of these is shown. In any case of the identification judgment of the state of the four game situations (scene classification) and the detection accuracy of "throwing", the best result is obtained by learning using the skeletal trajectory feature image (STI) according to the present invention. Obtained. Therefore, it was possible to confirm the effectiveness of the tactile metadata generation device 12 using the skeletal trajectory feature image (STI) according to the present invention, rather than the motion recognition using the bone image of the prior art or the Skl MHI of the prior art. Although Skl MHI was also evaluated by color-coding each skeleton coordinate, the reason why using the skeleton locus feature image (STI) according to the present invention improves the accuracy of motion recognition is that Skl MHI (bone image). In the same way), it is considered that the connecting line connecting each skeleton has an adverse effect on motion recognition.

(Controller unit)
FIG. 9 is a block diagram showing a schematic configuration of the control unit 13 in the video-tactile interlocking system 1 of the embodiment according to the present invention. The control unit 13 includes a metadata receiving unit 131, an analysis unit 132, a storage unit 133, and a driving unit 134-1,134-2.

The metadata receiving unit 131 is a functional unit that inputs tactile metadata (for impact presentation) from the tactile metadata generation device 12 and outputs it to the analysis unit 132. Tactile metadata includes identification of each person in the current frame image, position coordinates (and their team classification if in a team competition), as well as information indicating when and how fast the tactile presentation device is activated.

The analysis unit 132 refers to predetermined drive reference data based on the tactile metadata obtained from the tactile metadata generation device 12, and corresponds to each tactile presentation device via the drive units 134-1 and 134-2. It is a functional unit that controls to drive the vibration actuator 142 of 14L and 14R. For example, the analysis unit 132 refers to the tactile sense by referring to the predetermined drive reference data from the identification of the person in the tactile metadata, the position coordinates, (and the team classification), and the timing and speed of operating the tactile presentation device. The operation timing, strength, and operation time of the vibration actuator 142 of the presentation device 14L are determined and drive-controlled.

The storage unit 133 stores predetermined drive reference data for controlling the drive of the drive units 134-1, 134-2 based on the tactile metadata. The drive reference data is represented by a predetermined table or function regarding the operation timing, strength, and operation time of the vibration actuator 142 as a tactile stimulus associated with the tactile metadata. Further, the storage unit 133 stores a program for realizing the function of the control unit 13. That is, the function of the control unit 13 is realized by reading and executing the program by the computer constituting the control unit 13.

The drive units 134-1 and 134-2 are drivers that drive the vibration actuators 142 of the tactile presentation devices 14L and 14R.

As described above, according to the image tactile interlocking system 1 provided with the tactile metadata generation device 12 of the present embodiment, the person object is automatically extracted from the image, and the tactile metadata corresponding to the dynamic person object is automatically synchronized. Since it can be generated, the tactile presentation device and the image can be linked.

In particular, the tactile metadata generation device 12 of the present embodiment takes the image of the camera as an input, analyzes the image, and outputs the tactile metadata as described above. Therefore, the image is first analyzed to form a human skeleton in the image. Is detected, a person to be analyzed is specified using the detected skeleton information, and then a skeleton locus feature image (STI) is drawn and generated from the history of the skeleton position of the target person. On this skeleton locus feature image (STI), a locus image of a moving object other than a person such as a ball may be combined.

Then, the tactile metadata generation device 12 of the present embodiment uses the CNN that inputs the skeletal locus feature image (STI) to display the player's motion event (“throw”, “combination”, etc.) and the status of the motion. Recognize and generate tactile metadata (for impact presentation) of the corresponding human movement. Then, the control unit 13 is based on the tactile metadata (for impact presentation) obtained from the tactile metadata generation device 12 of the present embodiment, and the vibration stimulus corresponding to the movement of the person object Op1 (player) in the image is tactile. In the presentation device 14L, the vibration stimulus corresponding to the movement of the person object Op2 (player) is classified and controlled so as to be presented by the tactile presentation device 14R.

Therefore, the video-tactile interlocking system 1 of the present embodiment can continuously convey the situation such as pushing and pulling of the player in addition to the motion event such as "throwing", and the handicapped person feels the sense of urgency of the game. Can be conveyed and the sense of presence can be enhanced.

By the way, by analyzing an image called a conventional Motion History Image (MHI), it is possible to determine basic movements of a person such as "spreading arms", "squatting", and "raising hands". Since each part of a person's joint is not measured, it is limited to recognition of large movements using the whole body. On the other hand, the skeleton locus feature image (STI) according to the present invention is an image showing an image feature amount that can be said to be an improved version of this MHI, and is an image showing the locus of the skeleton of each person, or in addition to this, a person other than the person to be tracked. By drawing the locus information of the moving object, it is possible to perform highly accurate recognition while suppressing the influence of noise contained in the background. In addition, since the image is created using the transition of the skeletal coordinates of each person, it is possible to recognize not only the whole body movement but also the partial movement of the hands and feet with high accuracy.

In particular, the skeleton detection algorithm is limited to posture estimation in units of still images, but the skeleton locus feature image (STI) according to the present invention treats the transition of each skeleton position as a locus feature, and treats this locus feature amount as one image. By expressing with, it is possible to identify the operation by CNN. That is, by using the skeletal locus feature image (STI) according to the present invention as an input for the feature in the time axis direction, which was difficult in CNN, it is possible to recognize the motion of a person with high accuracy.

The tactile metadata generation device 12 of the above-described embodiment can be made to function as a computer, and the computer is provided with a program for realizing each component according to the present invention inside or outside the computer. It is stored in the memory to be stored. A program in which processing contents for realizing the functions of each component are appropriately read from a memory under the control of a central processing unit (CPU) provided in a computer, and tactile metadata generation of the present embodiment is performed. The function of each component of the device 12 can be realized in the computer. Here, the function of each component may be realized by a part of hardware.

Although the present invention has been described above with reference to examples of specific embodiments, the present invention is not limited to the examples of the above-described embodiments, and can be variously modified without departing from the technical idea. For example, in the example of the above-described embodiment, the video analysis of judo competition has been mainly described as an example, but it can be widely applied to badminton, table tennis, various other sports events, and non-sports video. For example, it will lead to improvement of services such as public viewing using tactile information, entertainment, and future tactile broadcasting. In addition, as an example other than sports, it can be applied to various applications such as a tactile alarm in a factory and a security system based on surveillance camera image analysis. Therefore, the present invention is not limited to the example of the above-described embodiment, but is limited only by the description of the scope of claims.

According to the present invention, a person object can be automatically extracted from an image, and tactile metadata corresponding to a dynamic person object can be automatically generated in synchronization, which is useful for linking a tactile presentation device and an image. be.

1 Video tactile interlocking system 10 Video output device 11 Display 12 Tactile metadata generation device 13 Control unit 14L, 14R Tactile presentation device 121 Multiple frame extraction unit 122 Person skeleton extraction unit 123 Person identification unit 124 Skeletal trajectory feature image generation unit 125 Dynamic object Detection unit 126 Person motion recognition unit 127 Metadata generation unit 131 Metadata reception unit 132 Analysis unit 133 Storage unit 134-1,134-2 Drive unit 141 Case 142 Vibration actuator

Claims (7)

A tactile metadata generator that extracts a person object from a video and generates tactile metadata corresponding to a dynamic person object.
A multi-frame extraction means for extracting a multi-frame image including a current frame image and a predetermined number of past frame images from the input video, and
For each of the plurality of frame images, a person skeleton extraction means for generating a first skeleton coordinate set of each person object based on a skeleton detection algorithm, and
For each of the plurality of frame images, the search range is variably set based on the first skeleton coordinate set, and the person object is identified by extracting the position and size of the skeleton of each person object and the peripheral image information thereof. , A person identification means for generating a second skeletal coordinate set with a person ID,
Based on the current frame image and the second skeleton coordinate set in each of the plurality of frame images, a skeleton trajectory that generates a single skeleton trajectory feature image showing only the direction of movement for each identified person skeleton. Feature image generation means and
A human motion recognition means that recognizes a specific motion of a person by a convolutional neural network that inputs a skeletal trajectory feature image and detects information for impact presentation that activates a predetermined tactile presentation device.
A metadata generation means that generates tactile metadata including information for presenting the impact corresponding to the current frame image and outputs it externally in frame units.
A tactile metadata generator characterized by comprising. The skeleton locus feature image generation means draws a locus connected for each skeleton coordinate of each person in the plurality of frame images as the skeleton locus feature image, and at the time of this drawing, the brightness is lowered toward the past. The tactile metadata generation device according to claim 1, wherein the image is generated by raising or drawing the image. The skeleton locus feature image generation means, as the skeleton locus feature image, common or distinguishes the skeleton coordinates of each person in the plurality of frame images for each person, and color-codes each person's skeleton coordinates for each person. The tactile metadata generation device according to claim 1 or 2, wherein the motion of each skeleton coordinate is drawn so as to be gradation in time series on a frame-by-frame basis to form a single image. The person identification means limits the search range to a person search range that surrounds the entire human skeleton at the maximum, and performs variable setting by narrowing down a predetermined area of the human skeleton as the attention search range at the minimum, and estimates the state. It is characterized by having a means for determining the search range so as to include at least the attention search range based on the state transition estimation value of the skeleton of the person obtained by the algorithm, and performing a process of identifying the person object. , The tactile metadata generator according to any one of claims 1 to 3. A moving object is detected based on a difference image between adjacent frames using each of the plurality of frame images, and among the moving objects detected from each difference image, a skeletal trajectory feature showing only the direction of movement for each of the identified human skeletons. A moving object that selects a moving object other than a person in comparison with an image, and generates a moving object locus image that is connected with the coordinate position, size, and movement direction obtained from each difference image as elements for the moving object other than the person. With more detection means
The person motion recognition means is a convolutional neural network in which the motion object locus image is added and synthesized on a skeleton locus feature image showing only the movement direction of each identified person skeleton, and the person's motion recognition means is used. The tactile metadata generator according to any one of claims 1 to 4, characterized in that it recognizes a specific motion. The tactile metadata generator according to any one of claims 1 to 5.
A tactile presentation device that presents tactile stimuli,
A control unit that controls to drive the tactile presentation device by referring to predetermined drive reference data based on the tactile metadata obtained from the tactile metadata generator.
A video-tactile interlocking system characterized by being equipped with. A program for causing a computer to function as the tactile metadata generator according to any one of claims 1 to 5.

Images