JP7344096B2 - Haptic metadata generation device, video-tactile interlocking system, and program - Google Patents

Description

The present invention provides a tactile metadata generation device that extracts a person object from a video and generates tactile metadata corresponding to the dynamic person object, and a video-tactile link that drives and controls a tactile presentation device based on the generated tactile metadata. Regarding systems and programs.

Viewing video content, such as general camera footage, appeals to the two senses of sight and hearing, but by stimulating the tactile sense at the same time as the video, video viewing with a higher sense of realism and immersion can be achieved. It becomes possible. For example, when watching a baseball video, a tactile presentation device provides stimulation to the viewer at the timing when the ball hits the bat, allowing the viewer to simulate the sensation of a batter's hitting. It is also believed that providing tactile stimulation to people with visual impairments will help them understand the situation in sports matches. In this way, the sense of touch is expected to be the third sense in video viewing.

In particular, since real-time video viewing is important in sports, presentation of tactile stimulation for video needs to be performed automatically and in real time. Therefore, presenting tactile stimulation in synchronization with the player's movements is often effective for viewing video content that also uses the sense of touch.

Therefore, in order to realize video viewing of video content using the sense of touch, it is necessary to extract the movement of a human object from the video content and generate tactile information corresponding to the extracted movement of the human object as haptic metadata. It becomes necessary.

However, with conventional haptic metadata generation methods, even if video viewing using tactile sensation is realized, haptic metadata that indicates at what timing and what kind of stimulus is presented to the user by a haptic presentation device is not available. It was necessary to manually edit the data in synchronization with the video.

In the case of recorded programs, it is possible to manually edit tactile metadata over time. However, in order to link the stimulus presentation by a tactile presentation device to live broadcast video, it is not possible to edit the tactile information in advance, so it is necessary to perform video analysis of the video content in real time and generate tactile metadata. required.

In recent years, sports video analysis technology has achieved remarkable growth. The tennis Hawkeye system, which is also used at Wimbledon, uses multiple fixed camera images as sensors to track tennis balls three-dimensionally and makes IN/OUT decisions that are relevant to the judges. Also, at the 2014 FIFA World Cup, goal line technology was used to analyze footage from several fixed cameras and automate goal decisions. Furthermore, real-time video analysis technology in sports is becoming more sophisticated, such as the TRACAB system, which installs many stereo cameras in soccer stadiums and tracks all players on the field in real time.

On the other hand, in order to measure the posture of a player as a dynamic human object, measurement using a marker-type motion capture method has conventionally been common. However, this method requires many markers to be attached to the players' bodies, and cannot be applied to actual matches. Therefore, in recent years, it has become possible to measure human posture without markers by using depth sensors that read the infrared pattern projected onto the athlete's body and obtain depth information from the distortion of the infrared pattern. Furthermore, various techniques have been disclosed that apply an optical motion capture method instead of a marker method (for example, see Patent Documents 1 and 2).

For example, in Patent Document 1, in a virtual reality system using stereoscopic vision, when a user is taught a motion by displaying a model motion image of another person, an optical motion capture method is used to An apparatus for measuring the three-dimensional position of a skeleton is disclosed. Further, Patent Document 2 discloses a training evaluation device that measures a player's motion using an optical motion capture method and evaluates the player's form based on the measured data and data regarding the form of the model. has been done. However, since these technologies use motion capture methods, they cannot be applied to actual matches, and it is difficult to measure the playing movements of people from general-purpose camera images.

In addition, devices have been disclosed that simulate the movement of a person based solely on camera footage of a badminton match or badminton practice taken by one person or a group of two people, without using the motion capture method (for example, a patent (See Reference 3). The technology of Patent Document 3 detects actions such as shots from captured camera images, but it is based on the premise that processing is performed from images captured by a specially installed camera, and it is not compatible with general-purpose broadcast camera images. It is difficult to measure a person's playing movements from the ground.

By the way, with the recent development of deep learning technology, it has become possible to estimate the skeletal position of a person from a normal still image that does not include depth information, which was difficult in the past, without using a depth sensor. By using this deep learning technology, it is now possible to extract still images from regular camera footage and automatically measure the athlete's posture contained in those still images.

Japanese Patent Application Publication No. 2002-8063 Japanese Patent Application Publication No. 2002-253718 Japanese Patent Application Publication No. 2018-187383

As mentioned above, in order to realize video viewing of video content using the sense of touch, the movement of a human object is extracted from the video content, and tactile information corresponding to the extracted movement of the human object is generated as haptic metadata. It becomes necessary.

However, with the conventional technology, it is difficult to generate haptic metadata only from video analysis of video content in real time. That is, when generating tactile metadata only from video, it is necessary to analyze the movement of a human object in real time from the camera video. In real-time sports competitions, it is undesirable to affect the competition, so instead of using a motion capture method using markers or a depth sensor with limited shooting distance, we use general-purpose broadcasting with no restrictions on shooting conditions. It is desirable to generate haptic metadata only from camera images.

In other words, a technique is desired that automatically and in real time generates tactile metadata regarding the movements of human objects (players, etc.) only from ordinary camera images taken of sports.

Furthermore, with the recent development of deep learning technology, it has become possible to estimate a person's skeletal position from a normal still image that does not include depth information without using a depth sensor, which was difficult in the past. The skeletal detection algorithm typified by this basically detects the skeletal position on a still image basis. Therefore, further ingenuity is required to automatically and in real time generate tactile metadata regarding the movements of human objects (athletes, etc.) only from ordinary camera images taken of sports.

In view of the above-mentioned problems, an object of the present invention is to provide a tactile metadata generation device that automatically extracts a person object from a video, synchronizes and automatically generates tactile metadata corresponding to a dynamic person object, An object of the present invention is to provide a video-tactile interlocking system and a program that drive and control a tactile presentation device based on data.

The tactile metadata generation device of the present invention is a tactile metadata generation device that extracts a person object from a video and generates tactile metadata corresponding to the dynamic person object. a plurality of frame extraction means for extracting a plurality of past frame images including the current frame image, and a first skeletal coordinate of each human object based on a skeletal detection algorithm for each of the plurality of frame images including the current frame image. A human skeleton extracting means that generates a set, and for each of a plurality of frame images including the current frame image, the position and size of the skeleton of each human object, and its surroundings, based on the first skeleton coordinate set. person identification means for identifying a person object by extracting image information and generating a second set of skeletal coordinates to which a person ID is assigned; a trajectory feature generation means for connecting two skeletal coordinate sets in time series to generate a skeletal trajectory set as a set of trajectory features indicating the trajectory of the skeleton of each human object; a human motion recognition means for detecting information for activating the tactile presentation device by machine learning; and a tactile metadata for activating the tactile presentation device corresponding to the current frame image, and generating haptic metadata for activating the tactile presentation device on a frame-by-frame basis. The present invention is characterized by comprising a metadata generating means for externally outputting.

Further, in the tactile metadata generation device of the present invention, a moving object is detected based on a difference image between adjacent frames using each of a plurality of frame images including the current frame image, and is detected from each difference image. A specific moving object is selected using the set of skeletal trajectories of all human objects among the moving objects, and the moving object is connected using the coordinate position, size, and movement direction of the specific moving object obtained from each difference image as elements. The human motion recognition means further includes a moving object detection means for generating information, and the human motion recognition means selects a set of skeletal trajectories for activating the tactile presentation device from among the set of skeletal trajectories of all human objects, based on the moving object information. The present invention is characterized in that information for operating the tactile presentation device is detected by machine learning based on the trajectory features of the selected set of skeletal trajectories.

Further, in the tactile metadata generation device of the present invention, the human motion recognition means may identify, position coordinates, and classify each human object in the current frame image, and the machine The present invention is characterized in that information indicating the timing and speed at which the tactile presentation device is actuated is detected by learning in order to generate the tactile metadata.

Furthermore, the video-tactile interlocking system of the present invention is based on the tactile metadata generation device of the present invention, a tactile presentation device that presents tactile stimulation, and tactile metadata obtained from the tactile metadata generation device. The present invention is characterized by comprising a control unit that refers to drive reference data and controls the tactile presentation device to be driven.

Furthermore, the program of the present invention is configured as a program for causing a computer to function as the haptic metadata generation device of the present invention.

According to the present invention, a person object can be automatically extracted from a video, and tactile metadata corresponding to a dynamic person object can be automatically generated in a synchronized manner. This makes it possible to present tactile stimulation when viewing sports videos in real time. In other words, by appealing not only to the visual and auditory senses but also to the sense of touch, it is possible to provide a sense of presence and immersion that cannot be conveyed through conventional video viewing. Furthermore, it will be possible to convey sports conditions in an easy-to-understand manner to people with visual impairments.

In particular, when viewing sports videos, tactile presentation is possible by generating tactile metadata that includes the identification, position coordinates, and classification (team classification) of each player, as well as information indicating the timing and speed at which the tactile presentation device is activated. The device allows the user to be presented with tactile stimuli related to play type, timing, intensity, etc. This will lead to improvements in services such as public viewing, entertainment, and future tactile broadcasting using tactile information. In addition to sports, it will also be possible to apply this technology to a variety of other uses, such as tactile alarms in factories and security systems based on video analysis from surveillance cameras.

1 is a block diagram showing a schematic configuration of a video-tactile interlocking system including a tactile metadata generation device according to an embodiment of the present invention; FIG. 2 is a flowchart illustrating a processing example of a haptic metadata generation device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram regarding human skeleton extraction processing in the tactile metadata generation device according to one embodiment of the present invention. (a) is a diagram illustrating one frame image, and (b) is a diagram illustrating an example of extracting a human skeleton from one frame image in the haptic metadata generation device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of trajectory feature amounts in the haptic metadata generation device according to one embodiment of the present invention. FIG. 3 is a diagram showing an example of a difference image generated for detecting a moving object in a haptic metadata generation device according to an embodiment of the present invention. FIG. 1 is a block diagram showing a schematic configuration of a control unit in an image-tactile interlocking system according to an embodiment of the present invention.

(System configuration)
DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a video-tactile link system 1 including a tactile metadata generation device 12 according to an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a video-tactile interlocking system 1 including a tactile metadata generation device 12 according to an embodiment of the present invention.

The video-tactile interlocking system 1 shown in FIG. 1 inputs video from a video output device 10 such as a camera or a recording device, automatically extracts human objects from the input video, and extracts tactile metadata corresponding to dynamic human objects. Based on the tactile metadata generation device 12 that synchronizes and automatically generates tactile metadata, and the tactile metadata generated, in this example, two tactile presentation devices 14L, 14R and each tactile presentation device 14L, 14R are individually driven and controlled. A control unit 13 is provided.

First, in this example, it is assumed that the video output by the video output device 10 is displayed on the display 11 as a doubles badminton match between teams A and B taken in real time, and is viewed by the user U.

Badminton is a game in which players are divided into their own team and opponent's team with a net in between, and each team hits a shuttlecock with a racket.By presenting tactile stimulation to the user U using the tactile presentation devices 14L and 14R at the moment the shuttlecock is hit with the racket, the game becomes more realistic. It also makes it possible to convey the game situation to visually impaired people.

Therefore, it is assumed that the user U holds the tactile presentation device 14L with the left hand HL and the tactile presentation device 14R with the right hand HR, and in this example, vibration stimulation synchronized with video analysis is presented. Note that the control unit 13 may be configured to drive and control only one tactile presentation device, or may be configured to individually drive and control three or more tactile presentation devices. Further, although not limited to this, the control unit 13 of this example uses the tactile presentation device 14L to generate the vibration stimulation corresponding to the movement of the human object of team A, and uses the tactile sense presentation device 14L to generate the vibration stimulation corresponding to the movement of the human object of team B. It is assumed that the presentation device 14R classifies and controls the presentation.

In the tactile presentation devices 14L and 14R, a vibration actuator 142 capable of presenting vibration stimulation under the control of the control unit 13 is housed in a spherical case 141. Note that the tactile presentation devices 14L and 14R may be devices that provide electromagnetic pulse stimulation in addition to vibration stimulation. In this example, the control unit 13 and each tactile presentation device 14L, 14R are connected by wire, and the tactile metadata generation device 12 and the control unit 13 are also connected by wire. They may each be wirelessly connected via short-range wireless communication.

The tactile metadata generation device 12 includes a multiple frame extraction unit 121 , a human skeleton extraction unit 122 , a person identification unit 123 , a trajectory feature generation unit 124 , a moving object detection unit 125 , a human motion recognition unit 126 , and a metadata generation unit 127 Equipped with

The multiple frame extraction unit 121 extracts T (T is an integer of 2 or more) past frame images including the current frame image from the input video, and sends them to the human skeleton extraction unit 122 and the moving object detection unit 125. Output.

The human skeleton extraction unit 122 calculates a skeleton coordinate set P ⁿ _b ( n: detected number of people, b: skeleton ID) and outputs it to the person identification unit 123 together with T frames of frame images including the current frame image.

The person identification unit 123 extracts the position and size of each person's skeleton and surrounding image information for each of T frames of frame images including the current frame image, based on the skeleton coordinate set P ⁿ _b . A person is identified, and a set of skeletal coordinates P ⁱ _b (i: person ID, b: skeletal ID) to which a person ID is assigned is generated and output to the trajectory feature generation unit 124 .

The trajectory feature generation unit 124 connects the skeletal coordinate set P ⁱ _b in T frames of frame images in time series using the current frame image as a reference, and generates a skeleton as a set of trajectory features indicating the skeletal trajectory of each person. A trajectory set T ⁱ _b (i: person ID, b: skeleton ID) is generated and output to the moving object detection section 125 and the human motion recognition section 126.

The moving object detection unit 125 detects moving objects based on difference images between adjacent frames using each of the T frame images including the current frame image, and detects trajectory features of the moving objects detected from each difference image. A specific moving object is selected using the set of skeletal trajectories T ⁱ _b of all the people obtained from the quantity generation unit 124, and the coordinate position, size, and movement direction of the specific moving object obtained from each difference image are used as elements. Connected moving object information is generated and output to the human motion recognition unit 126.

The human motion recognition unit 126 selects a skeletal trajectory set T ⁱ _b for operating the tactile presentation device from among the skeletal trajectory set T ⁱ _b of all the people based on the moving object information, and selects the selected skeletal trajectory set T i b. Based on the trajectory feature amount of T ⁱ _b , information indicating the timing and speed of operating the tactile presentation device is detected by machine learning (support vector machine, neural network, etc.) and output to the metadata generation unit 127.

The metadata generation unit 127 generates, corresponding to the current frame image, information indicating the identification, position coordinates, and classification (team classification) of each person in the current frame image, as well as the timing and speed at which the tactile presentation device is activated. tactile metadata including tactile metadata is generated and output to the control unit 13 in units of frames.

Hereinafter, the haptic metadata generation process in the haptic metadata generation device 12 will be described in more detail based on FIG. 2 and with reference to FIGS. 3 to 6.

(Tactile metadata generation processing)
FIG. 2 is a flowchart showing a processing example of the haptic metadata generation device 12 according to an embodiment of the present invention. FIG. 3 is an explanatory diagram regarding human skeleton extraction processing in the haptic metadata generation device 12. Further, FIG. 4(a) is a diagram illustrating one frame image, and FIG. 4(b) is a diagram illustrating an example of human skeleton extraction from one frame image in the haptic metadata generation device 12. FIG. 5 is an explanatory diagram of trajectory feature amounts in the haptic metadata generation device 12. FIG. 6 is a diagram showing an example of a difference image generated for detecting a moving object by the haptic metadata generation device 12.

As shown in FIG. 2, the haptic metadata generation device 12 first extracts T (T is an integer of 2 or more) past frames including the current frame image from the input video using the multiple frame extraction unit 121. Extract an image (step S1).

Next, in the tactile metadata generation device 12, the human skeleton extraction unit 122 calculates a skeleton coordinate set P ⁿ _b (n : detected number of people, b: skeleton ID) (step S2).

Recent developments in deep learning technology have made it possible to estimate the skeletal position of a person from ordinary images. There are also open source skeleton detection algorithms, such as OpenPose and VisionPose (NextSystem). Therefore, the human skeleton extraction unit 122 of this example uses VisionPose to detect 30 human skeleton points for each frame image, as shown in FIG. 3, and generates a skeleton coordinate set P ⁿ _b indicating the position coordinates. do.

In VisionPose, in FIG. 3, P ⁿ ₁ : “Head”, P ⁿ ₂ : “Nose”, P ⁿ ₃ : “Left eye”, P ⁿ ₄ : “Right eye”, P ⁿ ₅ : “Left ear”, P ⁿ ₆ : “Right ear”, P ⁿ ₇ : “Neck”, P ⁿ ₈ : “Spine (shoulder)”, P ⁿ ₉ : “Left shoulder”, P ⁿ ₁₀ : “Right shoulder”, P ⁿ ₁₁ : “Left elbow ”, P ⁿ ₁₂ : “Right elbow”, P ⁿ ₁₃ : “Left wrist”, P ⁿ ₁₄ : “Right wrist”, P ⁿ ₁₅ : “Left hand”, P ⁿ ₁₆ : “Right hand”, P ⁿ ₁₇ : “ P ⁿ ₁₈ : “Right thumb”, P ⁿ ₁₉ : “Left finger tip”, P ⁿ ₂₀ : “Right finger tip”, P ⁿ ₂₁ : “Spine (center)”, P ⁿ ₂₂ : “Spine (base)”. P ⁿ ₂₃ : “Left buttock”, P ⁿ ₂₄ : “Right butt”, P ⁿ ₂₅ : “Left knee”, P ⁿ ₂₆ : “Right knee”, P ⁿ ₂₇ : “Left ankle” ”, P ⁿ ₂₈ : “Right ankle”, P ⁿ ₂₉ : “Left foot”, and P ⁿ ₃₀ : “Right foot”, and each coordinate position can be connected with a line as shown in the diagram. It is.

Based on this VisionPose skeleton detection algorithm, a frame image Fa obtained by extracting human skeletons from one frame image F of a doubles badminton competition between teams A and B shown in Figure 4(a) is created as shown in Figure 4(b). It is shown in Frame image F shown in FIG. 4(a) includes person objects Op1 and Op2 of team A, person objects Op3 and Op4 of team B, and non-person moving objects Om1, Om2, Om3, Om4, and Om5 (such as rackets and shuttle) is in the image, but when VisionPose's skeleton detection algorithm is applied, skeleton coordinate sets P ¹ _b and P 2 corresponding to human objects Op1, Op2, Op3, and ^OP4 , respectively, are obtained as shown in FIG. 4(b). _b , P ³ _b , and P ⁴ _b can be estimated and generated. As can be seen from FIG. 4(b), even in a badminton game that involves violent movements, the skeleton of each person can be estimated with relative accuracy. Incidentally, since the skeleton detection algorithm is limited to estimation in units of still images, the tactile metadata generation device 12 identifies the person as a subsequent process and quantifies the transition of the skeleton position of each person as a trajectory feature. , performs highly accurate motion recognition that takes the time axis into consideration.

Next, the tactile metadata generation device 12 uses the person identification unit 123 to determine the position and size of the skeleton of each person based on the skeleton coordinate set P ⁿ _b for each of the T frames of frame images including the current frame image. The person is identified by extracting the surrounding image information, and a skeletal coordinate set P ⁱ _b (i: person ID, b: skeletal ID) to which a person ID is assigned is generated (step S3).

The human skeleton extraction unit 122 described above can detect one or more human skeletons as a skeleton coordinate set P ⁿ _b for each of T frames of frame images including the current frame image. However, since there is no "who" information in the skeletal coordinate set P ⁿ _b of each frame image, it is necessary to identify the skeletal structure of each person. For this identification, image information near the coordinates of each skeletal coordinate set P ⁿ _b in each frame image is used. That is, the person identification unit 123 extracts the position and size of each person's skeleton and surrounding image information (color information and texture information near the face or back) based on the skeleton coordinate set P ⁿ _b . , a person is identified and a skeletal coordinate set P ⁱ _b (i: person ID, b: skeleton ID) to which a person ID is assigned is generated.

For example, in a badminton game, when the court is photographed at an angle of view with the court held vertically, if the position of the skeleton of each skeleton coordinate set P ⁿ _b is at the top of the screen in frame image F, then the player is in the back, and the player is at the bottom of the screen. If so, it can be identified as the player in the foreground. In addition, in judo, matches are held in white and blue uniforms, and by referring to the color information in the frame image F as image information near the position of the skeleton in each skeleton coordinate set P ⁿ _b , the athlete can be identified. It becomes possible.

Note that the human skeleton extraction unit 122 described above often detects the skeletons of other people other than athletes, such as referees and spectators, who are not the targets of tactile stimulation. Referees often wear different clothing from the players, so they can be identified by color information. Additionally, since spectators are often farther away than the athletes, they can be identified by the size of their skeletons. In this way, by taking into consideration the rules of each competition and the shooting conditions, and setting appropriate surrounding image information (color information and texture information near the face or back) for person identification, we can determine which athletes are the targets of tactile stimulation. identification becomes possible.

Next, the tactile metadata generation device 12 uses the trajectory feature generation unit 124 to connect the skeletal coordinate set P ⁱ _b in the frame images for T frames in chronological order based on the current frame image, and generates a skeletal structure for each person. A skeleton trajectory set T ⁱ _b (i: person ID, b: skeleton ID) is generated as a collection of trajectory features indicating the trajectory of (step S4).

Here, in generating the skeletal trajectory set T ⁱ _b , first, let the skeletal coordinate set P ⁱ _b in an arbitrary frame image be P ⁱ _b (t), set the current frame image as t=0, and calculate the skeletal coordinate set in the current frame image. P ⁱ _b is represented by P ⁱ _b (0), and the skeleton coordinate set P ⁱ _b in the frame images of the past T frames is represented by P ⁱ _b (T). In other words, if the frame number of the current frame image is t=0 and the frame numbers up to the past T frames are expressed as t=T, then the trajectory feature generation unit 124 generates t=0, 1, ..., T can be used to generate a skeletal trajectory set T ⁱ _b as a set of trajectory features indicating the skeletal trajectory of each person.

It should be noted that the skeleton coordinates used to generate the skeleton trajectory set T ⁱ _b do not necessarily need to use all 30 points shown in FIG. You can also. In addition, the skeletal trajectory set T ⁱ _b may be a concatenation of the coordinate representations of the skeletal coordinate set P ⁱ _b , but since it is sufficient as long as it shows the skeletal trajectory of each person, the rules for each competition The information may be converted into information (amount of movement, movement acceleration, etc.) appropriate for representing the trajectory characteristics, taking into consideration the shooting conditions and shooting conditions.

For example, in badminton, the movement acceleration of the arms and body increases when swinging a racket, so it is preferable to create a second-order differential of the movement amount of each skeleton and convert it into a trajectory feature corresponding to the acceleration. Therefore, by representing the trajectory of the skeletal coordinate set P ⁱ _b as a skeletal trajectory set T ⁱ _b corresponding to acceleration, it is possible to improve the accuracy of motion recognition in the human motion recognition unit 126 in the subsequent stage.

First, as shown in equation (1), the difference in position coordinates (Euclidean distance) of the corresponding skeletal coordinate sets P ⁱ _b (t) and P ⁱ _b (t+1) between adjacent image frames is calculated, and the movement Find the quantity D ⁱ _b (t).

Here, P ⁱ _b (t),x represents the x coordinate in P ⁱ _b (t), and P ⁱ _b (t), y represents the y coordinate in P ⁱ _b (t).

D ⁱ _b (t) is the feature amount corresponding to the velocity of each coordinate point, but as shown in equation (2), by further taking the absolute value of the difference, the feature amount A ⁱ corresponding to the acceleration _b (t) is obtained. Here, abs() is a function that returns an absolute value.

Using the feature amount A ⁱ _b (t) corresponding to this acceleration, it is possible to generate a skeleton trajectory set T ⁱ _b indicating the trajectory of each person's motion. A frame image Fb is shown in which trajectory feature amounts T ¹ _b and T ³ _b of skeletal coordinate sets corresponding to objects Op1 and Op3 are drawn in an easy-to-understand manner.

Next, the haptic metadata generation device 12 uses the moving object detection unit 125 to detect moving objects based on the difference images between adjacent frames using each of the T frame images including the current frame image, and detects each moving object based on the difference image between adjacent frames. Among the moving objects detected from the difference images, a specific moving object is selected using the set of skeletal trajectories T ⁱ _b of all the people obtained from the trajectory feature generation unit 124, and the specific moving object obtained from each difference image is Moving object information is generated by linking the coordinate position, size, and movement direction as elements (step S5).

The human motion recognition unit 126 in the latter stage can recognize human motion using the skeletal trajectory set T ⁱ _b , but the motions of people (athletes) vary widely, and false detections or omissions may occur. There are many cases where this is the case. Therefore, the moving object detection unit 125 uses each of T frames of frame images including the current frame image to extract information regarding the position and movement of a moving object such as a racket or a shuttlecock in addition to the movement of a person. By using this information, the subsequent human motion recognition unit 126 can further improve the accuracy of motion recognition.

In the case of badminton, for example, by considering the position and direction of movement of the shuttlecock, it is possible to concentrate on analyzing only the player on the team who will hit the shuttlecock next, and it is possible to suppress false detections and omissions. Therefore, in tracking the movement of the shuttle, the moving object detection unit 125 first extracts only the region of the moving object from the frame image by creating a difference image Fc between adjacent frames as shown in FIG.

As shown in the difference image Fc shown in FIG. 6, it can be seen that human objects Op1' and Op3' and non-human moving objects (racquets or shuttles) Om1', Om3', and Om5' have been detected. In this way, moving objects other than the shuttle are detected in the differential image Fc, but the human (athlete) region is masked on the differential image Fc using the skeleton trajectory set T ⁱ _b obtained from the trajectory feature generation unit 124. Shuttle detection can be performed by That is, using the skeletal trajectory set T ⁱ _b obtained from the trajectory feature generation unit 124, it is possible to create a difference image in which noise in the person (player) region is excluded from the difference image Fc, and the noise related to the movement of the person can be created. Moving objects (especially shuttles) can be detected stably.

More specifically, the moving object detection unit 125 performs labeling processing on the created difference image Fc to give IDs to moving objects other than people, and generates a skeleton trajectory set T ⁱ _b obtained from the trajectory feature generation unit 124. From each difference image, the object most likely to be a shuttle is selected from the color information, size, and motion information of each moving object. Next, the moving object detection unit 125 generates moving object information in which N frames are connected, using the coordinate position, size, and movement direction of a specific moving object (shuttle) obtained from the difference image Fc between adjacent frames as elements. generate.

Next, the tactile metadata generation device 12 uses the human motion recognition unit 126 to generate a skeletal trajectory set T for operating the tactile presentation device from among the skeletal trajectory set T ⁱ _b of all the people based on the moving object information. ^ib is selected, and the timing and speed at which the tactile presentation devices 14R and 14L are activated is indicated by machine learning (support vector machine, neural network, etc.) based on the trajectory feature amount of the selected ^skeleton _trajectory set _Tib . Information is detected (step S6).

When performing machine learning (support vector machine, neural network, etc.), trajectory features for learning are created in advance and learned. For example, when using a support vector machine, the human motion recognition unit 126 learns the trajectory feature at the moment when the shuttlecock is hit as a positive example and the other trajectory features as a negative example. It becomes possible to detect information indicating the timing and speed at which 14L is actuated as motion recognition. Furthermore, the human motion recognition unit 126 improves the accuracy of motion recognition by considering the position coordinates of the shuttlecock from the selected set of skeletal trajectories T ⁱ _b , and also identifies information such as which player hit the shuttlecock in the current frame image. It is also possible to detect information on each person's identification, location coordinates, and classification (team classification).

Finally, the haptic metadata generation device 12 uses the metadata generation unit 127 to identify, position coordinates, and classify (team classification) each person in the current frame image, and to identify the tactile sense Haptic metadata including information indicating the timing and speed at which the presentation device is actuated is generated and output to the control unit 13 in frame units (step S7).

Then, the haptic metadata generation device 12 repeats the processing of steps S1 to S7 every time a video frame image is input from the video output device 10.

(Controller unit)
FIG. 7 is a block diagram showing a schematic configuration of the control unit 13 in the video-tactile interlocking system 1 according to one embodiment of the present invention. The control unit 13 includes a metadata receiving section 131, an analyzing section 132, a storage section 133, and driving sections 134-1 and 134-2.

The metadata receiving unit 131 is a functional unit that inputs haptic metadata from the haptic metadata generating device 12 and outputs it to the analysis unit 132. The haptic metadata includes information indicating the identity, location coordinates, and classification (team classification) of each person in the current frame image, as well as the timing and speed at which the tactile presentation device is activated.

The analysis unit 132 refers to predetermined driving reference data based on the haptic metadata obtained from the haptic metadata generation device 12, and generates each corresponding haptic presentation device via the driving units 134-1 and 134-2. This is a functional unit that controls the vibration actuators 142 of 14L and 14R to be driven. For example, when one of Team A hits a shuttlecock, the analysis unit 132 determines the person's identification, position coordinates, and classification (team classification) in the haptic metadata, as well as the timing and speed at which the haptic presentation device is activated. Then, with reference to predetermined drive reference data, the operation timing, strength, and operation time of the vibration actuator 142 of the tactile presentation device 14L are determined and the drive is controlled.

The storage unit 133 stores predetermined drive reference data for controlling the drive of the drive units 134-1 and 134-2 based on tactile metadata. The drive reference data is expressed in a predetermined table or function regarding the operation timing, strength, and operation time of the vibration actuator 142 as a tactile stimulus associated with the tactile metadata. Furthermore, the storage unit 133 stores programs for realizing the functions of the control unit 13. That is, the functions of the control unit 13 are realized by reading and executing the program by the computer forming the control unit 13.

The drive units 134-1 and 134-2 are drivers that drive the vibration actuators 142 of the respective tactile presentation devices 14L and 14R.

As described above, according to the video-tactile interlocking system 1 including the tactile metadata generation device 12 of the present embodiment, a human object is automatically extracted from a video, and the tactile metadata corresponding to the dynamic human object is synchronized and automatically generated. Since it can be generated, it becomes possible to link the tactile presentation device and the video. In particular, when viewing sports videos, by generating haptic metadata that includes the identification, position coordinates, and classification (team classification) of each player, as well as information indicating the timing and speed at which the haptic presentation device is activated, one The above-described tactile presentation device allows tactile stimulation regarding the type, timing, intensity, etc. of play to be presented to the user U.

(Experimental verification)
Table 1 shows experimental results regarding detection of shot timing extracted from haptic metadata generated by the haptic metadata generation device 12 of this embodiment. Table 1 shows the experimental results when evaluating 20 shots of the Rio de Janeiro Olympics video and measuring shot timing for each athlete (±5 frames allowed). Precision rate indicates detection accuracy and reproduction rate. Rate indicates detection sensitivity. The F value is a value obtained by taking the harmonic mean of precision and recall.

In Table 1, we were able to confirm a certain degree of effectiveness of over 70% for real-time extraction of haptic metadata.

The haptic metadata generation device 12 of the embodiment described above can function as a computer, and a program for causing the computer to implement each component according to the present invention may be provided inside or outside the computer. stored in memory. The object tracking device 1 of the present embodiment reads a program in which processing contents for realizing the functions of each component are appropriately read from memory under the control of a central processing unit (CPU) or the like included in a computer. The functions of each component can be realized by a computer. Here, the functions of each component may be realized by a part of hardware.

Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the technical idea thereof. For example, in the above-described embodiment, video analysis of a badminton game was mainly explained as an example, but it is also widely applicable to judo, table tennis, various other sports events, and videos other than sports. For example, this will lead to improvements in services such as public viewing, entertainment, and future tactile broadcasting using tactile information. In addition to sports, it can also be applied to a variety of uses, such as tactile alarms in factories and security systems based on surveillance camera video analysis. Accordingly, the invention is not limited to the example embodiments described above, but only by the scope of the claims.

According to the present invention, it is possible to automatically extract a person object from a video and synchronize and automatically generate tactile metadata corresponding to a dynamic person object, which is useful for applications in which a tactile presentation device and a video are linked. be.

1 Video-tactile interlocking system 10 Video output device 11 Display 12 Tactile metadata generation device 13 Control unit 14L, 14R Tactile presentation device 121 Multiple frame extraction section 122 Human skeleton extraction section 123 Person identification section 124 Trajectory feature generation section 125 Moving object detection Section 126 Human motion recognition section 127 Metadata generation section 131 Metadata reception section 132 Analysis section 133 Storage section 134-1, 134-2 Drive section 141 Case 142 Vibration actuator

Claims (5)

A tactile metadata generation device that extracts a person object from a video and generates tactile metadata corresponding to the dynamic person object,
a plurality of frame extraction means for extracting a plurality of past frame images including a current frame image from the input video;
human skeleton extraction means for generating a first skeleton coordinate set of each human object based on a skeleton detection algorithm for each of a plurality of frame images including the current frame image;
For each of a plurality of frame images including the current frame image, a person object is extracted by extracting the position and size of the skeleton of each person object and surrounding image information based on the first skeleton coordinate set. person identification means for identifying and generating a second skeletal coordinate set to which a person ID is assigned;
Using the current frame image as a reference, the second skeleton coordinate set in the plurality of frame images is connected in chronological order to generate a skeleton trajectory set as a collection of trajectory features indicating the trajectory of the skeleton of each human object. a trajectory feature generation means for
human motion recognition means for detecting information for operating a tactile presentation device by machine learning based on trajectory features of the skeleton trajectory set;
metadata generation means for generating tactile metadata for operating the tactile presentation device in accordance with the current frame image and outputting it to the outside in frame units;
A tactile metadata generation device comprising: A moving object is detected based on a difference image between adjacent frames using each of a plurality of frame images including the current frame image, and the skeletal trajectories of all human objects among the moving objects detected from each difference image are detected. Further comprising a moving object detection means for selecting a specific moving object using the set and generating linked moving object information using the coordinate position, size, and movement direction of the specific moving object obtained from each difference image as elements,
The human motion recognition means selects a skeletal trajectory set for operating the tactile presentation device from among the skeletal trajectory sets of all human objects based on the moving object information, and determines the trajectory characteristics of the selected skeletal trajectory set. The tactile metadata generation device according to claim 1, wherein information for operating the tactile presentation device is detected by machine learning based on the amount. The human motion recognition means is configured to identify, position coordinates, and classify each human object in the current frame image, and determine the timing and speed at which the tactile presentation device is actuated by the machine learning, in response to the current frame image. The tactile metadata generation device according to claim 1 or 2, wherein information indicating the tactile metadata is detected for generating the tactile metadata. The haptic metadata generation device according to any one of claims 1 to 3;
a tactile presentation device that presents a tactile stimulus;
a control unit that controls the tactile presentation device to be driven by referring to predetermined drive reference data based on the tactile metadata obtained from the tactile metadata generation device;
A video-tactile interlocking system characterized by comprising: A program for causing a computer to function as the haptic metadata generation device according to claim 1.