JP2021197110A - Tactile sense metadata generation device, video tactile sense interlocking system, and program - Google Patents

Abstract

To provide a tactile sense metadata generation device that automatically extracts a dynamic person object from a video and synchronizes and automatically generates corresponding tactile sense metadata, a video tactile sense interlocking system that drives and controls a tactile sense presentation device on the basis of the generated tactile sense metadata, and a program.SOLUTION: A tactile sense metadata generation device 12 includes: a person skeleton extraction section 122 and a person identification section 123 that generate a skeleton coordinate set of each person from an input video and identify a person in a variable search range; a track feature quantity generation section 124 and a motion state measurement section 128 that generate state change amount of person motion and motion state feature quantity on the basis of generation of skeleton track set and its track feature quantity; a person motion recognition section 126 that detects information for a tactile sense presentation device by machine learning on the basis of each feature quantity; and metadata generation sections 127 and 129 that generate various tactile sense metadata. A video tactile sense interlocking system 1 comprises the tactile sense metadata generation device 12 and a control unit 13 for tactile sense presentation device control.SELECTED DRAWING: Figure 1

Description

The present invention is a tactile metadata generator that extracts a person object from an image and generates tactile metadata corresponding to a dynamic person object, and an image tactile interlocking that drives and controls a tactile presentation device based on the generated tactile metadata. Regarding systems and programs.

The video content of a general camera image such as a broadcast image is a medium that provides information that appeals to the two senses of sight and hearing. However, audiovisual information alone is not sufficient for the visually impaired and the hearing impaired, and it is not possible to accurately convey the status of the program content. Therefore, many people with disabilities do not have a TV, or even if they do, they do not watch. Therefore, it is desired to construct a system that allows visually or hearing impaired people to understand television broadcasting by presenting information that can be felt by "tactile sense" other than visual and auditory senses to video contents.

Further, even for a healthy person having a sense of sight and hearing, by presenting a tactile stimulus, it can be expected to improve the sense of presence and immersiveness when watching a broadcast program. In particular, the movement of a person in sports content is important information, and by presenting this with a tactile stimulus, the sense of presence in viewing the content is enhanced.

For example, when watching a baseball video, the batter's hitting sensation can be simulated by stimulating the viewer via a tactile presentation device at the timing when the ball hits the bat. In addition, by providing tactile stimuli to visually impaired people, it is thought that it will lead to understanding the game situation of sports. Thus, tactile sensation is expected as a third sensation in video viewing.

In particular, since real-time video viewing is important in sports, the presentation of tactile stimuli for video needs to be performed automatically and in real time. Therefore, the presentation of tactile stimuli synchronized with the player's movements regarding the type, timing, situation, etc. of play is often effective for viewing video content that also uses tactile sensation. Then, it becomes possible to convey the situation of sports to people with visual or hearing disabilities.

Therefore, in order to realize video viewing of video content that also uses tactile sensation, it is necessary to extract the movement of a person object from the video content and generate tactile information corresponding to the movement of the extracted person object as tactile metadata. You will need it.

However, even if the conventional method of generating tactile metadata realizes video viewing using tactile sensation, tactile metadata indicating at what timing and what kind of stimulus is presented to the user by the tactile presentation device. It was necessary to manually edit the data in a manner synchronized with the video.

In the case of recorded programs, it is possible to manually edit the tactile metadata over time. However, in order to link the stimulus presentation by the tactile presentation device to the live broadcast video, it is not possible to edit the tactile information in advance, so it is necessary to perform video analysis of the video content in real time and generate tactile metadata. Required.

In recent years, sports video analysis technology has achieved remarkable growth. The tennis Hawkeye system, which is also used in Wimbledon, tracks a tennis ball three-dimensionally using a plurality of fixed camera images as sensors, and determines IN / OUT related to the judge. In addition, at the 2014 FIFA World Cup, called goal line technology, images from several fixed cameras are analyzed to automate goal determination. Furthermore, the sophistication of real-time video analysis technology in sports, such as the TRACAB system that sets a large number of stereo cameras in a soccer stadium and tracks all players in the field in real time, is advancing.

On the other hand, in order to measure the posture of a player as a dynamic person object, the measurement using a marker-type motion capture method is generally used. However, this method requires a large number of markers to be attached to the player's body and cannot be applied to actual games. Therefore, in recent years, it has become possible to measure a person's posture without a marker by using a depth sensor that reads an infrared pattern projected on a player's body and obtains depth information from the distortion of the infrared pattern. Further, various techniques applying an optical motion capture method instead of a marker type are disclosed (see, for example, Patent Documents 1 and 2).

For example, in Patent Document 1, when teaching an operation to a user by displaying a model operation image of another person in a virtual reality system using stereoscopic vision, a measurement target person uses an optical motion capture method. A device for measuring the three-dimensional position of the skeleton of the above is disclosed. Further, Patent Document 2 discloses a training evaluation device that measures a player's motion by using an optical motion capture method and evaluates the player's form based on the measured data and data related to the model's form. Has been done. However, since these techniques use a motion capture method, they cannot be applied to an actual game, and it is difficult to measure a person's playing motion from a general-purpose camera image.

Further, regardless of the motion capture method, a device that simulates the movement of a person is disclosed only from a camera image of a badminton game or badminton practice taken by one or two people as a group (for example, a patent). See Document 3). In the technology of Patent Document 3, the motion such as a shot is detected from the captured camera image, but it is premised that the processing is performed from the captured image by the dedicated camera, and the general-purpose broadcast camera image is used. It is difficult to measure a person's playing behavior from.

By the way, with the development of deep learning technology in recent years, it has become possible to estimate the skeleton position of a person from a normal still image that does not include depth information, which was difficult in the past, without using a depth sensor. By using this deep learning technique, it is possible to extract a still image from a normal camera image and automatically measure the posture of the athlete included in the still image.

Japanese Unexamined Patent Publication No. 2002-8063 Japanese Unexamined Patent Publication No. 2002-253718 Japanese Unexamined Patent Publication No. 2018-187383

As described above, in order to realize video viewing of video content using tactile sensation, the movement of a person object is extracted from the video content, and tactile information corresponding to the extracted movement of the person object is generated as tactile metadata. Is needed.

However, with the prior art, it is difficult to generate tactile metadata only from video analysis of video content in real time. That is, when generating tactile metadata only from a video, it is necessary to analyze the movement of a person object in real time from the camera video. In real-time sports competitions, it is not desirable to affect the competition, so general-purpose broadcasting with unlimited shooting conditions without using a motion capture method with markers or a depth sensor with a limited shooting distance. It is desirable to generate tactile metadata only from camera images.

That is, a technique for automatically and in real time generating tactile metadata regarding the movement of a person object (player, etc.) is desired from only a normal camera image for shooting sports.

Also, in order to detect the movement of a person object with high accuracy, a technique of referring to a motion object other than the person (for example, shuttle, racket in the case of badminton competition) can be considered, but the motion object other than the reference person is Even in non-existent competitions (for example, judo and wrestling), a technique for detecting the movement of a person object with high accuracy is desired.

With the recent development of deep learning technology, it has become possible to estimate the skeleton position of a person from a normal still image that does not include depth information, which was difficult in the past, without using a depth sensor. The skeleton detection algorithm represented by this basically detects the skeleton position in units of still images. For this reason, further ingenuity is required to automatically and in real time generate tactile metadata related to the movement of a person object (player, etc.) from only a normal camera image for shooting sports.

In view of the above problems, an object of the present invention is a tactile metadata generator that automatically extracts a person object from a video and synchronizes and automatically generates tactile metadata corresponding to a dynamic person object, and a generated tactile meta. It is an object of the present invention to provide a video-tactile interlocking system and a program for driving and controlling a tactile presentation device based on data.

The tactile metadata generation device of the present invention is a tactile metadata generation device that extracts a person object from an image and generates tactile metadata corresponding to a dynamic person object, and is a current frame image of the input image. The first skeletal coordinates of each person object based on the skeletal detection algorithm for each of the multi-frame extraction means for extracting the past frame images for a plurality of frames including the current frame image and the frame images for the plurality of frames including the current frame image. The search range is variably set based on the first skeleton coordinate set for each of the person skeleton extraction means for generating a set and the frame images for a plurality of frames including the current frame image, and the position of the skeleton of each person object is set. A person identification means that identifies a person object by extracting the size and its surrounding image information and generates a second skeleton coordinate set to which a person ID is given, and the plurality of frames based on the current frame image. A locus feature amount generating means that connects the second skeleton coordinate set in the frame image of the above in time series and generates a skeleton locus set as a set of locus features showing the locus of the skeleton for each person object, and the current frame image. The amount of change in the person movement of each person object is calculated from the distance between each person object based on the skeletal trajectory set in the frame image for the multiple frames and the optical flow amount of each joint for each person object. At the same time, the success or failure of skeletal detection, the amount of movement of the center of gravity of the human skeleton, and the amount of change in the aspect ratio of the skeletal circumscribing rectangle representing the search range are calculated, and the feature vector using these as elements is used as the operating status to represent the operating status. The tactile presentation device is operated by machine learning based on the motion status measuring means generated as the feature quantity, the trajectory feature quantity of the skeletal trajectory set, and the motion status feature quantity corresponding to the trajectory feature quantity of the skeletal trajectory set. A first tactile sensation including a person motion recognizing means for detecting information for impact presentation and information for impact presentation for activating the tactile sensation presenting device obtained from the person motion recognizing means corresponding to the current frame image. The amount of change in the person movement of each person object obtained from the operation state measurement means corresponding to the first metadata generation means that generates metadata and externally outputs it in frame units and the current frame image is shown. It is characterized by comprising a second metadata generation means for generating a second tactile data including information for presenting an operation status and externally outputting it in frame units.

Further, in the tactile metadata generation device of the present invention, the person identification means is limited to a person search range that surrounds the entire human skeleton at the maximum as the search range, and a predetermined area of the human skeleton is the attention search range at the minimum. The search range is determined so as to include at least the attention search range based on the state transition estimated value of the skeleton of the person obtained by the state estimation algorithm, and the person object is identified. It is characterized by having a means for performing the processing.

Further, in the tactile metadata generation device of the present invention, a moving object is detected based on the difference image between adjacent frames using each of the frame images for a plurality of frames including the current frame image, and the moving object is detected from each difference image. A specific moving object is selected using the skeleton locus set of all the moving objects among the moving objects, and the moving object is connected with the coordinate position, size, and moving direction of the specific moving object obtained from each difference image as elements. The moving object detecting means for generating information is further provided, and the person motion recognition means is a skeleton locus set for operating the tactile presentation device among the skeleton locus sets of all the person objects based on the moving object information. Based on the locus feature amount of the selected skeletal locus set and the operation status feature amount corresponding to the selected skeletal locus set, the timing and speed for impact presentation for each person object can be determined by machine learning. It is characterized by detecting the indicated information.

Further, the video-tactile interlocking system of the present invention uses the tactile metadata generation device of the present invention, the tactile presentation device for presenting the tactile stimulus, and the first and second tactile metadata obtained from the tactile metadata generation device. Based on this, it is characterized by including a control unit that controls to drive the tactile presentation device by referring to predetermined drive reference data.

Further, the program of the present invention is configured as a program for making the computer function as the tactile metadata generator of the present invention.

According to the present invention, a person object can be automatically extracted from a video, and tactile metadata corresponding to the dynamic person object can be automatically generated in synchronization. Then, not only the first tactile metadata representing the "type and timing of impact" generated in the person object but also the second tactile metadata representing the continuous "operation status" related to the person object can be automatically generated. become. As a result, it is possible to present tactile stimuli during real-time viewing of sports images, and it is also possible to present detailed tactile sensations in match images such as judo where overlapping of human objects and occlusion are likely to occur. In other words, by appealing not only to the sense of sight and hearing but also to the sense of touch, it is possible to convey the situation of sports to people with visual and hearing disabilities in an easy-to-understand manner. Furthermore, even for healthy people who have a sense of sight and hearing, it is possible to provide a sense of presence and immersiveness that cannot be conveyed by conventional video viewing.

In particular, when watching sports videos, the identification of each player, the position coordinates, and the classification if it is a team competition, the type (type of impact strength) and timing (including speed) of operating the tactile presentation device, and the operation. By generating tactile metadata that includes information indicating the situation, the tactile presentation device can present the user with tactile stimuli regarding the type, timing, intensity, etc. of play as well as changes in the situation of movement. This will lead to improvements in services such as public viewing using tactile information, entertainment, and future tactile broadcasting. In addition to sports, it can also be applied to various applications such as tactile alarms in factories and security systems based on surveillance camera image analysis.

It is a block diagram which shows the schematic structure of the image tactile interlocking system provided with the tactile metadata generation apparatus of one Embodiment by this invention. It is a flowchart which shows the processing example of the tactile metadata generation apparatus of one Embodiment by this invention. It is explanatory drawing about the person skeleton extraction processing in the tactile metadata generation apparatus of one Embodiment by this invention. (A) is a diagram illustrating a one-frame image, and (b) is a diagram showing an example of extracting a human skeleton in a one-frame image in the tactile metadata generation device of one embodiment according to the present invention. (A) and (b) are diagrams showing a processing example of a search range of a person object related to a person skeleton extraction process in the tactile metadata generation device of one embodiment according to the present invention, respectively. It is explanatory drawing of the locus feature amount in the tactile metadata generation apparatus of one Embodiment by this invention. It is a figure which shows the example of the difference image generated for the motion object detection in the tactile metadata generation apparatus of one Embodiment by this invention. (A) to (c) are explanatory views of the state change amount of the person movement with respect to the operation state detection processing in the tactile metadata generation apparatus of one Embodiment by this invention, respectively. It is a block diagram which shows the schematic structure of the control unit in the image tactile interlocking system of one Embodiment by this invention.

(System configuration)
Hereinafter, the image tactile interlocking system 1 provided with the tactile metadata generation device 12 according to the embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a video tactile interlocking system 1 including the tactile metadata generation device 12 of the embodiment according to the present invention.

The video tactile interlocking system 1 shown in FIG. 1 inputs video from a video output device 10 such as a camera or a recording device, automatically extracts a person object from the input video, and tactile metadata corresponding to the dynamic person object. Based on the tactile metadata generator 12 that automatically generates (two types of first tactile metadata and the second tactile metadata) and the generated tactile metadata, two tactile presentations are presented in this example. The devices 14L and 14R and a control unit 13 for individually driving and controlling the tactile presentation devices 14L and 14R are provided.

First, it is assumed that the video output by the video output device 10 is displayed on the display 11 as a real-time shot of the judo competition as an example, and is viewed by the user U.

Judo competition is a sport in which two athletes combine to compete for techniques such as "holding down" and "throwing". By presenting it to the user U as a tactile stimulus by 14R, it is possible to further enhance the sense of presence and to inform the audiovisually impaired person of the game situation.

In particular, in judo competitions, there are many overlaps and occlusions between players on the video, so in addition to the timing and speed according to the type of impact that occurs on each player, it is also possible to combine and throw each player's push and pull. By making it possible to continuously present the tactile sensation, it is possible to convey the sense of urgency of the game to the visually and hearing impaired, and to enhance the sense of presence.

Therefore, it is assumed that the user U grips the tactile presentation device 14L with the left hand HL and the tactile presentation device 14R with the right hand HR, and in this example, the vibration stimulus synchronized with the image analysis is presented. The control unit 13 includes first tactile metadata including information for impact presentation indicating timing and speed according to the type of impact generated in each person object Op1 and Op2 obtained from the tactile metadata generation device 12, and each of them. Two tactile presentations associated with each person object Op1 and Op2 based on the second tactile metadata including information for presenting the operation status representing the continuous "operation status" related to the person objects Op1 and Op2. The tactile presentation of the devices 14L and 14R is individually controlled. However, the control unit 13 may be driven and controlled only for one tactile presenting device, or may be individually driven and controlled for three or more tactile presenting devices. Further, although not limited to, in the control unit 13 of this example, the vibration stimulus corresponding to the movement of the person object Op1 (player) in the video is the tactile presentation device 14L, and corresponds to the movement of the person object Op2 (player). The vibration stimulus generated shall be classified and controlled so as to be presented by the tactile presentation device 14R.

In the tactile presentation devices 14L and 14R, a vibration actuator 142 capable of presenting a vibration stimulus under the control of the control unit 13 is housed in a spherical case 141. The tactile presentation devices 14L and 14R may present electromagnetic pulse stimulation in addition to vibration stimulation. In this example, a mode in which the control unit 13 and the tactile presentation devices 14L and 14R are connected by wire and the tactile metadata generation device 12 and the control unit 13 are also connected by wire will be described. Each may be wirelessly connected by short-range wireless communication.

The tactile metadata generation device 12 includes a plurality of frame extraction unit 121, a person skeleton extraction unit 122, a person identification unit 123, a locus feature amount generation unit 124, a motion object detection unit 125, a person motion recognition unit 126, and a first metadata generation unit. It includes a unit 127, an operation status measurement unit 128, and a second metadata generation unit 129.

The multi-frame extraction unit 121 extracts past frame images for T (T is an integer of 2 or more) frames including the current frame image from the input video, and causes the human skeleton extraction unit 122 and the moving object detection unit 125 to extract the past frame images. Output.

The human skeleton extraction unit 122 has a skeleton coordinate set P of each person object (hereinafter, also simply referred to as “person”) Op1 and Op2 based on the skeleton detection algorithm for each of the frame images for T frames including the current frame image. ⁿ _b (n: number of detected persons, b: skeleton ID) is generated and output to the person identification unit 123 together with the frame images for T frames including the current frame image.

The person identification unit 123 variably sets a search range (details will be described later) based ^{on the skeleton coordinate set P n} _b for each of the frame images for T frames including the current frame image, and positions the skeleton of each person. And the size and the surrounding image information are extracted to identify the person, and the skeleton coordinate set ^Pi _b (i: person ID, b: skeleton ID) to which the person ID is given is generated, and the locus feature amount generation unit 124. Output to.

Locus feature amount generating unit 124, based on the current frame image, T frames connected in time series skeleton coordinate set P ⁱ _b in the frame image, skeletons as a set of trajectories feature quantity indicating a trajectory of the skeleton of each person The locus set ^Ti _b (i: person ID, b: skeleton ID) is generated and output to the motion object detection unit 125, the person motion recognition unit 126, and the operation status measurement unit 128.

The moving object detection unit 125 detects a moving object based on the difference image between adjacent frames using each of the frame images for T frames including the current frame image, and among the moving objects detected from each difference image, the locus feature. ^{A specific moving object is selected using the skeleton locus set Ti} _b of all the persons obtained from the quantity generation unit 124, and the coordinate position, size, and moving direction of the specific moving object obtained from each difference image are used as elements. The concatenated moving object information is generated and output to the person motion recognition unit 126.

Activity measurement unit 128, based on the current frame image, the distance between the person object relative to the skeleton trajectory set T ⁱ _b in T frames of frame images, and the optical flow of each joint of each person object The amount of change in the situation of the person's movement of each person object K (t) is calculated and output to the second metadata generation unit 129, and the success or failure of skeleton detection D (t) and the amount of movement of the center of gravity of the person's skeleton (vertical Gv). (T), lateral Gh (t)), and the amount of change S (t) in the aspect ratio of the skeleton circumscribing rectangle representing the search range are calculated, and these K (t), D (t), and Gv (t) are calculated. , Gh (t), the feature vectors to S a (t) component, operating condition characteristic quantity ^M _i b representing the operation status (i: People ID, b: skeletal ID) generated as, the person operation recognition portion 126 Output.

The human motion recognition unit 126 selects a skeletal trajectory set Ti b for operating the tactile presentation device from ^{the skeletal trajectory set Ti} _b of all persons based on the motion object information, ^{and selects the skeletal trajectory set Ti} _b. a feature trajectory of T ⁱ _b, based on the operating state characteristic quantity M ⁱ _b corresponding to the feature trajectory of the selected skeletal trajectory set T ⁱ _b, by machine learning (support vector machine, or neural networks, etc.) , Identification of each person in the current frame image, position coordinates (and, in this example, it is excluded because it is a judo competition, but if it is a team competition, its team classification), and the timing to activate the tactile presentation device and Information for impact presentation indicating the speed is detected and output to the first metadata generation unit 127.

Since the first metadata generation unit 127 corresponds to the current frame image and is obtained from the person motion recognition unit 126 for identification and position coordinates of each person in the current frame image (and in this example, it is a judo competition). Generates first tactile metadata (for impact presentation) that includes information for impact presentation that indicates the timing and speed at which the tactile presentation device is activated, as well as the team classification), which is not covered, if it is a team competition. Then, it is output to the control unit 13 in frame units.

The second metadata generation unit 129 includes information for presenting an operation status indicating the amount of change in the person movement K (t) of each person object obtained from the operation status measurement unit 128 corresponding to the current frame image. The second tactile metadata (for presenting the operating status) is generated and output to the control unit 13 in frame units.

Hereinafter, the tactile metadata generation process in the tactile metadata generation device 12 will be described more specifically with reference to FIGS. 3 to 8 with reference to FIG. 2.

(Tactile metadata generation process)
FIG. 2 is a flowchart showing a processing example of the tactile metadata generation device 12 according to the embodiment of the present invention. FIG. 3 is an explanatory diagram relating to the human skeleton extraction process in the tactile metadata generation device 12. Further, FIG. 4A is a diagram illustrating a one-frame image, and FIG. 4B is a diagram showing an example of extracting a human skeleton in a one-frame image in the tactile metadata generation device 12. 5 (a) and 5 (b) are diagrams showing a processing example of a search range of a human object related to a human skeleton extraction process in the tactile metadata generation device 12 of the embodiment according to the present invention, respectively. FIG. 6 is an explanatory diagram of a locus feature amount in the tactile metadata generation device 12. FIG. 7 is a diagram showing an example of a difference image generated for detecting a moving object in the tactile metadata generation device 12 according to the embodiment of the present invention. 8 (a) to 8 (c) are explanatory views of the amount of change in the state of movement of the person related to the operation state detection process in the tactile metadata generation device 12 of the embodiment according to the present invention, respectively.

As shown in FIG. 2, first, the tactile metadata generation device 12 first receives past frames for T (T is an integer of 2 or more) frames including the current frame image for the video input by the plurality of frame extraction units 121. The image is extracted (step S1).

Subsequently, the tactile metadata generation device 12 uses the person skeleton extraction unit 122 to obtain the skeleton coordinate sets P of each person object Op1 and Op2 based on the skeleton detection algorithm for each of the frame images for T frames including the current frame image. ^{Generate n} _b (n: number of detected people, b: skeleton ID) (step S2).

Recent developments in deep learning technology have made it possible to estimate the skeleton position of a person from ordinary images. As represented by OpenPose and VisionPose (NextSystem), there are some open source skeleton detection algorithms. Therefore, the human skeleton extraction unit 122 of this example detects 30 points of the human skeleton for each frame image using VisionPose, and generates ^{a skeleton coordinate set P n} _{b indicating the position coordinates thereof.} do.

In VisionPose, in FIG. 3, P ⁿ ₁ : "head", P ⁿ ₂ : "nose", P ⁿ ₃ : "left eye", P ⁿ ₄ : "right eye", P ⁿ ₅ : "left ear", P ^n. ₆ : "Right ear", P ⁿ ₇ : "Neck", P ⁿ ₈ : "Spine (shoulder)", P ⁿ ₉ : "Left shoulder", P ⁿ ₁₀ : "Right shoulder", P ⁿ ₁₁ : "Left elbow" , P ⁿ ₁₂ : "Right elbow", P ⁿ ₁₃ : "Left wrist", P ⁿ ₁₄ : "Right wrist", P ⁿ ₁₅ : "Left hand", P ⁿ ₁₆ : "Right hand", P ⁿ ₁₇ : " Left thumb ”, P ⁿ ₁₈ :“ Right thumb ”, P ⁿ ₁₉ :“ Left fingertip ”, P ⁿ ₂₀ :“ Right fingertip ”, P ⁿ ₂₁ :“ Spine (center) ”, P ⁿ ₂₂ :“ Spine (base) End) ”, P ⁿ ₂₃ :“ Left ^{butt ”, P n} ₂₄ :“ Right butt ”, P ⁿ ₂₅ :“ Left knee ”, P ⁿ ₂₆ :“ Right knee ”, P ⁿ ₂₇ :“ Left ankle , P ⁿ ₂₈ : "Right ankle", P ⁿ ₂₉ : "Left foot", and P ⁿ ₃₀ : "Right foot", and each coordinate position can be drawn by connecting them with a line as shown in the figure. Is.

Based on this Vision Pose skeleton detection algorithm, the frame image Fa obtained by extracting the skeleton of a person from the one-frame image F of the judo competition shown in FIG. 4 (a) is shown in FIG. 4 (b). The frame image F shown in FIG. 4A shows that only each person object Op1 and Op2 (players) are reflected, but other person objects such as the referee's moving object are reflected. In another sports competition, a moving object other than a person (such as a racket or shuttle in a badminton competition) or an object such as a spectator (substantially a static object) may be reflected. However, by applying VisionPose's skeleton detection algorithm, it is possible to extract the skeleton extraction of a person only for the person of the player and the referee's person object. In this example, as shown in FIG. 4 (b), the skeleton coordinate sets P ¹ _b and P ² _b corresponding to the person objects Op 1 and Op 2 can be estimated and generated. As can be understood from FIG. 4B, the skeleton of each person can be estimated relatively accurately even in judo competition. Since the skeleton detection algorithm is limited to estimation in units of still images, the tactile metadata generation device 12 identifies a person as a subsequent process, and quantifies the transition of the skeleton position of each person as a locus feature amount. , Performs highly accurate motion recognition considering the time axis.

Subsequently, the tactile metadata generation device 12 variably sets the search range for each of the frame images for T frames including the current frame image by the person identification unit 123 based on the skeleton coordinate set ^Pn _{b, and each person.} The person is identified by extracting the position and size of the skeleton of the skeleton and the peripheral image information thereof, and a skeleton coordinate set ^Pi _b (i: person ID, b: skeleton ID) to which the person ID is given is generated (step S3). ).

The human skeleton extraction unit 122 described above makes it possible to detect the skeleton of one or more people as the ^{skeleton coordinate set P n} _b for each of the frame images for T frames including the current frame image. However, since the information of "who" does not exist in the skeleton coordinate set P ⁿ _{b of} each frame image, it is necessary to identify the skeleton of each person. For this identification, image information near the coordinates of each skeleton coordinate set P ⁿ _{b in each frame image is used.} That is, the person identification unit 123 extracts the position and size of the skeleton of each person and the peripheral image information (color information and texture information near the face or back) based on ^{the skeleton coordinate set P n} _b. , A person is identified, and a skeleton coordinate set ^Pi _b (i: person ID, b: skeleton ID) to which a person ID is given is generated.

For example, in judo, a match is played in white and blue dogi, but by referring to the color information in the frame image F as the image information near the position of the skeleton of ^{each skeleton coordinate set P n} _{b, the player can be identified.} It will be possible. In the badminton competition, when the court is held vertically at an angle of view, if ^{the position of the skeleton of each skeleton coordinate set P n} _b is on the upper side of the screen in the frame image F, the player in the back and the lower side of the screen. If there is, it can be identified as the player in the foreground.

Therefore, the skeleton detection algorithm in the human skeleton extraction unit 122 is limited to estimation in units of still images, but it is possible to recognize a person as a moving object based on ^{the skeleton coordinate set P n} _{b, and the transition of each skeleton position is traced.} By treating it as a feature, highly accurate motion recognition in consideration of the time axis becomes possible.

In addition to the athletes, the above-mentioned human skeleton extraction unit 122 often detects the skeletons of other persons such as referees and spectators who are not the targets of the presentation of tactile stimuli. Referees often wear different clothing than athletes, so they can be identified by color information. Also, since the spectators are often farther than the athletes, they can be identified by the size of the skeleton. In this way, by setting the peripheral image information (color information and texture information near the face or back) appropriate for person identification in consideration of the rules and shooting conditions of each competition, the athlete to be presented with the tactile stimulus. Can be identified.

By the way, the person identification unit 123 of the present embodiment variably sets the ^{search range (person search range R i} and attention search range R ^bi ) for each frame image in order to deal with the overlap and occlusion of each person. ^{For example, when the person skeleton extraction unit 122 extracts each skeleton coordinate set P n} _b (not shown) for the person objects Op1 and Op2 (players) and the person object Op3 (referee) shown in FIG. 5 (a). , the person identification unit 123 can be variably set the person search range R ⁱ and target search range Rb ⁱ in units of frame images. This search range R ⁱ is set for each person ID (i) in FIG. 5A, and has the position coordinates of the person on the image coordinates of the frame image and the size (width and height) of the person. It is represented by an circumscribing rectangle. Also, it represents the area surrounding the waist region of each person _{^{(P n 22, P n 23}} , P n 24) as the attention search range Rb ^i.

More specifically, the person identification unit 123 of the present embodiment, the search range of a person in each frame image, and limited to a person search range R ⁱ that surrounds the entire human skeleton at maximum, predetermined one person backbone with a minimum region performs variable setting by narrowing that defines the target search range Rb ⁱ (the area surrounding the waist region _{^{(P n 22, P n 23}} , P n 24) in this example), the person obtained by the state estimation algorithm backbone based on the state transition estimate to determine the search range to include at least interest search range Rb ^i, performs processing for identifying the person object. This prevents misrecognition of other people and the processing speed even when the movement of each person changes or the size of the person relative to the frame image changes, for example, as shown in FIG. 5 (b). Can also be improved. In particular, it is effective to use the search range in order to accurately identify a player from an image in which the persons to be identified are heavily overlapped and the background is complicated as in judo.

That is, the person identification unit 123 of the present embodiment has a predetermined range (in this example, the waist region) that enables color identification among the ^{skeleton coordinate sets Pn} _b in Op1, Op2, and Op3 of the person objects of each player and the referee. _{^{(P n 22, P n 23}} , P n 24) of the color (blue, white, brown) since predetermining) as the attention search range Rb ^i, skeletal coordinate set ^P _{n b} of the plurality of person detected overlap by searching focuses attention search range Rb ⁱ is the case, we extract and track accurately person in each frame image. Since there is a case of detecting the skeleton other than the analysis target in the background, the backbone of the analyzed person, by determining by applying a person ID (i), skeletal coordinates P ⁱ _b tracked person Can be identified.

The width and shape of the search range (person search range R ⁱ and attention search range R ^bi ) are determined at least based on the state transition estimates of the human skeleton obtained by a state estimation algorithm such as a Kalman filter or a particle filter. ⁽ⁱⁿ this example, the waist region of each person) interest search range Rb i is determined to contain.

Then, the range can be narrowed when the search range (person search range R ⁱ and attention search range R ^bi ) is stably detected, and the range can be expanded when the detection is unstable. For example, each person ID (i) can be expanded. A search range determined based on the state transition estimated value of the skeleton of is set, and if the state transition estimated value is within a predetermined value from the immediately preceding frame, it is made stable, otherwise it is made unstable, or a state estimation algorithm. Based on the state transition estimation value of the skeleton of the person obtained in, the ratio of successful detection is calculated during the time window for T frames, and if the ratio is equal to or more than the predetermined value, it is stabilized and falls below the predetermined value. In this case, the search range can be variably set by making it unstable.

^{Subsequently, the tactile metadata generation device 12 connects the skeleton coordinate set Pi} _b in the frame image for T frames in time series with the current frame image as a reference by the locus feature amount generation unit 124, and the skeleton for each person. ^{A skeleton locus set Ti} _b (i: person ID, b: skeleton ID) is generated as a set of locus feature quantities showing the loci of (step S4).

Here, in generating the skeleton locus set ^Ti _b , first, the skeleton coordinate set ^Pi _b in an arbitrary frame image is set to ^Pi _b (t), the current frame image is set to t = 0, and the skeleton coordinate set in the current frame image is set. P ⁱ _b is represented by P ⁱ _b ^{(0), and the skeleton coordinate set P i} _b in the frame image of the past T frame is represented by P ⁱ _b (T). That is, when the locus feature amount generation unit 124 expresses the frame number up to the past T frame by t = T, where the frame number of the current frame image is t = 0, t = 0,1, with respect to the current frame image. ^{..., Using each frame image F of T, a skeletal locus set Ti} _b can be generated as a set of locus features indicating the locus of the skeleton for each person. Incidentally, skeletal trajectory set T ⁱ _b can calculate the trajectory separately for each person search range R ⁱ and target search range Rb ^i.

It should be noted that the skeleton ^{coordinates used to generate the skeleton locus set Ti} _b do not necessarily have to use all 30 points shown in FIG. 3, and only a predetermined specific skeleton locus is used to improve the processing speed. You can also do it. Further, the skeleton locus set ^Ti _b may be a concatenation of the coordinate representations of the skeleton coordinate set ^Pi _b , but it may be a rule of each competition as long as it shows the locus of the skeleton for each person. It may be converted into information (movement amount, movement acceleration, etc.) appropriate for expressing the locus characteristics in consideration of the shooting conditions and the shooting conditions.

For example, it is preferable that the skeleton locus set ^Ti _b creates a second derivative of the movement amount of each skeleton and converts it into a value corresponding to acceleration. Therefore, by expressing the locus of the skeleton coordinate set ^Pi _b by the skeleton locus set ^Ti _b corresponding to the acceleration, the accuracy of motion recognition in the person motion recognition unit 126 in the subsequent stage can be improved.

First, as shown in equation (1), between adjacent image frames, taking the difference between the position coordinates of the corresponding backbone coordinate set ^{_{P i b (t), P}} i b (t + 1) ( Euclidean distance), the movement Find the quantity D ⁱ _b (t).

_{^{Here, P i b (t),}} x is the x coordinate of ^{_{P i b (t), P}} i b (t), y represents the y coordinate of ^P _i b (t).

D ⁱ _b (t) is a feature amount corresponding to the velocity of each coordinate point, but as shown in the equation (2), by taking the absolute value of the difference, the feature amount A ^{i corresponding to the acceleration.} _b (t) is obtained. Here, abs () is a function that returns an absolute value.

Using the feature amount A ⁱ _b ^{(t) corresponding to this acceleration, it is possible to generate a skeletal locus set Ti} _b showing a locus that traces the movement of each person, and FIG. 6 shows a person in a certain frame image. ^{The frame image Fb in which the locus feature quantities T 1} _b and T ² _b of the skeleton coordinate set corresponding to the objects Op1 and Op2 are drawn in an easy-to-understand manner is shown.

Subsequently, the tactile metadata generation device 12 is subjected to the operation status measurement unit 128, ^{the distance between each person object based on the skeleton locus set Ti} _b in the frame image for T frames, and the distance between each person object based on the current frame image. The amount of change in the person movement of each person object K (t) is calculated from the amount of optical flow of each joint for each person object and output to the second metadata generation unit 129, and the success or failure of skeleton detection D (t). , The amount of movement of the center of gravity of the human skeleton (vertical Gv (t), horizontal Gh (t)), and the amount of change S (t) in the aspect ratio of the skeleton circumscribing rectangle representing the search range are calculated, and these K (t) are calculated. ), D (t), Gv (t), Gh (t), the feature vectors to S a (t) component, operating condition characteristic quantity representing the operating status ^M _i b _(i: People ID, b: skeletal ID ) And output to the person motion recognition unit 126 (step S5).

For example, in judo, it is often the case that the opponent's gap is heard from the state of the kumite and the action of throwing is suddenly started. By presenting the situation from this quiet to sudden as a tactile stimulus, it is possible to enhance the sense of presence in viewing video content. T frame distance between each person object (player) relative to the skeleton trajectory set T ⁱ _b in the frame image, and each each person object from the optical flow of each joint of a person per object (player) of the (player) The amount of change in the situation of the person's movement K (t) can be calculated.

More specifically, the amount of change in the situation of the person's movement K (t) is defined by the following equation (3). The situation change amount K (t) of the movement of the person is externally output as the second second tactile metadata (for presenting the operation status) by the second metadata generation unit 129.

Here, d (t) represents the distance between each person object (player), and F (t) is an optical of 30 points of the skeleton for each person object at the time t of the current frame image and the time t-1 of the immediately preceding frame. Represents the average value of the flow. g in calculating d (t) represents the center of gravity (in this example, the median coordinates of ^{P n} ₂₂ , P ⁿ ₂₃ , and P ⁿ _{24, which are the positions of the waist).} Therefore, P ¹ _g (t) represents the position of the center of gravity of the person object Op1 of the person ID of i = 1 at the time t of the current frame image, and P ² _g (t) represents i = at the time t of the frame image. Represents the position of the center of gravity of the person object Op2 having the person ID of 2. Further, L is the average value of the distances from the necks to the waists of both players out of the 30 skeletons of both person objects Op1 and Op2, and by normalizing with L, the influence of the shooting size of each player can be eliminated.

On the other hand, operation conditions characteristic quantity ^M _i b representing the operation status (i: People ID, b: skeletal ID), respectively formula (4), Equation (5), equation (6), as shown in Equation (7) , Success / failure of skeleton detection D (t), amount of movement of the center of gravity of the human skeleton (vertical Gv (t), horizontal Gh (t)), and amount of change in aspect ratio of the skeleton extrinsic rectangle (person search range) S (t) Is calculated, and in addition to the amount of change in the situation of the person's movement K (t), D (t), Gv (t), Gh (t), and S (t) are generated as elements.

^{Here, R} i (t) represents a skeleton enclosing rectangle representing the search range of a person object Op1 in the frame image F at time t (t), and its upper left coordinates (x, y), width (width) and It has height information, and "aaa.bbb" represents the bbb component of aaa.

During the "throwing" motion of judo, the center of gravity suddenly drops, often resulting in a horizontally long aspect ratio on the image. In addition, they often have an unnatural posture and often fail to detect the skeleton. In addition, the amount of joint movement increases overall. Therefore, in addition to the status change amount K (t) of the person moves, D (t), Gv ( t), Gh (t), the operation conditions characteristic quantity representing the operating status of elements of S (t) ^M _{i b} Is used as a machine learning classifier in the person motion recognition unit 126 in the subsequent stage, for example, a "throwing" motion can be detected with high accuracy.

Subsequently, the tactile metadata generation device 12 detects a moving object based on the difference image between adjacent frames by using each of the frame images for T frames including the current frame image by the moving object detecting unit 125, and each of them. selects a particular dynamic object using the framework trajectory set T ⁱ _b of all persons obtained from the trajectory feature amount generating unit 124 of the moving object detected from the difference image, the particular dynamic object obtained from each difference image The moving object information is generated by connecting the coordinate position, the size, and the moving direction as elements (step S6). As shown in the difference image Fc shown in FIG. 7, it can be seen that the person objects Op1'and Op2' can be detected.

The person motion recognition unit 126 in the subsequent stage can recognize the motion of the person ^{using the skeletal trajectory set Ti} _b , but the motion of the person (player) is diverse, and erroneous detection or omission of detection occurs. There are many cases of doing so. Therefore, the moving object detection unit 125 extracts information on the position and movement of the moving object of the person to be analyzed by using each of the frame images for T frames including the current frame image (note that, in badminton competition, etc.). When a moving object such as a racket or shuttle other than a person can be referred to, that moving object can also be used.) By using this information, the person motion recognition unit 126 in the subsequent stage can further improve the accuracy of motion recognition.

Subsequently, in the tactile metadata generation device 12, the person motion recognition unit 126 operates the skeleton locus set T for operating the tactile ^{locus presentation device among the skeleton locus set Ti} _{b of all the persons based on the moving object information.} selects the ⁱ _b, based on the feature trajectory of selected skeletal trajectory set T ⁱ _b, and the operation conditions characteristic quantity M ⁱ _b corresponding to the feature trajectory of the selected skeletal trajectory set T ⁱ _b, machine learning Information for impact presentation that operates the tactile presentation devices 14R and 14L is detected by (support vector machine, neural network, etc.) (step S7). The information for impact presentation includes identification of each person in the current frame image, position coordinates (and, in this example, it is excluded because it is a judo competition, but if it is a team competition, its team classification), and tactile sensation. It contains information indicating when and how fast the presentation device is activated.

At the time of machine learning (support vector machine, neural network, etc.), a locus feature for learning is created and trained in advance. For example, when using a support vector machine, the human motion recognition unit 126 can use the tactile presentation device 14R, by learning the locus feature amount at the moment representing the impact as a positive example and the other locus feature amount as a negative example. Information indicating the timing and speed at which the 14L is operated can be detected as motion recognition. Furthermore, the person operating the recognition unit 126, from the trajectory characteristic quantity of the selected backbone trajectory set T ⁱ _b, to increase the accuracy of the motion recognition, etc. which players are receiving any impact, each in the current frame image It is also possible to detect person identification, position coordinates, and (and team classification) information.

Furthermore, the addition of operation conditions characteristic quantity M ⁱ _b to the machine learning classifier, it is possible to recognize the impact that occurs in response to movement, such as the throwing operation in judo. Therefore, the person motion recognition unit 126 may refer to the information of the motion object other than the person in the motion recognition, but in this example, even if the motion object other than the person cannot be referred to, the judo competition or the like can be performed with high accuracy. to recognize a person, so that using the operating conditions characteristic quantity M ⁱ _b. The operating condition wherein the amount by adding the M ⁱ _b, thereby improving the determination of the kind of detection accuracy and movement of the motion of the person.

However, the change of the characteristic of the trajectory characteristic amount and operating conditions characteristic quantity M ⁱ _b of these selected skeletal trajectory set T ⁱ _b are not necessarily the total feature value is generated so as to change at the same time. Therefore, a feature amount provided with a time window for a certain period of time such as T = 15 frames is used. By performing machine learning with SVM or the like using each feature amount within a certain period of time, it is possible to construct a robust classifier even if there is a time lag in the change of each feature amount.

Finally, the tactile metadata generation device 12 provides the identification, position coordinates, (and team classification), and tactile presentation device of each person corresponding to the current frame image by the first metadata generation unit 127. First tactile metadata (for impact presentation) including information for impact presentation indicating the timing and speed of operation is generated and output to the control unit 13 in frame units (step S8).

Further, in the tactile metadata generation device 12, the second metadata generation unit 129 includes information for presenting an operation status indicating the amount of change in the status of the movement of the person of each person object corresponding to the current frame image. Tactile metadata (for presenting the operating status) is generated and output to the control unit 13 in frame units (step S9).

Then, the tactile metadata generation device 12 repeats the processes of steps S1 to S9 every time a frame image of a video is input from the video output device 10.

By controlling the tactile presentation devices 14L and 14R in response to the second tactile metadata based on the amount of change in the situation of the person's movement K (t), the control unit 13 tries to throw when the athletes are in close proximity to each other. The corresponding tactile presentation device can be vibrated significantly when the body moves significantly. The control unit 13 only gives a single stimulus representing an impact for the first tactile metadata, but also acquires the second tactile metadata indicating the operating status to control the corresponding tactile presentation device. As a result, the status of the video content can be constantly presented as a continuous value, and the user U can always grasp the operating status of each person object Op1 and Op2.

FIG. 8 shows an example of visualizing the amount of change in the situation of the person's movement K (t). The left figure of FIG. 8A is a frame image F of the current frame in which the state of the operation state “combination” of the person objects Op1 and Op2 (players) and the person object Op3 (referee) can be seen, and is the frame image F of FIG. 8 (a). The figure on the right shows the magnitude of the amount of change in the situation of the person movement K (t) normalized by the maximum value of 100 from the time tp of the past frame image to the time ct of the current frame image (elapsed period of 15 frames). There is. As can be understood from the right figure of FIG. 8 (a), the amount of change in the person movement K (t) increases with the passage of time during the “combination” operation status (the same applies to the throwing operation status). It can be determined that the amount of movement of each person is increasing.

Further, the left figure of FIG. 8B is a frame image F of the current frame in which the operation state “suppression” of the person objects Op1 and Op2 (players) and the person object Op3 (referee) can be seen, and is a frame image F of FIG. 8 (b). The figure on the right of) shows the magnitude of the change in the amount of human movement K (t) normalized by the maximum value of 100 from the time tp of the past frame image to the time ct of the current frame image (elapsed period of 15 frames). Represents. As can be understood from the right figure of FIG. 8 (b), the amount of change in the situation of the person's movement K (t) becomes smaller with the passage of time during the "suppressed" operation state, and the amount of movement of each person decreases. Can be determined.

Further, the left figure of FIG. 8C is a frame image F of the current frame in which the operation status “waiting” of the person objects Op1 and Op2 (players) and the person object Op3 (referee) can be seen, and is a frame image F of FIG. 8 (c). The figure on the right of) shows the magnitude of the change in the amount of human movement K (t) normalized by the maximum value of 100 from the time tp of the past frame image to the time ct of the current frame image (elapsed period of 15 frames). Represents. As can be understood from the right figure of FIG. 8 (c), the amount of change in the movement of the person K (t) during the “wait” operation status (the same applies before the start of the game and when the two players are far apart). ) Is small even after the passage of time, and it can be determined that there is almost no movement amount of each person.

Also, status change amount K of the person moves (t) is an element contained in the operating condition characteristic quantity M ⁱ _b, by using the operating condition characteristic quantity M ⁱ _b, as shown in Equation (4) if- Even if the then rule is used, the movement recognition of each person can be performed with high accuracy. That is, as shown in FIG. 8C, the referee raises his hand and "waits" by independently recognizing the two athletes and the referee and discriminating the transition of each person's joint from the locus feature amount. Can be automatically recognized. The tactile metadata generation device 12 outputs the first and second tactile metadata corresponding to the "wait" operation to the tactile presentation devices 14L and 14R, so that the user U who uses the tactile presentation devices 14L and 14R can use the tactile metadata generation device 12. , It is possible to forcibly show from a continuous fixed amount of stimulus presentation to almost zero stimulus presentation, and there is less risk of misunderstanding the match situation as if it was before the start of the match by the action before and after "wait". Become. Therefore, the tactile metadata generation device 12 of the present embodiment can present the tactile stimulus according to the movement of the person even during real-time viewing of the sports image.

(Controller unit)
FIG. 9 is a block diagram showing a schematic configuration of a control unit 13 in the video-tactile interlocking system 1 of the embodiment according to the present invention. The control unit 13 includes a metadata receiving unit 131, an analysis unit 132, a storage unit 133, and a driving unit 134-1,134-2.

The metadata receiving unit 131 inputs the first tactile metadata (for impact presentation) and the second tactile metadata (for operating status presentation) from the tactile metadata generation device 12, and outputs them to the analysis unit 132. Is. The first tactile metadata includes identification of each person in the current frame image, position coordinates (and their team classification if in a team competition), as well as information indicating when and how fast the tactile presentation device is activated. .. The second tactile metadata includes information on the operating conditions illustrated in FIG.

The analysis unit 132 refers to predetermined drive reference data based on the first and second tactile metadata obtained from the tactile metadata generation device 12, and via the drive unit 134-1, 134-2, the analysis unit 132 refers to the predetermined drive reference data. It is a functional unit that controls to drive the vibration actuator 142 of each of the corresponding tactile presentation devices 14L and 14R. For example, when one player shifts from the union to the throwing motion, the analysis unit 132 identifies a person in the first tactile metadata in addition to the vibration presentation according to the motion state in the second tactile metadata. From the position coordinates, (and team classification), and the timing and speed at which the tactile presentation device is activated, the activation timing, strength, and activation of the vibration actuator 142 of the tactile presentation device 14L are referred to with reference to predetermined drive reference data. The operation time is determined and the drive is controlled.

The storage unit 133 stores predetermined drive reference data for controlling the drive of the drive units 134-1, 134-2 based on the first and second tactile metadata. The drive reference data is represented by a predetermined table or function regarding the operation timing, strength, and operation time of the vibration actuator 142 as the tactile stimulus associated with the first and second tactile metadata. Further, the storage unit 133 stores a program for realizing the function of the control unit 13. That is, the function of the control unit 13 is realized by reading and executing the program by the computer constituting the control unit 13.

The drive units 134-1 and 134-2 are drivers that drive the vibration actuators 142 of the tactile presentation devices 14L and 14R.

As described above, according to the video tactile interlocking system 1 provided with the tactile metadata generation device 12 of the present embodiment, the person object is automatically extracted from the video, and the tactile metadata corresponding to the dynamic person object is automatically synchronized. Since it can be generated, the tactile presentation device and the image can be linked. Then, not only the first tactile metadata representing the "type and timing of impact" generated in the person object but also the second tactile metadata representing the continuous "operation status" related to the person object can be automatically generated. become. As a result, it is possible to present tactile stimuli during real-time viewing of sports images, and it is also possible to present detailed tactile sensations in match images such as judo where overlapping of human objects and occlusion are likely to occur. In other words, by appealing not only to the visual and auditory senses but also to the sense of touch, it is possible to convey the situation of sports to people with visual and auditory disabilities in an easy-to-understand manner. Furthermore, even for healthy people who have a sense of sight and hearing, it is possible to provide a sense of presence and immersiveness that cannot be conveyed by conventional video viewing. In particular, when watching sports footage, it is more accurate to generate tactile metadata that includes information indicating the identification, position coordinates, (and team classification) of each athlete, and the timing and speed at which the tactile presentation device is activated. More specifically, one or more tactile presentation devices can present tactile stimuli to the user U regarding the type, timing, intensity, and the like of play.

The tactile metadata generation device 12 of the above-described embodiment can be made to function as a computer, and the computer is provided with a program for realizing each component according to the present invention inside or outside the computer. It is stored in the memory to be stored. A program in which processing contents for realizing the functions of each component are appropriately read from a memory under the control of a central processing unit (CPU) provided in a computer, and tactile metadata generation of the present embodiment is performed. The function of each component of the device 12 can be realized in the computer. Here, the function of each component may be realized by a part of hardware.

Although the present invention has been described above with reference to examples of specific embodiments, the present invention is not limited to the examples of the above-described embodiments, and can be variously modified without departing from the technical idea. For example, in the example of the above-described embodiment, the video analysis of the badminton competition has been mainly described as an example, but it can be widely applied to judo, table tennis, various other sports events, and non-sports video. For example, it will lead to improvement of services such as public viewing using tactile information, entertainment, and future tactile broadcasting. In addition, as an example other than sports, it can be applied to various applications such as a tactile alarm in a factory and a security system based on surveillance camera image analysis. Therefore, the present invention is not limited to the example of the above-described embodiment, but is limited only by the description of the scope of claims.

According to the present invention, a person object can be automatically extracted from a video, and tactile metadata corresponding to a dynamic person object can be automatically generated in synchronization, which is useful for linking a tactile presentation device and a video. be.

1 Video tactile interlocking system 10 Video output device 11 Display 12 Tactile metadata generation device 13 Control unit 14L, 14R Tactile presentation device 121 Multiple frame extraction unit 122 Human skeleton extraction unit 123 Person identification unit 124 Trajectory feature amount generation unit 125 Dynamic object detection Unit 126 Person motion recognition unit 127 First metadata generation unit 128 Operation status measurement unit 129 Second metadata generation unit 131 Metadata reception unit 132 Analysis unit 133 Storage unit 134-1,134-2 Drive unit 141 Case 142 Vibration actuator

Claims (5)

A tactile metadata generator that extracts a person object from a video and generates tactile metadata corresponding to a dynamic person object.
A multi-frame extraction means for extracting past frame images for multiple frames including the current frame image for the input video, and
A person skeleton extraction means that generates a first skeleton coordinate set of each person object based on a skeleton detection algorithm for each of the frame images for a plurality of frames including the current frame image.
For each of the frame images for a plurality of frames including the current frame image, the search range is variably set based on the first skeleton coordinate set, and the position and size of the skeleton of each person object and its peripheral image information are extracted. A person identification means that identifies a person object and generates a second skeletal coordinate set to which a person ID is given.
Based on the current frame image, the second skeleton coordinate set in the frame images for the plurality of frames is connected in time series, and a skeleton locus set is generated as a set of locus features showing the locus of the skeleton for each person object. Trajectory feature amount generation means and
Based on the current frame image, the distance between each person object based on the skeletal trajectory set in the frame images for the plurality of frames, and the optical flow amount of each joint for each person object, the state of the person movement of each person object. In addition to calculating the amount of change, the success or failure of skeleton detection, the amount of movement of the center of gravity of the human skeleton, and the amount of change in the aspect ratio of the skeleton circumscribing rectangle representing the search range are calculated, and the feature vector using these as elements is used as the operating status. The operation status measuring means generated as the operation status feature quantity representing
Based on the locus feature amount of the skeletal locus set and the motion state feature amount corresponding to the locus feature amount of the skeletal locus set, the human motion that detects the information for impact presentation that operates the tactile presentation device by machine learning. Recognition means and
A first tactile metadata including information for impact presentation that operates the tactile presentation device obtained from the person motion recognition means corresponding to the current frame image is generated and externally output in frame units. Metadata generation means and
Corresponding to the current frame image, second tactile metadata including information for presenting the motion status indicating the amount of change in the status of the person movement of each person object obtained from the motion status measuring means is generated, and is generated in frame units. A second metadata generation means to output externally,
A tactile metadata generator characterized by comprising. The person identification means limits the search range to a person search range that surrounds the entire human skeleton at the maximum, and performs variable setting by narrowing down a predetermined area of the human skeleton as the attention search range at the minimum, and estimates the state. It is characterized by having a means for determining the search range so as to include at least the attention search range based on the state transition estimation value of the skeleton of the person obtained by the algorithm, and performing a process of identifying the person object. , The tactile metadata generator according to claim 1. A moving object is detected based on the difference image between adjacent frames using each of the frame images for a plurality of frames including the current frame image, and the skeleton locus of all the person objects among the moving objects detected from each difference image. It is further equipped with a moving object detection means that selects a specific moving object using a set and generates moving object information concatenated with the coordinate position, size, and moving direction of the specific moving object obtained from each difference image as elements.
The person motion recognition means selects a skeleton locus set for operating the tactile presentation device from the skeleton locus sets of all the person objects based on the motion object information, and the locus features of the selected skeleton locus set. A claim characterized by detecting information indicating the timing and speed for impact presentation for each person object by machine learning based on the amount and the operation state feature amount corresponding to the selected skeletal locus set. Item 2. The tactile metadata generator according to Item 1 or 2. The tactile metadata generator according to any one of claims 1 to 3.
A tactile presentation device that presents tactile stimuli,
A control unit that controls to drive the tactile presentation device by referring to predetermined drive reference data based on the first and second tactile metadata obtained from the tactile metadata generator.
A video-tactile interlocking system characterized by being equipped with. A program for causing a computer to function as the tactile metadata generator according to any one of claims 1 to 3.

Images