JP2023115775A - Tactile sense presentation signal generation device, video tactile sense interlocking system, and program - Google Patents

Abstract

To provide: a tactile sense presentation signal generation device which generates a tactile sense presentation signal for driving a tactile sense presentation device by using an audio signal with respect to a sound signal based on tactile sense metadata automatically generated from only a video signal of video content and synchronizing the audio signal with the video signal; a video tactile sense interlocking system which drives and controls the tactile sense presentation device on the basis of the generated tactile sense presentation signal; and a program.SOLUTION: A tactile sense presentation signal generation device 1 of the present invention includes: a low-frequency audio extraction section 11 which extracts a low-frequency audio signal from an audio signal; a tactile sense metadata generation section 12 which generates tactile sense metadata from video analysis of a video signal; a signal conversion section 13 which converts each tactile sense metadata into a sound signal corresponding to tactile sense stimulation; and a synchronization synthesis section 15 which generates a tactile sense presentation signal obtained by synchronously synthesizing the low-frequency audio signal and the sound signal. A video tactile sense interlocking system 10 of the present invention includes: the tactile sense presentation signal generation device 1; a tactile sense presentation device 5; and a control unit 4 for control the tactile sense presentation device.SELECTED DRAWING: Figure 1

Description

The present invention provides a tactile presentation signal generation apparatus that generates a tactile presentation signal for driving a tactile presentation device in conjunction with video content, a video-tactile interlocking system that drives and controls the tactile presentation device based on the generated tactile presentation signal, and program.

2. Description of the Related Art A video content consisting of a video signal and an audio signal of general camera video such as broadcast video is a medium that provides information that appeals to the two senses of sight and hearing. However, audio-visual information alone is insufficient for visually impaired and hearing impaired people, and it is not possible to accurately convey the situation of program content. Therefore, there are many people with disabilities who either do not own a TV or do not watch it even if they do. Therefore, it is desired to construct a system that allows visually or hearing impaired people to understand television broadcasting by presenting information that can be felt by "tactile sensation" as a third sense other than sight and hearing.

In addition, for healthy people who have visual and auditory senses, the presentation of tactile stimuli can be expected to improve the sense of realism and immersion when viewing broadcast programs. In particular, the movement of a person in sports content is important information, and by presenting this with tactile stimulation, the presence of the content can be enhanced.

For example, when watching a baseball video, the viewer can simulate the sensation of hitting by a batter by stimulating the viewer via a tactile presentation device when the ball hits the bat. It is also thought that providing tactile stimulation to visually impaired people will help them understand the state of a sports match. Thus, tactile sensation is expected as the third sense in video viewing.

In sports, in particular, real-time video viewing is important, so presentation of tactile stimuli to video must be automatic and real-time. Therefore, presenting tactile stimuli synchronized with the player's movements regarding the type of play, timing, situation, etc. is often effective for video viewing of video content using tactile sensations. It is also possible to convey the sports situation to people with visual or hearing impairments.

Therefore, in order to realize video viewing of video content that uses the sense of touch, it is necessary to extract the movement of the human object from the video content and generate haptic information corresponding to the extracted movement of the human object as haptic metadata. become necessary.

However, with the conventional tactile metadata generation method, even if video viewing using tactile sensation is realized, tactile metadata indicating at what timing and what kind of stimulus is to be presented to the user by the tactile presentation device. The data had to be manually edited in a synchronized manner with the video.

For recorded programs, it is possible to manually edit the haptic metadata over time. However, since it is not possible to edit the tactile information in advance in order to link the stimulation presentation by the tactile presentation device to the live broadcast video, it is necessary to analyze the video content in real time and generate the tactile metadata. requested.

In recent years, sports video analysis technology has achieved remarkable growth. The Hawkeye system of tennis, which is also used at Wimbledon, tracks a tennis ball three-dimensionally using multiple fixed camera images as sensors, and judges whether it is in or out in relation to the judges. Also, at the 2014 FIFA World Cup, goal line technology was used to analyze images from several fixed cameras and automate goal determination. In addition, real-time video analysis technology in sports is becoming more sophisticated, such as the TRACAB system, which tracks all players on the field in real time by setting up a large number of stereo cameras in a soccer stadium.

In order to measure the posture of a player as a dynamic human object, conventionally, measurement using a marker-type motion capture method is common. However, this method requires a large number of markers to be attached to the player's body, and cannot be applied to actual games. Therefore, in recent years, markerless human posture measurement has become possible by using a depth sensor that reads the infrared pattern projected onto the player's body and obtains depth information from the distortion of the infrared pattern. Also, various techniques have been disclosed that apply an optical motion capture method instead of the marker method (see, for example, Patent Documents 1, 2, and 3).

For example, in Patent Literature 1, when instructing a user to perform a motion by displaying a model motion image of another person in a virtual reality system using stereoscopic vision, an optical motion capture method is used to teach a person to be measured. Disclosed is an apparatus for measuring the three-dimensional position of the skeleton of a In addition, Patent Document 2 discloses a technique for recognizing actions by using information obtained from images of gymnastics and other motion capture data. It has the feature of being removed. In addition, Patent Document 3 discloses a training evaluation device that measures a player's motion using an optical motion capture method and evaluates the player's form based on the measured data and model form data. It is However, since these techniques use a motion capture method, they cannot be applied to an actual game, and it is difficult to measure a person's playing motion from a general-purpose camera image.

In addition, a device is disclosed that simulates the movement of a person from only camera images of a badminton match or practice of one person or two people in a group without relying on the motion capture method (for example, patent Reference 4). In the technique of Patent Document 4, actions such as shots are detected from captured camera images. It is difficult to measure the playing motion of a person from the

For this reason, without using motion capture with markers or depth sensors with limited shooting distances, video analysis is performed in real time from the video signal of arbitrary video content to determine the type and timing of tactile stimulation in advance. A system that presents prepared tactile stimuli is desirable. In particular, sports broadcasting is content in which real-time performance is emphasized.

Therefore, a method of performing video analysis of arbitrary video content, determining the type and timing of tactile stimulus, and presenting prepared tactile stimulus as tactile metadata is referred to herein as a “tactile metadata method”. I will call.

With the tactile metadata method, for example, tactile information related to the competition can be presented as video content in real time at the same time as the video, editing is easy, and tactile stimulation synchronized with the movements of the athletes involved in the competition can be effectively presented.

In recent years, advances in deep learning technology have made it possible to estimate a person's skeletal position from a normal still image that does not contain depth information, without using a depth sensor, which has been difficult in the past. By using this deep learning technology as a tactile metadata method, it is possible to extract still images from ordinary camera images and automatically measure the player's posture contained in the still images. That is, by measuring the player's posture from a normal camera image, it is possible to acquire information about tactile stimulation without affecting the game.

Although it is possible to measure a person's posture by acquiring skeletal information, recognition processing is required to give meaning to the posture. For example, when judo video is input, it is necessary to determine from the image characteristics and skeletal characteristics whether the action performed in the frame is a "fight", a "throwing technique", or a "submission". Convolutional Neural Network (CNN) is widely used for recognition processing in image processing. A CNN is a neural network with many deep layers, and is a network that exhibits excellent performance especially in the field of image recognition. This network is constructed by piling up layers with several characteristic functions such as "convolution layer" and "pooling layer", and is currently used in a wide range of fields.

In a general neural network, neurons are arranged in layers, and the neurons included in the preceding and succeeding layers are normally connected exhaustively. By using a technique called weight sharing, it expresses processing equivalent to image convolution within the framework of a neural network. This layer is called a “convolutional layer” and is the most important feature of CNN. Another major feature of this convolutional neural network is the “pooling layer”. In CNN, if the “convolutional layer” plays the role of feature extraction such as edge extraction from the image, the “pooling layer” is robust so that such extracted features are not affected by translation etc. giving.

On the other hand, besides using skeleton information, an image called Motion History Image (MHI) has conventionally been used as a technique for recognizing motion from an image (see, for example, Non-Patent Document 1 and Patent Document 5). MHI is an image in which the area where the luminance difference occurs in each frame is filled with high luminance, and the subsequent frames are drawn with the luminance gradually lowered. It's becoming

Patent Literature 5 discloses a technique for detecting a pitching motion from a baseball video using image recognition technology, in which a Motion History Image (MHI) is created for the baseball video and the pitching motion is detected. there is By analyzing this MHI, it becomes possible to determine the basic movements of a person, such as "spread arms," "crouch," and "raise hands." However, the MHI of the technique disclosed in Patent Document 5 does not detect the skeleton of a person, it is difficult to recognize detailed movements, and it is limited to recognizing large movements using the whole body.

Therefore, MHI is generated for the image (bone image) showing the connection line connecting the human skeleton and each skeleton obtained by skeletal detection, and the skeleton motion history image (Bone image) that measures the posture of the person from the camera image by deep learning technology. Skl MHI) is also disclosed (for example, see Non-Patent Document 2). The Skl MHI is an image in which the area where the luminance difference occurs in each frame is filled with high luminance, and the subsequent frames are drawn with the luminance gradually lowered. The image is a still image that does not contain time information and has embedded time axis information based on changes in brightness.

By the way, there has long been known a system called "body sonic (sensory sound system)" that presents a tactile stimulus linked to music on a chair (see, for example, Non-Patent Document 3). Also known is a small device called "Ontenna" that converts sound into vibration (see, for example, Non-Patent Document 4).

For example, there is known a technique for presenting multimodal information, such as the above-mentioned body sonic, which presents bass components of music as vibrations using a vibrator built into a chair or the like (see, for example, Patent Document 6). In addition to conventional video and audio, there are theme parks and movie theaters that present tactile information such as vibration and somatosensory information such as a sense of movement. There is also a method of transmitting recorded vibration information in addition to the video and audio of television broadcasts via communication using a broadcast-communication cooperation service.

Also disclosed is a game device having a function of providing a player (user) with a visual display as well as a physical sensation (see, for example, Patent Document 7). In this game device, a signal amplified by a high-power amplifier is input to a low-frequency speaker at specific visual display timing. In response to this input signal, the low-frequency speaker becomes a low-frequency sound source, making it possible to give the player a so-called bodysonic experience.

Also disclosed is a sensory acoustic system in which unique tactile sensation presentation devices are installed in the backrest and seat (see, for example, Patent Document 8). In this body-sensory acoustic system, a seat having a backrest and a seat, a first vibration element that band-divides an input audio signal and vibrates according to the first audio signal is arranged in the backrest, and a second vibration element is arranged in the backrest. The second vibration element that vibrates according to the audio signal is placed in the seat, and it is said that it realizes a bodily sensation that does not cause discomfort or oppression even when used for a long time.

In this way, attempts have been made to convert audio signals into tactile stimuli and present them, and in this specification, tactile sensation presentation by audio signal input is referred to as "audio input method".

In addition, mechanoreceptors of tactile stimuli possessed by the human body (receptors for mechanical deformation and vibration possessed by the human body, tactile receptors such as Merkel's tactile discs, Meissner's corpuscles, Pacinian corpuscles, and hair root endings in the skin) is known to be sensitive in the range of 10 Hz to 100 Hz and dull in other frequency ranges (see, for example, Non-Patent Document 5). For example, even if a high-frequency sound exceeding 200 Hz in the audio signal of the video content is directly input to the tactile sense presentation device as a tactile stimulus sound signal, the user of the tactile sense presentation device can hardly perceive it as vibration.

JP-A-2002-8063 JP-A-2002-253718 Japanese Patent Application Laid-Open No. 2020-38440 JP 2018-187383 A Japanese Unexamined Patent Application Publication No. 2008-22142 JP-A-3-102499 JP-A-6-339576 JP 2008-141477 A

"Motion History Image", [online], [searched January 25, 2020], Internet <https://www.ece.iastate.edu/~alexs/classes/2007_Spring_575X/slides/09_MotionHistory.pdf> C. N. Ohyo, T. T. Zin, P. Tin., “Skeleton motion history based human action recognition using deep learning”, [online], [searched on January 25, 2020], Internet <https://ieeexplore.ieee. org/document/8229448〉 “Let’s listen to the body through concerts and BODYSONIC”, [online], [searched on January 25, 2020], Internet <https://jpn.pioneer/ja/corp/sustainability/karadadekikou/about/> “Ontenna”, [online], [searched on January 25, 2020], Internet <https://ontenna.jp> G. A. Gescheide, Stanley J. Bolanowski, K. R. Hardick, “The frequency selectivity of information-processing channels in the tactile sensory system”, Somatosensory & Motor Research, Vol.18, No.3, pp.191-201, 2001.

As described above, when presenting tactile information to the user in conjunction with video content, video analysis is performed to determine the type and timing of tactile stimuli, and tactile stimuli prepared in advance are presented as tactile metadata. Metadata method is enabled. With the tactile metadata method, for example, tactile information related to the game can be presented as video content in real time at the same time as the video, and tactile stimulation synchronized with the movements of the athletes involved in the game can be effectively presented.

Therefore, there is a demand for a technique for automatically generating tactile metadata regarding the movement of a moving object such as an object or a person with high accuracy in real time from only video signals of arbitrary video content.

In particular, in order to detect the movement of human objects with high accuracy, a technique of referring to moving objects other than humans (for example, shuttlecocks and rackets in the case of badminton) is conceivable. Techniques for detecting movements of human objects with high accuracy are desired even in sports that do not exist (for example, judo, wrestling, etc.).

With the recent development of deep learning technology, it has become possible to estimate the skeletal position of a person from a normal still image that does not contain depth information, without using a depth sensor, which was difficult in the past. A skeleton detection algorithm typified by this basically detects a skeleton position for each still image. For this reason, further ingenuity is required to automatically and in real time generate tactile metadata relating to the movement of human objects (athletes, etc.) from only normal camera images for shooting sports.

By the way, as machine learning for motion recognition, it is possible to recognize motions at high speed by using SVM, which is a conventional supervised learning method, but further improvement in accuracy can be expected by using deep learning, which is desired to develop in recent years. CNN is often used for action recognition based on video analysis. However, CNN is a static image-based identification algorithm and does not consider the time axis. In order to understand the motion content of a video scene, it is necessary to handle feature values related to human motion, but since still images do not contain information on the time axis, CNN cannot be expected to identify the motion content with high accuracy. .

Therefore, as a method for recognizing motion from images by CNN, it is conceivable to use images called Motion History Images (MHI). By analyzing this MHI, it becomes possible to recognize and determine the basic movements of a person, such as "spread arms," "crouch," and "raise hands." However, since MHI does not measure each part of a person's joints, it is limited to recognizing large movements using the whole body. For example, in order to create a Motion History Image (MHI) for a baseball video and detect a pitching motion as disclosed in Patent Document 5, the pitcher's area is defined to suppress the influence of noise included in the background. It is necessary to detect with high precision, and furthermore, it is difficult to recognize detailed motions because skeleton detection is not performed.

Here, as disclosed in Non-Patent Document 2, a motion history image (MHI) is generated for an image (bone image) showing a human skeleton obtained by performing skeleton detection and a connection line connecting each skeleton, and deep learning is performed. A technology called Skeleton motion history Image (Skl MHI), which measures the posture of a person from camera images, has improved the accuracy of motion recognition, but there is a demand for further improvements in motion recognition accuracy. .

Furthermore, in the tactile metadata method, if the timing of the occurrence of an event on the video and the timing of the tactile stimulation do not match, a sense of discomfort is caused. It has also been reported that if the visual information and the tactile information are not within ±100 msec, a discrepancy is felt. In other words, if visual information and tactile information are within ±100 msec (approximately within 3 frames as video frames), no sense of discomfort occurs. Moreover, the sound signal based on the haptic metadata in the haptic metadata method is prepared in advance, and is different from the audio signal from the actual video content. Due to this, a sense of reality may be lost.

Therefore, it is necessary to improve the sense of realism and immersion in the sound signal based on the haptic metadata in the haptic metadata method so that there is no deviation from the video signal and the sense of reality is not lost. become.

By the way, as typified by the above-mentioned "Bodysonic", there is known an audio input method that performs vibration presentation by inputting an audio signal of video content. Since both the audio signal and the tactile stimulus are waves with similar frequencies, the affinity between the audio signal and the tactile signal is high. In addition, compared to the separately prepared metadata method that presents tactile stimuli, the voice input method converts the sound of the video into tactile stimulus as it is, so it is less likely to cause discomfort with the video, and can enhance the sense of presence and immersion.

However, as shown in Non-Patent Document 5, humans can generally only feel tactile stimulation at low frequencies of about 200 Hz or less. cannot be presented. For example, even if it is desired to present large vibrations in an explosion scene, powerful vibrations cannot be presented if the sound of the explosion has a high frequency. Alternatively, if a visually impaired person wants to watch a baseball video, the sound of the ball hitting the bat has a high frequency, so the audio input method hardly conveys tactile stimulation. In this way, when the audio input method is used for the purpose of "information compensation" that conveys the content of video content to the visually or hearing impaired, there are cases where discrepancies between the video and the tactile stimulation occur with only audio input.

Therefore, comprehensively, as a haptic metadata method, only from the video signal of the video content, haptic metadata related to haptic stimuli including the movement of objects and moving objects of people are generated automatically and with high accuracy in real time, For a tactile presentation device configured to enhance the sense of presence and immersion while taking advantage of the audio input method, without causing any deviation from the video signal for the sound signal based on the tactile metadata. Techniques for generating tactile presentation signals are desired.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, an object of the present invention is to automatically generate haptic metadata in real time with high accuracy only from a video signal of video content, and to generate an audio signal of the video content for an audio signal based on the haptic metadata. A tactile presentation signal generation device that generates a tactile presentation signal for driving a tactile presentation device in synchronization with a video signal, a video-tactile interlocking system that drives and controls the tactile presentation device based on the generated tactile presentation signal, and a program is to provide

The tactile presentation signal generation device of the present invention automatically generates tactile metadata in real time and with high precision only from the video signal of the video content for video content having a video signal and an audio signal, and based on the haptic metadata. A tactile sense presentation signal generation device for generating a tactile sense presentation signal for driving a tactile sense presentation device in synchronization with a video signal using an audio signal of video content, wherein the audio signal has a low frequency of a predetermined frequency or less. A low-frequency audio extraction unit for extracting audio signals, and a plurality of frame images including a current frame image and a predetermined number of past frame images are extracted from the video signal, video analysis is performed, and video movement based on the first person viewpoint. , a haptic metadata generator for generating haptic metadata for operating a predetermined haptic presentation device, individually indicating information about one or more of motion of an object in a current frame image and motion of at least a moving object of each person. a signal conversion unit that converts each haptic metadata into a sound signal corresponding to a predetermined haptic stimulus; a synchronous synthesizing unit for generating a tactile presentation signal for driving a predetermined tactile presentation device by individually adjusting the gain to a value and synthesizing the low-frequency audio signal in synchronization with the low-frequency audio signal. .

Further, in the tactile sense presentation signal generation device of the present invention, the haptic metadata generation unit uses each of the plurality of frame images to generate haptic metadata indicating movement of the video based on the first-person viewpoint. An optical flow image is generated by optical flow measurement, and based on this optical flow image, it is determined whether or not it is a first-person viewpoint video. and a video motion recognition unit that generates tactile metadata including at least the direction of the first-person viewpoint and the moving speed of the video that indicates the motion of the video based on the first-person viewpoint.

Further, in the tactile sense presentation signal generation device of the present invention, the haptic metadata generation unit generates tactile sense metadata representing movement of an object in the current frame image by using adjacent frame images using each of the multiple frame images. Based on this difference image, detection processing of moving objects related to the object in the current frame image is performed. An object motion recognition unit that generates haptic metadata including at least the identification of each object exhibiting motion, the size of each object, the speed of each object relative to the background, and the distance between objects in the current frame image. .

Further, in the tactile sense presentation signal generation device of the present invention, the haptic metadata generation unit performs skeleton detection for each of the plurality of frame images in order to generate haptic metadata indicating movement of the moving object of at least each person. Based on an algorithm, a first skeleton coordinate set for each person object is generated, and a search range is variably set based on the first skeleton coordinate set to determine the position and size of the skeleton of each person object and its surrounding image. A person object is identified by extracting information, a second set of skeletal coordinates assigned with a person ID is generated, and the second set of skeletal coordinates in each of the plurality of frame images is generated based on the current frame image. Based on this, a single skeletal trajectory feature image showing only the direction of movement of each identified human skeleton is generated, and a convolutional neural network that receives the skeletal trajectory feature image as an input recognizes a specific motion of the person and performs a predetermined motion. Event detection for detecting information about movement of at least each person's moving object that activates a tactile presentation device, and generating haptic metadata including information about the movement of at least each person's moving object corresponding to the current frame image. A part is provided.

Further, in the tactile sensation presentation signal generation device of the present invention, the event detection unit draws, as the skeleton trajectory feature image, a trajectory that is connected for each person's skeletal coordinates in the plurality of frame images. , and one image generated by drawing with decreasing or increasing luminance toward the past.

Further, in the tactile sensation presentation signal generation device of the present invention, the event detection unit may share the skeleton coordinates of each person in the plurality of frame images as the skeleton trajectory characteristic image, or distinguish between the skeleton coordinates of each person in the plurality of frame images. The feature is that each coordinate is color-coded, and the motion of each person's skeleton coordinate is drawn so as to be time-serially gradated in units of frames to create one image.

Further, in the tactile sense presentation signal generation device of the present invention, the event detection unit limits the search range to a maximum person search range surrounding the entire human skeleton, and at least a predetermined region of the human skeleton as an attention search range. and determining the search range so as to include at least the attention search range based on the state transition estimated value of the human skeleton obtained by the state estimation algorithm, and identifying the person object. It is characterized by having a means for performing processing for processing.

Further, in the tactile sensation presentation signal generation device of the present invention, the event detection unit detects a moving object based on a difference image between adjacent frames using each of the plurality of frame images, and detects the moving object detected from each difference image. A moving object other than a person is selected by comparing with the skeleton trajectory feature image showing only the direction of movement of each identified human skeleton, and the coordinate position and size obtained from each difference image are selected for the moving object other than the person. A moving object detecting means for generating a moving object trajectory image connected with movement direction as an element is provided, and the moving object trajectory image is added to the skeletal trajectory feature image showing only the movement direction of each of the identified human skeletons. It is characterized by recognizing a specific action of a person by a convolutional neural network that receives an input obtained by synthesizing

Further, the video-haptic interlocking system of the present invention provides a tactile presentation signal generation device of the present invention, a predetermined tactile presentation device that presents a tactile stimulus, and a tactile presentation signal obtained from the tactile presentation signal generation device. and a control unit that refers to determined drive reference data and controls the predetermined tactile presentation device to be driven.

Furthermore, the program of the present invention constitutes a program for causing a computer to function as the tactile presentation signal generation device of the present invention.

According to the present invention, tactile metadata related to tactile stimuli including the movement of moving objects of objects and people can be automatically generated in real time and with high accuracy only from the video signal of the video content, and the advantages of the voice input method can be utilized. and generating a tactile presentation signal for a tactile presentation device that is configured to improve presence and immersion in a sound signal based on the tactile metadata without causing deviation from a video signal. can be done. In particular, in addition to constant tactile stimulation based on audio signals of video content, haptic metadata related to tactile stimulation including movement of objects and moving objects of people is generated based on video signals of video content, and occasional tactile stimulation is presented. By making it possible, it is possible to realize information compensation for those who are visually or hearing impaired. Moreover, according to the present invention, it is possible to improve the video viewing experience by providing not only the visually and hearing impaired but also the general able-bodied people with a sense of realism and immersion in video viewing. In particular, by providing tactile stimulation related to the type, timing, and situation of play when watching sports videos, it is possible to convey the sports situation to people with visual impairments.

1 is a block diagram showing a schematic configuration of a video-haptic interlocking system including a haptic presentation signal generation device according to one embodiment of the present invention; FIG. 4 is a flow chart showing a processing example of a haptic metadata generation unit in the haptic presentation signal generation device of one embodiment according to the present invention. FIG. 10 is an example showing an example of estimating a movement speed related to motion of a video image based on optical flow measurement by the video viewpoint estimation unit in the tactile sense presentation signal generation device according to one embodiment of the present invention; FIG. It is an example showing an estimation example of object motion by an object detection unit in the tactile sense presentation signal generation device of one embodiment according to the present invention. FIG. 10 is an explanatory diagram relating to human skeleton extraction processing of the event detection unit in the tactile sense presentation signal generation device of one embodiment according to the present invention; (a) is a diagram illustrating one frame image, and (b) is a diagram illustrating an example of human skeleton extraction in one frame image in the tactile presentation signal generation device of one embodiment according to the present invention. 8A and 8B are diagrams each showing a processing example of a search range of a person object in relation to human skeleton extraction processing in the tactile sense presentation signal generation device according to one embodiment of the present invention; FIG. (a) is a diagram showing an image example of a skeletal trajectory feature image (JTI: Joint Trajectory Image) according to the present invention, and (b) is an explanatory diagram of the trajectory feature image (JTI). (a) is a diagram schematically showing an example of one frame image, (b) is an example of a bone image of the prior art, (c) is an example of a Skl MHI (Skeleton Motion History Image) of the prior art, (d) ) is a diagram showing an image example of a skeletal trajectory feature image (JTI) according to the present invention. FIG. 2 is a diagram showing a comparative evaluation of human motion detection accuracy of a prior art bone image, a prior art Skl MHI, and a skeletal trajectory feature image (JTI) according to the present invention; 1 is a block diagram showing a schematic configuration of a control unit and a tactile sense presentation device in a video-haptic interlocking system according to one embodiment of the present invention; FIG.

(System configuration)
A video-haptic interlocking system 10 including a haptic presentation signal generation device 1 according to an embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a video-haptic interlocking system 10 including a haptic presentation signal generation device 1 according to one embodiment of the present invention.

A haptic presentation signal generation device 1 shown in FIG. 1 receives video and audio signals of video content output from a video content output device 2 such as a camera or recording device, and combines the audio input method and the haptic metadata method. , in order to provide the user with tactile stimulation suitable for the video signal via the tactile presentation device 5, tactile metadata is automatically generated in real time with high precision only from the video signal of the video content, and based on the tactile metadata. It is configured as a device for generating a tactile sense presentation signal for driving a tactile sense presentation device in synchronization with a video signal by using an audio signal of the generated sound signal.

In this example, the audio signal of the video content is, for example, a 22.2ch audio signal, and a low-frequency audio signal having a predetermined frequency or lower is extracted from this audio signal and used for tactile stimulation. As for the video signal of the video content, haptic metadata is automatically generated for a specific scene of the video, and the haptic metadata is converted into a sound signal indicating a haptic stimulus prepared in advance and used for the haptic stimulus. Further, in the tactile sense presentation signal generation apparatus 1 shown in FIG. 1, the gain of the low-frequency audio signal and the sound signal based on the haptic metadata are individually adjusted, synchronized, and synthesized, thereby forming a tactile sense presentation device. Generate a tactile presentation signal for driving. As a result, it is possible to change the stimulus to a powerful stimulus in a specific scene while providing a tactile stimulus by voice input that does not feel out of place with video. Since the haptic presentation signal generated by this haptic presentation signal generation device 1 conforms to the framework of the already standardized object-based acoustic system, it can contribute to the standardization of haptic metadata distribution methods in the future.

More specifically, the video-haptic interlocking system 10 shown in FIG.

Although the details will be described later, the haptic presentation signal generation device 1 receives a video signal and an audio signal from the video content output device 2, extracts a low-frequency audio signal from the input audio signal, and generates the input video signal. , converts automatically generated haptic metadata by video analysis into sound signals, and synthesizes the low-frequency audio signal and the sound signal based on the haptic metadata individually by adjusting the gain and synchronizing them. automatically generates a tactile presentation signal for driving the tactile presentation device.

The control unit 4 drives and controls the tactile presentation device 5 based on the tactile presentation signal generated by the tactile presentation signal generation device 1 .

The tactile sense presentation device 5 can be of any form as long as it presents a tactile sense stimulus to humans. The tactile presentation device 5 of this example has a plurality of actuators that are individually driven and controlled by the control unit 4 . However, as another example, the tactile presentation device 5 may have one actuator and may be a plurality of tactile presentation devices 5 individually driven and controlled by the control unit 4 .

A more specific example of the tactile presentation device 5 can be, for example, a “chair-type tactile device”, and the technology related to this “chair-type tactile device” has also been published (for example, STRL Open 2021, “Future Immersion Type VR display”, [online], [searched on January 25, 2020], Internet <https://www.nhk.or.jp/strl/open2021/tenji/5/index.html>). In this technique, a tactile stimulus is presented by a "chair-shaped tactile device" by converting an audio signal linked to a video signal of a tram displayed at a viewing angle of 180 degrees on a flexible display into vibration. With this technique, it is possible to create a highly immersive experience through first-person images and vibration stimulation from the seat.

In addition, as an example of the tactile sense presentation device 5, it is possible to use a “cube type haptic device”, and technology related to this “cube type haptic device” has been published (for example, STRL Open 2021, “Tactile Information Presentation Technology”, [online], [searched on January 25, 2020], Internet <https://www.nhk.or.jp/strl/open2021/tenji/9-3/index.html>). This "cube-shaped tactile device" is an example of the tactile metadata method, but by transmitting the shot timing and bounce timing with the vibration of the cube-shaped device held by the user, it is possible to express the situation of the table tennis rally as a bodily sensation in an easy-to-understand manner. It is a thing.

Further, in the future, in realizing broadcasting using tactile information, the video-tactile sense interlocking system 10 shown in FIG. 1 can be applied to an object-based sound system. In the object-based audio system, sound materials and information such as their structure and playback position (acoustic metadata) are transmitted from the program production site to the home (in-home equipment on the receiving side), and each TV receiver can perform It is a method to complete the sound of the program by synthesizing it with <https://www.nhk.or.jp/strl/open2019/tenji/15.html>). ITU-R (International Telecommunication Union Radiocommunication Sector) is working on the time-series representation format of object-based sound, and SMPTE (Society of Motion Picture and Television Engineers) is working on international standardization of transmission methods. An environment is being established in which it can be widely used around the world.

In such an object-based audio method, it is assumed that the sound of the material is recorded for each audio channel from the transmitting device, transmitted to the receiving-side in-home device, and freely rendered by the receiving-side in-home device. More specifically, on the object-based audio transmission device side, the haptic metadata is converted into a signal of audio waveform data for haptic stimulation (hereinafter also referred to as a "sound signal") and assigned to a specific audio channel. , to the receiving customer premises equipment. In the in-home device on the receiving side, the renderer selects the sound signal or adjusts the sound source, and drives and controls the tactile presentation device, thereby making it possible to present tactile information to the user of the in-home device on the receiving side. . The tactile presentation device in this case corresponds to a speaker functioning as an actuator, and can provide tactile information suitable for video content of the object-based audio method. Even if a device with a small number of speakers (actuators), such as a mobile phone, is used as the receiving device, the appropriate tactile stimulus is presented by mixing the sound signal with the renderer in the same way as a normal audio signal. can.

First, the video content output by the video content output device 2 is arbitrary, and is configured as a video shot in real time or a recorded program, and a predetermined delay processing (for example, T frames of the video signal handled by the tactile sense presentation signal generation device 1) are displayed on the display 3 and visually recognized by the user.

In this embodiment, for convenience of explanation, the video content output device 2 is provided in the transmission device on the broadcasting facility side, and the tactile presentation signal generation device 1, the control unit 4, the tactile presentation device 5, and the display 3 are provided in the home on the receiving side. It is assumed that the video content output device 2 and the tactile presentation signal generation device 1 are connected by broadcasting or communication. However, when the video-haptic interlock system 10 shown in FIG. and a receiving-side in-home device including the display 3. FIG. In the object-based audio system, it is also assumed that the receiving-side in-home device is configured by a mobile terminal such as a mobile phone. In this case, the transmitting device and the in-home device on the receiving side are connected by two-way communication, and the transmitting device assigns the tactile sense presentation signal to a specific audio channel and transmits it to the in-home device on the receiving side. In this case, the in-home device on the receiving side selects the material constituting the tactile sense presentation signal or adjusts the volume according to the viewpoint, for example, using a renderer (not shown) provided outside (or inside) the control unit 4. is performed, and the tactile sense presentation device 5 is driven and controlled.

Hereinafter, the tactile sense presentation signal generation device 1 shown in FIG. 1 will be described in detail. The haptic presentation signal generation device 1 includes a low-frequency audio extraction unit 11 , a haptic metadata generation unit 12 , a signal conversion unit 13 , a haptic stimulation audio data database (DB) 14 , and a synchronous synthesis unit 15 .

The low-frequency audio extraction unit 11 extracts a low-frequency audio signal having a predetermined frequency or less (200 Hz or less in this example) from the audio signal input from the video content output device 2 and outputs the low-frequency audio signal to the synchronous synthesizing unit 15 .

The low-frequency audio extraction unit 11 can extract the low-frequency audio signal by selection processing if the input audio signal has an audio signal from which only low frequencies of 200 Hz or less can be extracted in advance. For example, in 8K video of 22.2ch, low frequency sound of 120 Hz or less called LFE (Low Frequency Effect) is used in 0.2ch. Even if this LFE is used as it is, it becomes a tactile stimulation suitable for vibration. Alternatively, if the input audio signal is a mixed audio signal within the audible range, the low-frequency audio extraction unit 11 extracts the audio signal by digital filtering, or emphasizes the low-frequency audio signal by equalizing and removes the high-frequency part. By suppressing the noise, it is possible to extract a substantially low-frequency audio signal.

The haptic metadata generation unit 12 extracts a plurality of frame images including a current frame image and a predetermined number of past frame images from the video signal of the video content input from the video content output device 2, performs video analysis, and performs video analysis. generating haptic metadata for actuating a given haptic presentation device, including information about one or more of motion of a video relative to , motion of an object in a current frame image, and motion of at least each person's moving object It is a functional unit and includes a multiple frame extraction unit 121 , a video viewpoint estimation unit 122 , an object detection unit 123 and an event detection unit 124 .

The multi-frame extraction unit 121 extracts a multi-frame image including a current frame image and T (T is an integer equal to or greater than 1) frames of past frame images from the video signal of the video content input from the video content output device 2. and output to the video viewpoint estimation unit 12 , the object detection unit 123 , and the event detection unit 124 .

The video viewpoint estimation unit 12 includes a viewpoint estimation unit 1221 , a video motion recognition unit 1222 and a first metadata generation unit 1223 .

The viewpoint estimation unit 1221 generates an optical flow image by optical flow measurement using each of the plurality of frame images, determines whether or not it is a first-person viewpoint video based on this optical flow image, and determines whether or not it is a first-person viewpoint video. and the optical flow image are output to the video motion recognition unit 1222 .

Based on the determination result obtained from the viewpoint estimating unit 1221, the video motion recognition unit 1222, when the video is determined to be the first-person viewpoint video, recognizes the motion of the video based on the first-person viewpoint from the optical flow image (at least the motion of the first-person viewpoint). including the direction and moving speed of the video), outputs the estimation result to the first metadata generation unit 1223, and when it is determined that the video is not the first-person viewpoint video, the first metadata generation unit 1223 1223 output.

When the first metadata generation unit 1223 obtains the motion of the video based on the first-person viewpoint (including at least the direction of the first-person viewpoint and the movement speed of the video) as an estimation result obtained from the video motion recognition unit 1222, Only, haptic metadata indicating the motion of the video based on the first-person viewpoint is generated and output to the signal conversion unit 13 in units of frames.

For example, in the video viewpoint estimating unit 12, if the video signal of the video content to be processed indicates the scenery seen by the driver of the vehicle captured by the video camera, it is the first person viewpoint video from the optical flow image. It is possible to analyze whether or not it is, and to generate haptic metadata that estimates the movement of the image (the direction of the first-person viewpoint and the moving speed of the image).

The object detection unit 123 includes a moving object detection unit 1231 , an object motion recognition unit 1232 and a second metadata generation unit 1233 .

The moving object detection unit 1231 generates a difference image between adjacent frames using each of the multiple frame images, and based on this difference image, detects a moving object related to an object (including a person) in the current frame image. Then, a detection result indicating presence or absence of detection of an object (including a person) in the current frame image and the current frame image are output to the video motion recognition unit 1222 .

When the detection result obtained from the moving object detection unit 1231 is detection of one or more objects in the current frame image, the object movement recognition unit 1232 recognizes the movement of the object (including the person) in the current frame image (at least each object ID (identifier) of the object, size of each object, relative speed of each object with respect to the background, and distance between objects in the current frame image), and the calculation result is sent to the second metadata generation unit 1233 , and when one or more objects are not detected, that effect is output to the second metadata generation unit 1233 . When one or more objects are detected in the current frame image, the object motion recognition unit 1232 also requests the skeleton trajectory feature image generation unit 1243 in the event detection unit 124 to detect the object (person) in the current frame image. ) (including at least the object ID (identifier) of each object, the size of each object, the relative speed of each object with respect to the background, and the distance between objects in the current frame image).

The second metadata generation unit 1233 generates the detection result obtained from the object motion recognition unit 1232 as the motion of the object (including the person) in the current frame image (at least the object ID (identifier) of each object, the size of each object, each Only when the relative velocity of the object with respect to the background and the distance between the objects in the current frame image are obtained, haptic metadata indicating the movement of the object (including the person) in the current frame image is generated, and the frame It outputs to the signal converter 13 in units.

For example, in the object detection unit 123, if the video signal of the video content to be processed indicates the flight of a “bird” captured by a camera as a moving image, the difference image between the adjacent frames is used as the basis of the difference image between the adjacent frames. Birds can be detected, the size of each bird, the relative speed of each bird to the background, and the distance between birds in the current frame image can be calculated, and the relative speed of the birds to the background , haptic metadata can be generated that describe the movement of each "bird" or the movement of the "wind" relative to the overall movement of the "birds".

The event detection unit 124 includes a human skeleton extraction unit 1241 , a person identification unit 1242 , a skeleton trajectory feature image generation unit 1243 , a human motion recognition unit 1244 and a third metadata generation unit 1245 .

The human skeleton extraction unit 1241 extracts a skeleton coordinate set P ⁿ _b (n: number of detected persons, b: A skeleton ID) is generated and output to the person identification unit 1242 together with the multiple frame images including the current frame image.

The person identification unit 1242 variably sets a search range (details will be described later) based on the skeleton coordinate set P ⁿ _b for each of the multiple frame images, and determines the position and size of the skeleton of each person and the surrounding images. A person is identified by extracting information, and a skeleton coordinate set P ⁱ _b (i: person ID, b: skeleton ID) to which a person ID is assigned is generated and output to the skeleton trajectory feature image generation unit 1243 .

The skeletal trajectory feature image generation unit 1243 generates one skeletal trajectory feature image showing only the movement direction of each identified human skeleton based on the current frame image and the skeletal coordinate set P ⁱ _b in the multiple frame images. to generate Here, the skeleton trajectory feature image will be described in detail later, but is named JTI (Joint Trajectory Image) in the present specification.

Here, the above-described moving object detection unit 1231 does not necessarily have to be provided for recognizing movements in a judo competition as in this example, but each of the multiple frame images is used to detect a difference image between adjacent frames. Based on this, a moving object is detected, and a detection result indicating whether or not an object (including a person) is detected in the current frame image and the current frame image are output to the video motion recognition unit 1222 .

Therefore, when the detection result obtained from the moving object detection unit 1231 is detection of one or more objects in the current frame image, the skeleton trajectory feature image generation unit 1243 uses each of the multiple frame images to generate an adjacent frame image. A moving object other than a person is selected by comparing it with the skeletal trajectory feature image (JTI), and the coordinate position, size, and moving direction obtained from each difference image for the moving object other than the person. elements, and the moving object trajectory image is added to the skeletal trajectory feature image (JTI) and rendered (synthesized) and output to the human motion recognition unit 1244 .

Separately from the moving object detection unit 1231 described above, the event detection unit 124 uses each of the multiple frame images to detect a moving object based on the difference image between adjacent frames, and A moving object detector may be provided that uses each to generate a difference image between adjacent frames.

The human action recognition unit 1244 inputs the skeletal trajectory feature image (JTI) from the skeletal trajectory feature image generation unit 1243 and draws (combines) the moving object trajectory image, and recognizes the moving object trajectory image by CNN (convolutional neural network). , to recognize specific actions of a person and detect information about the motion of each person and moving objects other than the person. More specifically, the human motion recognition unit 1244 recognizes predetermined impact presentation information regarding the motion of each person and moving object other than the person who operates the tactile sense presentation device 5, that is, the identification of each person in the current frame image, Positional coordinates (and team classification in the case of a team sport, although not applicable in this example because it is a judo sport), an event ID indicating the type of movement of each person, and the timing and timing of activating the tactile presentation device The strength and timescale information for presenting impact indicating speed is detected and output to the third metadata generation unit 1245 .

The third metadata generation unit 1245 generates tactile metadata representing information on the movement of each person and moving objects other than the person obtained from the human motion recognition unit 1244, corresponding to the current frame image. Identification of each person, positional coordinates (and team classification in the case of a team sport, which is out of scope because this example is judo), an event ID indicating the type of movement of each person, and a tactile presentation device tactile sense metadata including strength and timescale information for presenting an impact indicating the timing and speed of activating the tactile sensation is generated, and output to the signal conversion unit 13 in units of frames.

For example, in the event detection unit 124, if the video signal of the video content to be processed indicates a Judo sports video captured by a camera, for example, a “throw” or a “submission” is detected for each “person”. Motion can be detected, and tactile metadata can be generated that identifiably represents the motion of each “person” of “throwing” and “submission” as an event ID.

The signal conversion unit 13 receives each haptic metadata for the current frame image generated by the haptic metadata generation unit 12, refers to the haptic stimulus audio data (DB) 14, and converts a predetermined haptic in each haptic metadata. A file of sound signals corresponding to stimuli (sound waveform data signals for tactile sense stimulation) is read out, converted into sound signals, and output to the synchronous synthesizing unit 15 .

Specifically, for example, when the video viewpoint estimating unit 12 generates tactile metadata indicating the scenery seen by the driver of the vehicle captured by the camera, the signal converting unit 13 detects that the vehicle is moving at high speed. It can be converted into a sound signal corresponding to a predetermined tactile sense stimulus, such as raising the level of the sound signal during stop and lowering the level of the sound signal during stoppage.

Further, the signal conversion unit 13 generates, for example, the object detection unit 123 with tactile metadata that expresses the movement of each “bird” or the movement of the “wind” relative to the movement of all the “birds”. When the motion is moving at high speed, the level of the sound signal is increased, and when the motion is moving at low speed, the level of the sound signal is decreased.

For example, when the event detection unit 124 generates the tactile metadata expressing the motion of each “person” of “throwing” and “submission”, the signal conversion unit 13 detects the “throwing” of each “person”. It can be converted into a sound signal corresponding to a predetermined tactile stimulus, such as raising the level of the sound signal when the impact is strong and strong, and lowering the level of the sound signal when the impact is gentle and small. .

The synchronous synthesizing unit 15 individually gains the low-frequency audio signal obtained from the low-frequency audio extracting unit 11 and the sound signal based on each tactile metadata obtained from the signal converting unit 13 up to a predetermined value. A tactile presentation signal for driving a predetermined tactile presentation device 5 is generated by adjusting and synthesizing in synchronization with a low-frequency audio signal (audio frame corresponding to a video signal). After synthesizing the low-frequency audio signal and the sound signal based on each haptic metadata, the synchronous synthesizing unit 15 may generate a tactile presentation signal by uniformly or by equalizing and amplifying. As a result, it is possible to change the stimulus to a powerful stimulus in a specific scene while providing a tactile stimulus by voice input that does not feel out of place with video. The haptic presentation signal generated by this haptic presentation signal generation device 1 can also conform to the already standardized framework of the object-based acoustic system.

In particular, when the synchronous synthesizing unit 15 individually adjusts the gains of the low-frequency audio signal and the sound signal based on each tactile metadata obtained from the signal converting unit 13 to predetermined values, Synchronize with the low-frequency audio signal (audio frame corresponding to the video signal) by adjusting the gain so that the signal strength of all the sound signals based on the haptic metadata is greater than the signal strength of the low-frequency audio signal. Synthesis is preferred.

Hereinafter, the haptic metadata generation processing in the haptic presentation signal generation device 1 will be described more specifically based on FIG. 2 and with reference to FIGS. 3 to 8. FIG.

(Haptic metadata generation processing)
FIG. 2 is a flowchart showing a processing example of the haptic metadata generation unit 12 in the haptic presentation signal generation device 1 according to one embodiment of the present invention. FIG. 3 is an example showing an example of estimating the movement speed related to the motion of the video based on the optical flow measurement by the video viewpoint estimation unit 122 in the tactile sense presentation signal generation device 1 according to one embodiment of the present invention. FIG. 4 is an example showing an estimation example of object motion by the object detection unit 123 in the tactile sense presentation signal generation device 1 according to one embodiment of the present invention. Furthermore, FIG. 5 is an explanatory diagram relating to the human skeleton extraction processing of the event detection unit 124 in the tactile sense presentation signal generation device 1. As shown in FIG. FIG. 6(a) is a diagram showing an example of a one-frame image, and FIG. 6(b) is a diagram showing an example of human skeleton extraction in the one-frame image in the tactile sense presentation signal generation device 1. As shown in FIG. FIGS. 7(a) and 7(b) are diagrams showing examples of human object search range processing related to human skeleton extraction processing in the tactile sense presentation signal generation device 1 according to one embodiment of the present invention. FIG. 8(a) is a diagram showing an image example of a skeletal trajectory feature image (JTI) according to the present invention, and FIG. 8(b) is an explanatory diagram of the trajectory feature image (JTI).

As shown in FIG. 2, the tactile sense presentation signal generation device 1 first extracts a current frame image and T (T is an integer equal to or greater than 1) frames for the video input from the video content output device 2 by the multi-frame extraction unit 121. A plurality of frame images including previous frame images are extracted (step S1).

Subsequently, the tactile sensation presentation signal generation device 1 generates an optical flow image by optical flow measurement using each of the plurality of frame images by the video viewpoint estimation unit 122, and determines whether or not the video is a first-person viewpoint video based on the optical flow image. (step S2).

When the video viewpoint estimation unit 122 determines that the video is the first-person viewpoint video (step S3: Yes), the video viewpoint estimation unit 122 determines the movement of the video with respect to the first-person viewpoint from the optical flow image (at least the direction of the first-person viewpoint and the direction of the video). movement speed), outputs the estimation result to the first metadata generation unit 1223, generates haptic metadata indicating the movement of the video based on the first person viewpoint, and outputs it in units of frames. (Step S4).

For example, in the video viewpoint estimating unit 12, the video signal of the video content to be processed represents the scenery seen from the driver of the vehicle captured by the camera as a moving image, and the optical flow image F' as shown in FIG. Suppose you got In the optical flow image F', the trajectories of the lights of the vehicle Ov traveling on the road Or, the street lights Os, and the like are shown, and the coordinate point Vp indicating the viewpoint direction can also be estimated. Therefore, the video viewpoint estimation unit 12 can generate haptic metadata that estimates the motion of the video (the direction of the first-person viewpoint and the moving speed of the video).

Note that when the video viewpoint estimation unit 122 does not determine that the video is the first-person viewpoint video (step S3: No), it does not generate tactile metadata indicating the movement of the video with respect to the first-person viewpoint, and does not generate the tactile metadata in step S5. transition to

Subsequently, the tactile sense presentation signal generation device 1 uses the object detection unit 123 to generate a difference image between adjacent frames using each of the plurality of frame images. ) is performed (step S5).

Then, when one or more objects are detected in the current frame image, the object detection unit 123 detects the movement of the object (including the person) in the current frame image (at least the object ID (identifier) of each object, each object , the relative speed of each object with respect to the background, and the distance between objects in the current frame image), and generate haptic metadata indicating the movement of objects (including people) in the current frame image, Output in units of frames (step S6).

For example, as shown in FIG. 4, in the object detection unit 123, the video signal of the video content to be processed indicates the flight of a “bird” captured by a camera as a moving image. , a "bird" can be detected in the current frame image F as shown in FIG. On the current frame image F, the attention range Rd in which the "bird" is detected as the object Ob can also be detected, and here, for example, clouds are shown as the background Oc. For this reason, the object detection unit 123 calculates the size of each "bird", the relative speed of each "bird" to the background, and the distance between the "birds" in the current frame image. , haptic metadata representing the movement of each "bird" or the movement of the "wind" relative to the overall movement of the "birds" can be generated.

Note that when the object detection unit 123 does not detect the movement of the object (including the person) in the frame image, it does not generate the haptic metadata indicating the movement of the object (including the person), and proceeds to step S7. do.

Subsequently, the human skeleton extraction unit 1241 in the event detection unit 124 of the tactile sense presentation signal generation device 1 extracts a skeleton coordinate set P ⁿ _b (n: Detected number of people, b: skeleton ID) is generated (step S7).

Recent advances in deep learning technology have made it possible to estimate the position of a person's skeleton from a normal image. As typified by OpenPose and VisionPose (NextSystem), there are some open source skeleton detection algorithms that are open to the public. Therefore, the human skeleton extraction unit 1241 of this example uses VisionPose to detect 30 points of the human skeleton for each frame image, as shown in ^FIG _. do.

In VisionPose, in _FIG . _{5 ,} ^Pn1 : "head", ^Pn2 : "nose" ^, _{Pn3 : "left eye", Pn4: "right eye", Pn5} ^{: "} left ear", ^Pn ₆ : "right ear _" , ^Pn7 : "neck", ^Pn8 : ^" spine (shoulder)", _Pn9 : " _left shoulder", ^Pn10 : " _right shoulder", ^Pn11 : " _left elbow" ”, ^Pn12 : “right elbow _{” ,} ^Pn13 : “left wrist” _{, Pn14: “right wrist”, Pn15 :} ^“ _left hand”, ^{Pn16 :} “right hand”, ^Pn17 _: “ Left Thumb", ^Pn18 : "Right Thumb" ^{, Pn19} : " _{Left Fingertip", Pn20: "Right Fingertip", Pn21} ^: _{"Spine (Center)", Pn22 : "} ^Spine (Base) _" end)”, ^Pn23 : “left hip _” , ^Pn24 : “right hip ^” _{, Pn25: “left knee”, Pn26 :} ^“ right knee”, ^Pn27 _: “left ankle _” ”, ^Pn28 : “right ankle”, ^Pn29 : “ _left foot”, and ^Pn30 : “ _right foot”, and drawing is possible by connecting each coordinate position with a line as shown in the _figure . is.

FIG. 6B shows a frame image Fa obtained by extracting a human skeleton from the one-frame image F of the judo competition shown in FIG. 6A based on the VisionPose skeleton detection algorithm. In the frame image F shown in FIG. 6A, only the person objects Op1 and Op2 (players) are shown. In other sports, moving objects other than people (such as rackets and shuttlecocks in badminton) or objects such as spectators (substantially static objects) may appear. However, by applying VisionPose's skeleton detection algorithm, it is possible to extract the skeleton of the person only for the person of the player and referee person objects. In this example, as shown in FIG. 6B, skeleton coordinate sets P ¹ _b and P ² _b corresponding to human objects Op1 and Op2 can be estimated and generated. As can be seen from FIG. 6B, the skeleton of each person can be estimated with relatively high accuracy even in the judo competition. Since the skeleton detection algorithm is limited to estimation in units of still images, the tactile sense presentation signal generation device 1 identifies a person as a subsequent process, and converts the transition of the skeleton position of each person into a single skeleton trajectory feature. It draws on an image (JTI) and performs highly accurate motion recognition considering the time axis by CNN.

Subsequently, the tactile sense _presentation signal generation device 1 causes the person identification unit 1242 to variably set the search range based on the skeleton coordinate set ^Pnb for each of the plurality of frame images, and determines the position and size of the skeleton of each person. , a person is identified by extracting peripheral image information thereof, and a skeleton coordinate set P ⁱ _b (i: person ID, b: skeleton ID) to which a person ID is assigned is generated (step S8).

The human skeleton extraction unit 1241 described above makes it possible to detect one or more human skeletons as _a skeleton coordinate set ^Pnb for each of the multiple frame images. However, since there is no "who" information in _the skeleton coordinate set ^Pnb of each frame image, it is necessary to identify the skeleton of each person. For this _{identification} , image information near the coordinates of each skeleton coordinate set ^Pnb in each frame image is used. That is, the person identification unit 1242 extracts the position and size of each person's skeleton and peripheral image information (color information and texture information around the face or back) based on the skeleton coordinate set ^Pnb _. , identify a person and generate a skeleton coordinate set P ⁱ _b (i: person ID, b: skeleton ID) to which a person ID is assigned.

For example, in judo, a match is played in white and blue uniforms. By referring to the _color information in the frame image F as the image information near the position of the skeleton of each skeleton coordinate set ^Pnb , the player can be identified. be possible. Also, in a badminton game, when the image is taken with the court held vertically, if the position of the skeleton of each skeleton coordinate set P ⁿ _b is on the upper side of the screen in the frame image F, the player is in the back, and the lower side of the screen is If there is, it can be identified as the player in front.

Therefore, although the skeleton detection algorithm in the human skeleton extraction unit 1241 is limited to estimation for each still image, it is possible to recognize a person as a moving object based on the skeleton coordinate set ^Pnb _.

It should be noted that the human skeleton extraction unit 1241 described above often detects the skeletons of other persons, such as referees and spectators, who are not subject to tactile stimulus presentation, in addition to athletes. Referees often wear different clothes than players, so they can be identified by color information. Also, spectators are often farther away than athletes, so they can be identified by the size of their skeletons. In this way, considering the rules and shooting conditions of each game, and setting the surrounding image information (color information and texture information around the face or back) suitable for person identification, the player who is the target of the tactile stimulus presentation can be identified.

By the way, the person identification unit 1242 of the present embodiment variably sets the search range (person search range R ⁱ and attention search range Rb ⁱ ) for each frame image in order to cope with overlapping and occlusion of each person. For example, when the human skeleton extraction unit 1241 extracts each skeleton coordinate set P ⁿ _b (not shown) for the human objects Op1 and Op2 (players) and the human object Op3 (referee) shown in FIG. , the person identification unit 1242 can variably set the person search range ^Ri and the attention search range ^Rbi for each frame image. This search range R ⁱ is set for each person ID (i) in FIG. It is represented by a circumscribing rectangle. _Also , _a region surrounding the waist region ( ^Pn22 , ^Pn23 _, ^Pn24 ) of each person is represented as a focused search range ^Rbi .

More specifically, the person identification unit 1242 of this embodiment limits the search range for a person in each frame image to a person search range R ⁱ that encloses the entire human skeleton at the maximum. A region (in this example, a region surrounding the waist region (P ⁿ ₂₂ , P ⁿ ₂₃ , P ⁿ ₂₄ )) is defined as the attention search range Rb ⁱ . Based on the state transition estimated value, a search range is determined so as to include at least the focused search range ^Rbi , and processing for identifying the person object is performed. As a result, for example, as shown in FIG. 7B, even when the motion of each person changes or when the size of the person relative to the frame image changes, erroneous recognition of other people can be prevented, and the processing speed can be improved. can also be improved. In particular, it is effective to use the search range to accurately identify a player from images such as judo, in which persons to be identified overlap heavily and the background is complicated.

In other words, the person identification unit 1242 of this embodiment selects a predetermined range (in this example, the waist region) that enables color identification among each skeleton coordinate set P ⁿ _b in Op1, Op2, and Op3 of the person object of each player and referee. Since the colors (blue, white _, brown) _of ( ^Pn22 , ^Pn23 , ^Pn24 ) ₎ are predetermined as the attention search range ^{Rb i} _, the skeletal coordinate sets ^Pnb of a plurality of detected persons overlap. In this case, by narrowing down the search to the attention search range ^Rbi , a person can be extracted and tracked with high accuracy in each frame image. In addition, since a skeleton other than the analysis target may be detected in the background, the skeleton of the person to be analyzed is assigned a person ID (i) to determine the skeleton coordinates P ⁱ _b of the person to be tracked. can be identified.

The size and shape of the search range (person search range R ⁱ and attention search range Rb ⁱ ) are determined based on the state transition estimated value of the human skeleton obtained by a state estimation algorithm such as a Kalman filter or a particle filter. It is determined so as to include the attention search range Rb ⁱ (the waist region of each person in this example).

When the search range (person search range R ⁱ and attention search range Rb ⁱ ) is detected stably, the range can be narrowed, and when the detection is unstable, the range can be widened. setting a search range determined based on the state transition estimated value of the skeleton of the state estimation algorithm Based on the state transition estimated value of the person's skeleton obtained in , the ratio of successful detection is calculated during the time window for T frames, and if the ratio is a predetermined value or more, it is considered stable, and if it is less than the predetermined value. The search range can be variably set by making the search range unstable.

Next, the tactile sensation presentation signal generation device 1 uses the skeleton trajectory feature image generation unit 1243 to generate the movement of each identified human skeleton based on the skeleton coordinate set P ⁱ _b in the multiple frame images based on the current frame image. A single skeletal trajectory feature image (JTI) showing only the direction is generated (step S9).

Here, in drawing and generating the skeleton trajectory feature image (JTI), first, let P ^ib (t) be the skeleton coordinate set P ⁱ _b in an arbitrary frame image, set the current frame image to t=0, and set the skeleton in the current frame _image . The coordinate set P ⁱ _b is represented by P ⁱ _b (0), and the skeleton coordinate set P ⁱ _b in the frame images of the past T frames is represented by P ⁱ _b (T). That is, if the frame number of the current frame image is t=0 and the frame numbers up to T frames in the past are represented by t=T, the skeleton trajectory feature image generation unit 1243 uses the current frame image as a reference, t=0, 1, and so on. , . . . , T, it is possible to generate one skeleton trajectory feature image (JTI) showing only the movement direction of each identified human skeleton. A skeletal trajectory feature image (JTI) is, so to speak, an image in which past optical flows are connected based on the current frame image, and information on the time axis is included as one image.

T, which is the duration of the trajectory feature amount in the skeletal trajectory feature image (JTI), can be set arbitrarily. Further, it is not always necessary to use all the 30 points shown in FIG. 5 for the skeletal coordinates used to generate one skeletal trajectory feature image (JTI). It can also be configured to improve.

The skeletal trajectory feature image (JTI) shall draw a trajectory connected to each person's skeletal coordinates by using the skeletal coordinates of each person in the frame images of the past T frames from the current frame image. In this case, one image is generated by drawing with decreasing or increasing luminance toward the past. Preferably, the skeletal trajectory feature image (JTI) is color-coded according to the skeletal coordinates of each person in the past frame images for T frames from the current frame, and the movement (transition) of each person's skeletal coordinates is displayed in units of frames. It is assumed that the image is drawn so as to have gradation in chronological order.

For example, when drawing a trajectory connecting each person's skeletal coordinates from the current frame to the past T frames, the brightness b is b=255×(T−t)/T.
shall be defined as

In addition, it may be drawn so that the brightness increases as it goes back in time, and in this case,
b=255×t/T
can be

Here, when t = 0 is the current frame image and the past T frames are to be processed (t = 0 to T), b is 0 to 255, and the value is color-coded for each person's skeleton coordinates. It is preferred to express For example, FIG. 8A is a diagram showing an image example of a skeletal trajectory feature image (JTI) according to the present invention. In FIG. 8(a), grayscale display is used for recognition processing. (or "white" with the highest luminance value), and for both human objects Op1 and Op2, for example, "head" (P ¹ ₁ ) and (P ² ₁ ) are set to "blue". In addition, the "left fingertips" (P ¹ ₁₉ ) and (P ² ₁₉ ) are drawn in different colors such as "red". Also, in this embodiment, as shown in FIG. 8B, the person objects Op1 and Op2 are not color-coded to distinguish them, but the person objects Op1 and Op2 may also be color-coded. To color-code up to 30 skeletal coordinates for , 60 colors need to be defined. In the skeletal trajectory feature image (JTI) according to the present invention, while the color is fixed for each person's skeletal coordinates, only the brightness is drawn darker (or brighter) as it goes back from the current frame image.

Therefore, the skeletal trajectory feature image generation unit 1243 color-codes the skeletal coordinates of each person in the multi-frame images as a skeletal trajectory feature image (JTI), commonly or separately for each person. , the movement (transition) for each frame coordinate of each person is drawn in units of frames so as to be gradated in time series.

Further, the skeleton trajectory feature image generation unit 1243 can also draw a trajectory other than the human skeleton, such as a ball, on the skeletal trajectory feature image (JTI) in the case of a ball game, using the function of the moving object detection unit 1231 . In this case, since the moving direction of the ball and the like are added to the feature amount, the determination accuracy of action recognition is improved.

That is, when the skeletal trajectory feature image generation unit 1243 detects detection of one or more objects in the current frame image as a detection result obtained from the moving object detection unit 1231, the tactile sense presentation signal generation device 1 detects the detection of one or more objects in the current frame image. Each of the images is used to generate a difference image between adjacent frames, a moving object other than the person is selected by comparison with the skeletal trajectory feature image (JTI), and for the moving object other than the person, the difference image obtained from each difference image A moving object trajectory image is generated by connecting coordinate position, size, and moving direction as elements, and the moving object trajectory image is added to the skeletal trajectory feature image (JTI) and rendered (synthesized) by the human action recognition unit. 1244 (step S10).

Separately from the moving object detection unit 1231 described above, the event detection unit 124 uses each of the multiple frame images to detect a moving object based on the difference image between adjacent frames, and A moving object detector may be provided that uses each to generate a difference image between adjacent frames.

In other words, the event detection unit 124 is not required in a case such as a judo competition in which there are no moving objects other than the person involved in the competition by the skeletal trajectory feature image generation unit 1243 (even if it is provided as a process, there is no harmful effect). ), when there are moving objects other than people involved in the game (for example, shuttlecocks and rackets in badminton competitions, balls and rackets in table tennis and tennis competitions, etc.), the trajectory of the movement of the moving objects other than the people is detected, An object trajectory image is generated, and the moving object trajectory image is added to the skeletal trajectory feature image (JTI) to draw (synthesize). As a result, for example, when there is a moving object other than the person involved in the game, the amount of information related to the movement of the person involved in the game increases, so the human action recognition unit 1244 in the subsequent stage can recognize the human action more accurately. Therefore, by providing the function of the moving object detection unit 1231 so that it can be used by the event detection unit 124 as well, any game can be handled with the same processing. Configurable.

Subsequently, the tactile sense presentation signal generation device 1 uses the human motion recognition unit 1244 to recognize a specific motion of the person by a CNN (convolutional neural network) that receives the skeletal trajectory feature image (JTI) as an input, and recognizes the tactile sense presentation device 5. Information about the movement of each predetermined person to be activated and the moving object other than the person is detected (step S11). The information about the movement of each person and moving objects other than the person includes the identification and position coordinates of each person in the current frame image. team classification), an event ID indicating the type of movement of each person, and information on intensity and time scale for impact presentation indicating the timing and speed of actuating the tactile presentation device.

In machine learning by CNN, a skeletal trajectory feature image (JTI) for learning is created in advance and learned. In this way, CNN (convolutional neural network), which is one of deep learning, is used for recognition processing in the human action recognition unit 1244 . A CNN is a neural network with many deep layers, and is a network that exhibits excellent performance especially in the field of image recognition. This network is constructed by piling up layers with several characteristic functions such as "convolution layer" and "pooling layer", and is currently used in a wide range of fields. The "convolution layer" processing achieves high accuracy, and the "pooling layer" processing achieves versatility that does not depend on the shooting angle of view.

By analyzing skeletal trajectory feature images (JTI) using this CNN, motion events such as grappling, throwing, and groundwork can be identified with high accuracy regardless of the player's shooting size and position. By assigning an event ID based on this information and generating haptic metadata for controlling the haptic device 5, it is possible to present haptic stimuli even when watching sports videos in real time. Become.

Finally, the third metadata generation unit 1245 of the tactile sense presentation signal generation device 1 generates information about the movement of each person and moving objects other than the person obtained from the human movement recognition unit 1244 in correspondence with the current frame image. That is, an event indicating the identification of each person in the current frame image, the position coordinates (and the team classification in the case of a team game, although this example excludes judo because it is a judo game), and the type of movement of each person. ID, and tactile metadata (for impact presentation) including strength and time scale information for impact presentation indicating the timing and speed of actuating the tactile presentation device, and output to the control unit 4 in units of frames ( step S12).

(Experimental verification)
An evaluation experiment was conducted to demonstrate the effectiveness of the tactile sense presentation signal generation device 1 according to the present invention.
FIG. 9(a) is a diagram schematically showing an example of one frame image, FIG. 9(b) is an example of a conventional bone image, and FIG. 9(c) is a Skl MHI (Skeleton Motion History Image) of the conventional technique. FIG. 9(d) is a diagram showing an image example of a skeletal trajectory feature image (JTI) according to the present invention. FIG. 10 is a diagram showing a comparative evaluation of human motion detection accuracy of the bone image of the prior art, the Skl MHI of the prior art, and the skeletal trajectory feature image (JTI) according to the present invention.

First, before comparative evaluation, from the judo match video (see FIG. 9(a)), the bone image of the conventional technology (see FIG. 9(b)), the Skl MHI of the conventional technology (see FIG. 9(c)), Approximately 2,000 positive and negative images were created for each of the skeletal trajectory feature images (JTI) (see FIG. 9D) according to the present invention, and pre-learning was performed by CNN.

FIG. 10 shows the result of identification using another game video. In FIG. 10, the precision rate, recall rate, and the value of F value (F-Measure), which is an integrated index of these. In any of the four game situations (scene classification) state identification determination and "throw" detection accuracy, learning using the skeletal trajectory feature image (JTI) according to the present invention yields the best results. Got. Therefore, it was confirmed that the tactile sense presentation signal generation device 1 using the skeletal trajectory feature image (JTI) according to the present invention is more effective than motion recognition using the bone image of the prior art and the Skl MHI of the prior art. Skl MHI was also evaluated by color-coding for each skeletal coordinate. ), it is thought that the connecting lines connecting each skeleton have an adverse effect on motion recognition.

(Controller unit)
FIG. 11 is a block diagram showing a schematic configuration of the control unit 4 and the tactile sense presentation device 5 in the visual haptic interlocking system 10 according to one embodiment of the present invention. The control unit 4 includes a receiving section 41, an analyzing section 42, a storage section 43, and n (an integer equal to or greater than 1) driving sections 44-1 to 44-n. Further, the tactile sense presentation device 5 includes n (an integer equal to or greater than 1) vibration actuators 51-1 to 51-n.

The receiving unit 41 is a functional unit that inputs a tactile sense presentation signal from the tactile sense presentation signal generation device 1 and outputs the tactile sense presentation signal to the analysis unit 42 . The tactile presentation signal is obtained by synthesizing the low-frequency audio signal extracted from the audio signal of the video content and the audio signal based on the haptic metadata generated from the video signal of the video content by individually adjusting the gain. The haptic metadata is uniquely generated by the video viewpoint estimation unit 12 , the object detection unit 123 and the event detection unit 124 . In particular, the tactile metadata generated by the event detection unit 124 includes the identification of each person in the current frame image, the position coordinates, (and the team classification in the case of a team competition), the event ID indicating the type of movement of each person, and intensity and timescale information for the impact presentation that indicates when and how quickly to activate the tactile presentation device.

Based on the tactile sense presentation signal obtained from the tactile sense presentation signal generation device 1, the analysis unit 42 refers to predetermined drive reference data stored in the storage unit 43, and generates n drive units 44-1 to 44- 51-1 to 51-n of the tactile presentation device 5 via .n. For example, the analysis unit 42 identifies the person in the sound signal based on the tactile metadata generated by the event detection unit 124, position coordinates (and team classification), an event ID indicating the type of movement of each person, and , the operating timing of each of the vibration actuators 51-1 to 51-n with reference to the predetermined drive reference data from information on the strength and timescale for presenting the impact indicating the timing and speed of operating the tactile presentation device; The intensity and operation time are determined to drive and control the drive units 44-1 to 44-n.

The storage unit 43 stores predetermined driving reference data for controlling driving of the driving units 44-1 to 44-n based on the tactile sense presentation signal. The drive reference data is a vibration actuator 51-1 as a tactile stimulus associated with a low-frequency audio signal forming a tactile presentation signal and a sound signal based on tactile metadata generated from a video signal of video content. . . . 51-n are expressed in a predetermined table or function for actuation timing, strength, and actuation time. The storage unit 43 also stores programs for realizing the functions of the control unit 4 . That is, the functions of the control unit 4 are realized by reading and executing the program by a computer that constitutes the control unit 4 .

The drive units 44-1 to 44-n are drivers that drive the vibration actuators 51-1 to 51-n.

As described above, according to the video-tactile sense interlocking system 10 including the tactile sense presentation signal generation device 1 of the present embodiment, while providing a tactile sense stimulus by voice input that does not feel out of place with video, the stimulus is changed to a powerful stimulus in a specific scene. In particular, the human object can be automatically extracted from the video signal, and the haptic metadata corresponding to the dynamic human object can be automatically generated in synchronization. can be linked.

Judo is a sport in which two athletes grapple and compete in techniques such as "hold down" and "throw", and the tactile presentation device 51- 1 to 51-n are presented to the user as tactile stimuli, it is possible to enhance the sense of realism and convey the game situation to the visually impaired.

Especially in judo, there are many overlaps and occlusions between athletes on the video. By enabling the tactile presentation of such action situations continuously, the sense of urgency of the game can be conveyed to visually and hearing impaired people, and the sense of realism can be enhanced.

Therefore, when the tactile sense presentation device 5 is composed of, for example, the above-described "chair-type tactile device" and is provided with, for example, two vibration actuators 51-1 and 51-2, the user watching the judo competition can say, " Vibration stimulation based on tactile presentation signals is presented simply by sitting on the chair-type tactile device. The control unit 4 obtains from the tactile sense presentation signal generation device 1 information about the movement of each person and moving objects other than the person indicating the timing and speed according to the type of impact that occurs on each of the person objects Op1 and Op2 related to the judo competition. The sound signals based on the tactile metadata including . However, the control unit 4 may be configured to drive and control only one vibration actuator, or may be configured to individually drive and control three or more vibration actuators. In addition, although not limited thereto, the control unit 4 of the present example is configured such that the vibration stimulus corresponding to the movement of the human object Op1 (athlete) in the video is generated by the vibration actuator 51-1, and the vibration stimulation corresponding to the movement of the human object Op2 (athlete) in the video is The corresponding vibration stimulus can be classified and controlled so as to be presented by the vibration actuator 51-2.

The tactile sense presentation device 5 may be of any form, such as the above-described "chair-shaped haptic device" or "cube-shaped haptic device", as long as it presents a tactile stimulus to humans. The tactile presentation device 5 of this example has a plurality of actuators that are individually driven and controlled by the control unit 4. As another example, the tactile presentation device 5 has one actuator, and the control unit 4 A plurality of tactile presentation devices 5 that are individually driven and controlled may be used. In this example, the control unit 4 and the vibration actuators 51-1 and 51-2 of the tactile presentation device 5 are wired, and the tactile presentation signal generator 1 and the control unit 4 are also wired. Although the configuration has been described as an example, a configuration in which each is connected by wireless communication or communication via a network may be used. Instead of using a vibration actuator or in addition to this, it is also possible to present vibration stimulation, electromagnetic pulse stimulation that changes according to sound signals, or temperature stimulation using a Peltier device or the like.

Therefore, the visual tactile sense interlocking system 10 of the present embodiment can continuously convey not only action events such as "throwing" but also situations such as the pushing and pulling of the players. can be conveyed, and the sense of presence can be enhanced.

As described above, according to the tactile sense presentation signal generation device 1 of one embodiment according to the present invention, an audio input method for converting an audio signal in a video content into a vibration stimulus and a tactile sense stimulus prepared in advance are presented at a specific timing. By fusing the two tactile metadata methods, it is possible to realize both realism improvement and tactile information compensation, and to generate tactile metadata related to the movement of each object, including people, at high speed and with high accuracy. can.

In particular, according to the tactile sense presentation signal generation device 1 of the embodiment of the present invention, the sound signal based on the haptic metadata generated from the video signal of the video content is synchronized with the audio signal of the video content to generate the video signal. By arranging it on the timeline, it is possible to perceive the occurrence of the event in the video with tactile information without causing a sense of deviation with respect to the video, and to improve the sense of immersion. In addition, by generating tactile metadata in real time from video content and converting it into sound signals, tactile presentation signals including sound signals and low-frequency audio signals based on haptic metadata can be generated together with video content. Once recorded, it can be easily edited for recorded programs. Tactile presentation linked to images contributes not only to information compensation, but also to an improvement in immersion, making it a service that can be enjoyed by many people, including those without disabilities.

By the way, by analyzing conventional images called Motion History Image (MHI), it is possible to determine the basic movements of a person such as "arms spread", "crouching", and "hands raised". , Since it does not measure each part of a person's joints, it is limited to recognizing large movements using the whole body. On the other hand, the skeletal trajectory feature image (JTI) according to the present invention is an image showing an image feature amount that can be said to be an improved version of this MHI. By drawing the trajectory information of the moving object, it is possible to perform highly accurate recognition while suppressing the influence of noise included in the background. In addition, since the image is created using the transition of the skeletal coordinates of each person, it is possible to recognize not only the movement of the whole body but also the partial movement of the hands and feet with high accuracy.

In particular, the skeleton detection algorithm is limited to posture estimation for each still image. By expressing with , it is possible to identify the motion by CNN. In other words, by using the skeletal trajectory feature image (JTI) according to the present invention as an input, the feature in the time axis direction, which was difficult with CNN, can be performed for highly accurate motion recognition of human motion.

Further, according to the tactile sense presentation signal generation device 1 of the embodiment of the present invention, the skeleton trajectory feature image (JTI), which can be handled as a still image showing the movement of the human skeleton as a trajectory in time series, can be easily edited. be. CNN also enables highly accurate motion recognition. In the specific example of the event detection unit 124 described above, judo video was explained as an example. It is also widely applicable to video images.

The tactile sense presentation signal generation device 1 of the above-described embodiment can be made to function as a computer, and a program for realizing each component according to the present invention in the computer is provided inside or outside the computer. stored in the memory By controlling a central processing unit (CPU) or the like provided in a computer, a program describing the processing details for realizing the function of each component is appropriately read from the memory to generate the tactile presentation signal of the present embodiment. The function of each component of the device 1 can be realized by a computer. Here, the function of each component may be realized by a part of hardware.

Although the present invention has been described with reference to specific embodiments, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the technical idea thereof. For example, in the above-described embodiments, video analysis of judo competitions has been mainly described as an example. For example, it leads to improvement of services such as broadcasting, Internet, public viewing, and entertainment using tactile information. As examples other than sports, it is also possible to apply to various uses such as application to tactile alarms in factories and security systems based on surveillance camera image analysis. Accordingly, the present invention is not limited to the examples of embodiments described above, but only by the description of the appended claims.

According to the present invention, a tactile presentation signal for driving a tactile presentation device is automatically generated in synchronization with a video signal using an audio signal based on haptic metadata automatically generated only from a video signal of video content. Since it can be generated and made editable, it is useful for linking a tactile presentation device and an image.

1 tactile presentation signal generation device 2 video content output device 3 display 4 control unit 5 tactile presentation device 10 video tactile sense interlocking system 11 low frequency audio extractor 12 haptic metadata generator 13 signal converter 14 tactile stimulus audio data database (DB)
15 synchronous synthesizing unit 41 metadata receiving unit 42 analyzing unit 43 storage unit 44-1, 44-n driving unit 51-1, 51-n vibration actuator 121 multi-frame extracting unit 122 video viewpoint estimating unit 123 object detecting unit 124 event detection Unit 1221 Viewpoint estimation unit 1222 Video motion recognition unit 1223 First metadata generation unit 1231 Moving object detection unit 1232 Object motion recognition unit 1233 Second metadata generation unit 1241 Human skeleton extraction unit 1242 Person identification unit 1243 Skeletal trajectory feature image Generation unit 1244 Human action recognition unit 1245 Third metadata generation unit

Claims (10)

For video content having a video signal and an audio signal, haptic metadata is automatically generated in real time with high precision only from the video signal of the video content, and the audio signal of the video content is used for the audio signal based on the haptic metadata. A tactile presentation signal generation device for generating a tactile presentation signal for driving a tactile presentation device in synchronization with a video signal,
a low-frequency audio extraction unit that extracts a low-frequency audio signal having a predetermined frequency or less from the audio signal;
A plurality of frame images including a current frame image and a predetermined number of past frame images are extracted from the video signal and video analysis is performed to determine the motion of the video based on the first person viewpoint, the motion of an object in the current frame image, and at least each of the a haptic metadata generator that generates haptic metadata for activating a predetermined haptic presentation device that individually indicates information about one or more of the motions of the moving object of the person;
a signal conversion unit that converts each tactile metadata into a sound signal corresponding to a predetermined tactile stimulus;
The low-frequency audio signal and the sound signal based on each of the haptic metadata are individually adjusted to a predetermined value, and synthesized in synchronization with the low-frequency audio signal to obtain a predetermined tactile sensation. a synchronous synthesizer that generates a haptic presentation signal for driving a presentation device;
A tactile sense presentation signal generation device comprising: The haptic metadata generation unit generates an optical flow image by optical flow measurement using each of the plurality of frame images in order to generate haptic metadata indicating motion of the video based on the first-person viewpoint. Based on the optical flow image, it is determined whether or not it is a first-person viewpoint video, and only when it is determined to be a first-person viewpoint video, the motion of the video based on the first-person viewpoint is estimated from the optical flow image, and the first-person viewpoint is used as a reference. 2. The tactile presentation signal generating apparatus according to claim 1, further comprising a video motion recognition unit for generating haptic metadata including at least a first-person viewpoint direction and a moving speed of the video indicating motion of the video. The haptic metadata generation unit generates a difference image between adjacent frames using each of the plurality of frame images in order to generate haptic metadata indicating movement of an object in the current frame image, and generates the difference image. Based on this, a moving object detection process is performed on the object in the current frame image, and only when the movement of the object in the current frame image is obtained, at least each object indicating the movement of the object in the current frame image is identified, and each object is identified. 3. A tactile presentation signal according to claim 1 or 2, characterized by comprising an object motion recognition unit for generating tactile metadata including size, relative velocity of each object with respect to the background, and distance between objects in the current frame image. generator. The haptic metadata generation unit generates haptic metadata indicating motion of the moving object of at least each person, for each of the plurality of frame images, based on a skeleton detection algorithm, a first skeleton of each person object. identifying a person object by generating a coordinate set, variably setting a search range based on the first skeleton coordinate set, and extracting the position and size of the skeleton of each person object and its peripheral image information; generating a second set of skeletal coordinates to which a person ID is assigned, and using the current frame image as a reference, and based on the second set of skeletal coordinates in each of the plurality of frame images; A single skeletal trajectory feature image is generated that shows only the skeletal trajectory, and a convolutional neural network that receives the skeletal trajectory feature image as an input recognizes a specific motion of the person and activates a predetermined tactile presentation device. At least each person's moving object and an event detection unit for generating tactile metadata including information about the movement of at least each person's moving object corresponding to the current frame image. 4. The tactile sense presentation signal generation device according to any one of 3 to 3. The event detection unit draws, as the skeletal trajectory feature image, a trajectory that connects the skeletal coordinates of each person in the multi-frame images, and during this drawing, the luminance decreases or increases toward the past. 5. The tactile sense presentation signal generation device according to claim 4, wherein the tactile sensation presentation signal generation device is characterized in that the tactile sensation presentation signal generation device is characterized in that the image is one image generated by drawing with the tactile sensation presentation signal. The event detection unit, as the skeletal trajectory feature image, color-codes the skeletal coordinates of each person in the plurality of frame images in common or separately for each person, and color-codes each person's skeletal coordinates. 6. The tactile sense presentation signal generation device according to claim 4, wherein the movement of the tactile sensation presentation signal is a single image generated by drawing the motion of each frame in a time-series gradation. The event detection unit limits the search range to a maximum human search range that encloses the entire human skeleton, and performs variable setting by narrowing down to a minimum specified region of the human skeleton as a focused search range, and performs state estimation. The apparatus is characterized by having means for determining the search range so as to include at least the attention search range based on the state transition estimated value of the skeleton of the person obtained by the algorithm, and performing processing for identifying the person object. 7. The haptic presentation signal generation device according to any one of claims 4 to 6. The event detection unit detects a moving object based on a difference image between adjacent frames using each of the plurality of frame images, and detects a movement direction of each of the identified human skeletons among the moving objects detected from each difference image. A moving object other than a person is selected by comparing with a skeletal trajectory feature image showing only the human body, and the moving object trajectory is connected using the coordinate position, size, and moving direction obtained from each difference image as elements for the moving object other than the person. A convolutional neural network comprising moving object detection means for generating an image, and receiving as an input a combination of said moving object trajectory image added to said skeletal trajectory feature image showing only the direction of movement of said identified human skeleton. 8. The tactile sense presentation signal generation device according to any one of claims 4 to 7, wherein the tactile sense presentation signal generation device according to any one of claims 4 to 7, wherein a specific motion of a person is recognized by: a tactile sense presentation signal generation device according to any one of claims 1 to 8;
a predetermined tactile presentation device that presents a tactile stimulus;
a control unit that refers to predetermined drive reference data based on the tactile sense presentation signal obtained from the tactile sense presentation signal generation device, and controls the predetermined tactile sense presentation device to be driven;
A video-tactile interlocking system characterized by comprising: A program for causing a computer to function as the tactile presentation signal generation device according to any one of claims 1 to 8.