KR20130067649A - Apparatus and method for reality effect detection - Google Patents

Abstract

PURPOSE: A sensory effect detection device and a method thereof are provided to easily and quickly produce 4D sensory effect media suitable for media contents by detecting motion effect through combination of detection result of a viewpoint motion detection unit, an object location detection unit and an object rotation detection unit, synchronizing the detected motion effect and combining the synchronized motion effect to media. CONSTITUTION: An object motion detection unit(210) detects motion effect of a specific object within an image. A viewpoint motion detection unit(220) detects viewpoint motion effect generated according to movement and angle variation of a camera. A motion parameter detection unit(230) detects the final motion effect using the detected object motion effect and the detected viewpoint motion effect. [Reference numerals] (211) Position detection unit; (211A) Object modeling unit; (211B) Object position detection unit; (212) Rotation detection unit; (212A) Object rotation modeling unit; (212B) Object rotation detection unit; (220) Viewpoint motion detection unit; (221) Viewpoint modeling unit; (222) Viewpoint detection unit; (230) Motion parameter detection un

Description

Sensory effect detection apparatus and method {APPARATUS AND METHOD FOR REALITY EFFECT DETECTION}

The present invention relates to an apparatus and method for detecting sensory effect, which detects sensory effect data for producing 4D sensory media content that can maximize the realism, realism, and immersion to a user, and analyzes the media content to automatically detect motion effects suitable for video content. The present invention relates to a sensory effect detecting apparatus and a method for detecting the same.

Nowadays, the production and screening of media contents with 3D or special sound effects are on the rise for realistic image reproduction on TVs and movie theaters, and more realistic image reproduction such as the release of 3D TVs that can watch 3D images at home. Attempts are being made. As part of this effort, the production and screening of 4D movies, which stimulate the five senses in conjunction with realistic effects reproduction devices such as fans, motion chairs, fragrances, and lighting devices, are on the rise in visual and auditory 3D movies.

In line with this trend, ISO / IEC MPEG-V, the International Organization for Standardization, produces light, flash, temperature, wind, vibration and sprays for producing realistic media and reproducing sensory effects. Standardization is underway to describe sensory effects such as spray, fog, color correction, and motion.

Therefore, the production of 4D immersive media that combines immersive effects with existing media contents and screening in 4D movie theaters is expected to increase, and the production and consumption of 4D immersive media contents for home 4D home theater systems is also expected.

However, until now, 4D sensational media production or 4D movie production has produced a program to determine the sensory effect elements to be added to the scene of a movie scene while the producers watch the pre-recorded videos, and to drive a sensory reproducing device according to the effect to be expressed. There are drawbacks to manufacturing by making the production complex and difficult, and the production period and costly. Therefore, it is necessary to develop a technology that is simple to manufacture and can increase the sensory effect.

Accordingly, an object of the present invention is to add sensory effect metadata to video media content and automatically detect sensory effects suitable for the characteristics of each scene when producing 4D sensory media. To provide a way.

In addition, another object of the present invention, by analyzing the existing media consisting of the image and audio signal by section to automatically detect the motion effect suitable for the image content, in particular of the viewpoint generated according to the movement of the specific object and the camera angle in the image. It aims to facilitate the creation of sensory media by analyzing the motion effect according to the change and automatically detecting vibration and motion effects.

In accordance with another aspect of the present invention, an apparatus for detecting sensory effects includes: an object motion detector for detecting a motion effect of a specific object in an image; and a viewpoint motion effect generated according to a movement and an angle change of a camera. A viewpoint motion detection unit for detecting and a motion parameter detection unit for detecting a final motion effect using the detected object motion effect and the detected viewpoint motion effect.

The present invention analyzes the video media to detect a shaking effect or a sensory effect to give a motion chair from an image signal, and combines the results detected from each part of the viewpoint detection, object position detection, and object rotation detection unit. By synchronizing the detected and detected motion effects to media in synchronism, it is easy and fast to create 4D realistic media that matches media content.

1 is a block diagram of a sensory effect detection apparatus according to an embodiment of the present invention,
2 is a block diagram of a motion effect detection unit according to an embodiment of the present invention;
3 is an overall operation flowchart of the motion effect detection unit;
4 is a conceptual diagram of finding a feature point and displaying a selection area based on photorealism;
5 is a conceptual diagram illustrating a model for extracting feature information in a region selected from FIG. 4;
6 is a flowchart of a process of extracting location information of an object according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, only parts necessary for understanding the operation according to the present invention will be described, and the description of other parts will be omitted so as not to disturb the gist of the present invention.

1 is a block diagram of a sensory effect detection apparatus according to an embodiment of the present invention.

The sensory effect detecting apparatus of FIG. 1 is a media player 110, an image display device 120, a controller 130, a motion effect detector 140, a synchronization controller 150, a sensory media fusion unit for producing 4D sensory media. It consists of 160.

An operation of each component of the sensory effect detecting apparatus will be described with reference to FIG. 1.

The media player 110 reproduces the image media, and the image display device 120 outputs the image reproduced by the media player 110 to the image display device. The controller 130 provides the motion effect detection unit 140 with the start frame and the end frame information in specific section information, for example, N frames, in which the sensory effect is to be detected. In addition, the controller 130 provides region information of a specific object, silhouette information of an object, or information for object detection and recognition in a frame having a pixel size of M × N.

The motion effect detector 140 detects a motion effect by analyzing an image signal reproduced from the media player 110. The synchronization controller 160 synchronizes the video signal reproduced by the media player 110 with the motion effect extracted by the motion effect detector 140. The sensory media fusion unit 150 receives the motion information detected by the motion effect detector 140 and converts the motion effect data into sensory effect data including the sync information by generating the XML data or converts the sensory effect data including the sync information into the image media. It converts the data into combinable form and combines the video media to produce the final 4D realistic media. The operation of the motion effect detection unit 140 will be described in detail with reference to FIGS. 2 to 6 to be described later.

2 is a block diagram of a motion effect detector according to an exemplary embodiment of the present invention.

The motion effect detector 140 of FIG. 2 is an object motion detector 210 that detects a motion effect of a specific object in an image, and a viewpoint motion detector 220 that detects a viewpoint motion effect generated according to a movement and an angle change of a camera. The motion parameter detector 230 detects the final motion effect by combining the object motion effect and the viewpoint motion effect.

In addition, the object motion detector 210 includes a position detector 211 that detects position information according to movement of an object, and a rotation detector 212 that detects rotation information according to left / right / up / down rotation of an object. The rotation detector 212 divides the object area into N sub-areas, detects motion of a feature point in each sub-area, and detects rotation information of the object. The position detector 211 includes an object modeling unit 211a and an object position detector 211b, and the rotation detector 212 includes an object rotation modeling unit 212a and an object rotation detector 212b.

In addition, the viewpoint motion detector 220 includes a viewpoint modeling unit 221 and a viewpont detector 222. An operation process of the motion effect detection unit 140 will be described in detail with reference to FIGS. 3 to 5 to be described later.

3 is an overall operation flowchart of the motion effect detection unit. 4 is a conceptual diagram of finding a feature point and displaying a selection area based on the real picture. FIG. 5 is a conceptual diagram illustrating modeling for extracting feature information in a region selected in FIG. 4.

3 is an overall operation flowchart of the motion effect detection unit.

Parameters of the motion effect for controlling the movement of the motion chair can be configured as an object moving angle, an object moving distance, an object left and right rotation angle, and an object up and down rotation angle as shown in Table 1 below.

frame n Object
Moving angle Object
Moving distance Object
Rotation angle Object
Rotation angle

In operation 301, the object modeling unit 211a of the location detector 211 provides modeling of an object whose location is to be tracked and includes an object provided by the controller 130 for the first frame of a specific section of the image. Alternatively, the object to be tracked is modeled in a method such as color histogram, intensity, and object edge detection in the region corresponding to the silhouette of the object.

In step 302, the object position detector 211b detects the position information of the object for each frame based on the object model information. At this time, the position information is obtained by the average value of the x coordinate and the average value of the y coordinate with respect to the x-y coordinates of all pixels in the object. When the position coordinates of the object are detected in the current frame, modeling of the object in the current frame can be re-executed and the object model can be updated to be used as a model for extracting object position information in the next frame.

In step 303, the object rotation modeling unit 212a of the rotation detection unit 212 is configured to detect rotation information of the object in order to detect left / right / up / down rotation information of the object itself independently of the global movement of the object in the image area. Model the feature information of the required object. Referring to FIG. 5, the object area is divided into N sub-areas, and one or more feature points for detecting rotation information are detected in each sub-area, M totals are detected in the object area, and each specific point is labeled and stored in the memory. . In this case, the feature point of the object may use edge information by edge detect or an eigenvalue of an nxn pixel matrix.

In step 304, the object rotation detector 212b detects the position of the feature point labeled in each sub-area every frame and stores the detected position information in the memory. The rotation information first calculates a moving angle for each of the labeled feature points based on the position information of the current frame and the position information of the past frame. The total rotation angle of the object can be obtained as an average of the moving angles of each feature point as shown in Equation 1 below. here

는 각 특징점의 이동각을 의미한다.Means the total rotation angle of the object

Denotes a moving angle of each feature point.

In step 305, the viewpoint modeling unit 221 divides the entire frame except the object region into N subregions and detects feature points for viewpoint detection in each subregion similarly to the rotation detector 212. The detected feature points are labeled and used for later feature point detection and viewpoint detection.

In step 306, the vewpoint detector 222 detects the position of each feature point every frame and detects the rotation angle, the movement angle, and the movement distance by comparing the viewpoint information of the past frame.

In step 307, the object position information is detected using the object position information detected in steps 301 and 302 and viewpoint information detected in steps 305 and 306, and in step 308, the object position information is detected in steps 304 and 305. The rotation information of the object is detected using the rotation information of the object and viewpoint information detected in steps 305 and 306.

The motion parameter detector 230 detects the final motion effect based on the information of the viewpoint detector 220 and the information of the object motion detector 210. For example, if the information detected from the viewpoint is rotated 10 degrees in the left direction and the rotation information of the object detected by the object motion detector is rotated 5 degrees in the left direction, the motion effect detector determines the left and right rotation angles of the object by 15 degrees. As another example, if the detection result of the viewpoint detector is 10 pixels in the left direction and the object position detection result is in the moving direction to the right and the moving distance is 30 pixels, the final motion effect is 20 pixels to the right in the moving direction.

6 is a flowchart illustrating a process of extracting location information of an object according to an embodiment of the present invention.

In step 610, the image to be analyzed is divided into N frames. In step 620, the feature points of each region divided in step 610 are extracted. In step 630, the positions of the feature points of each region extracted in step 620 are detected. In step 640, the left / right / up / down rotation angle is detected using the position of the feature point detected in step 630. In step 650, it is checked whether the currently extracted frame is the last frame. If it is the last frame, the process proceeds to step 660 to finish the extraction of the object rotation information. If it is not the last frame, the process proceeds to step 620.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention. Therefore, the scope of the present invention should not be limited by the described embodiments, but should be determined by the scope of the appended claims, as well as the appended claims.

Claims (1)

In the sensory effect detection device,
An object motion detector configured to detect a motion effect of a specific object in the image;
A viewpoint motion detector configured to detect a viewpoint motion effect generated according to the movement and angle change of the camera;
And a motion parameter detector configured to detect a final motion effect by using the detected object motion effect and the detected viewpoint motion effect.

Images