KR101111046B1 - A Similar Video Search System through Object Detection Information and A Method thereof - Google Patents

Abstract

The present invention provides a video retrieval system, WPF-based video editing program 600 capable of video editing and thumbnail extraction, object detection and object information calculation program 700 for calculating the object information by detecting the object using the extracted thumbnail Client 300 having a video, video and video, object information is uploaded and stored in the video information database 400 and the object information database 500 storage server 200, the object information and video information from the client 300 Similar to using the object detection information comprising a Web & Streaming server 100 for creating and showing the same list of images compared to the information stored in the video information database 400 and the object information database 500 A video search system and method.
Similar video search system and method using the object detection information according to the present invention, since the video producer can check whether there is a video and the video produced by his own can produce a unique video, provide a video service By reducing the number of similar images, companies can reduce the server's memory inefficiency and make faster and more accurate searches than when users search for images.

Description

A similar video search system through object detection information and a method

The present invention is a system for retrieving a video similar to a video created and edited by a user in a video server, by detecting a specific object in a video thumbnail and using object information such as an outline, a distance between each feature point, and an area of the detected object. The present invention relates to a system and method for searching for similar images using detailed information of video and object information of thumbnails so that users can create their own original images by finding similarities with images stored in the same.

Recently, due to the activation of UCC video, the number of videos created or edited by users is rapidly increasing, so the probability of having a video similar to the user's video on the server increases.

In order for users to create their own original videos, it is necessary to find a video similar to the video produced by them. However, the current video search system has a limitation in finding similar images because it uses search terms.

On the other hand, Patent Application No. 2009-35036 No. 2009-35036 in the similar video search system, characterized in that the standardized processing of at least one or more reference video data and comparison video data, the feature of a plurality of scenes from the reference video data comparison video data A data extraction server for extracting feature data for each feature; a database server for storing reference feature data for each reference video extracted from the data extraction server; and comparison feature data extracted from the data extraction server and the reference feature data Although a similar video search system including a similar search server for measuring the degree of similarity has been published, there has been a problem of slowness in extracting and searching feature data.

Accordingly, the present invention has been made to solve the above limitations, and in searching for similar videos, a specific video of thumbnails is detected to search for similar videos more accurately by using the information of the detected objects, detailed information of the videos, and search terms. An object of the present invention is to provide a similar video search system and method using object detection information.

In the video search system according to the present invention, the similar video search system using object detection information detects a feature object of a video thumbnail obtained by editing a video in a client in which a video editing program capable of video editing and encoding is installed.
The client transmits the result value obtained by calculating the area of the object and the mutual distance between the feature points using the detected object outline and the feature point to the video server together with the video information.
The server extracts images similar to the video edited by the user by comparing the object information and the video information of another video stored in the database of the server using the information transmitted from the client. The extracted similar images calculate the similarity probability of the video edited by the user and provide the user with a list of the similar videos so that they can make their own unique video.
Similar video search system using the object detection information of the present invention is a video editing program for editing a user-designated video, extracting video information and thumbnails, an object detection program for detecting feature objects using the output thumbnails to calculate the object information , A web site where upload and search of similar videos can be performed, a storage server that receives edited video and object information from the client, a web & streaming server that stores video, objects, and user information, video position, metadata, object information and And an object information database storing information of feature objects of a video thumbnail and a video information database storing detailed information of the same video.

delete

delete

delete

Similar video search system and method using the object detection information according to the present invention, since the video producer can check whether there is a video and the video produced by his own can produce a unique video, provide a video service By reducing the number of similar images, companies can reduce the server's memory inefficiency and make faster and more accurate searches than when users search for images.

1 is a block diagram of a similar video search system using object detection information according to an embodiment of the present invention.
2 is a flowchart illustrating object detection and object information calculation according to an embodiment of the present invention.
3 is a flowchart of a similar video search according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

1 is a block diagram of a similar video search system using object detection information according to an embodiment of the present invention. As shown in FIG. 1, a similar video search system using object detection information according to an embodiment of the present invention is shown in FIG. , WPF-based video editing program 600 capable of video editing, video information extraction and thumbnail extraction in video retrieval system, object detection and object information calculation program 700 that calculates object information by detecting objects using extracted thumbnails (700) Client 300 having a video, video and video information, object information is uploaded, video and video information is stored in the video information database, object information is stored in the object information database, the storage server 200, the object from the client 300 The video information database 400 and the object information database 50 It comprises a Web & Streaming server 100 to create and show a list of the same image compared to the detailed video information and the feature object information of the video thumbnail respectively stored in 0).
The client 300 detects a feature object by using the object detection program 700 to generate a thumbnail generated by editing the video editing program 600, calculates information of the object, and transmits the information to the Web & Streaming server 100. The similar object is searched through the similar object inspection process, and the video and object information are uploaded to the storage server 200.
The storage server 200 transmits the video transmitted from the client 300 to the Web & Streaming server 100 and stores the video information and the object information in the video information database 400 and the object information database 500.
The Web & Streaming server 100 compares the data in the video information database 400 and the object information database 500 by using the object information transmitted from the client 300, and calculates the similarity and then lists the images having high similarity. To show the client 300.
The video information database 400 receives and stores the total time, thumbnail, position of the thumbnail, and media item source number of the video transmitted from the storage server 200.
The object information database 500 receives and stores the area of the feature object, the mutual distance between the feature points, the outline, and the thumbnail object information of the pixel of the object transmitted from the storage server 200.

2 is a flowchart illustrating object detection and object information calculation according to an embodiment of the present invention. As shown in Figure 2, the order of calculating the object detection and object information according to an embodiment of the present invention
An average contrast extraction step 811 of setting a candidate region using a higher contrast of eye, nose, and mouth in a thumbnail extracted from a WPF-based video editing program capable of video editing, video information extraction, and thumbnail extraction; Smoothing the histogram for increasing the average contrast (step 812); Detecting a primary image outline using a Sobel algorithm (813); Detecting a secondary image outline using a Laplacian algorithm (814); Detecting (815) a tertiary object outline using a Canny algorithm; Face recognition using the Haar Wavelet (821); Identifying a face 822; Extracting facial feature points using Delaunay triangulation (823); Measuring mutual distance between feature points using Delaunay triangulation (824); Calculating an area within the face outline and a pixel using the third face outline (825); Detecting a local feature within the outline (831); Identifying 832 local features; Measuring 833 mutual distances between regional features; Calculating an area within the object outline and a pixel using the tertiary object outline (834); Designating an object rectangular region following the x and y coordinates of the end point of the object using a cubic outline (841); Extracting pixel information of the boundary portion of the object outline 842; Calculating 843 an area within the object outline; Checking whether there are other feature objects after the analysis of the primary object information detected by the face detector, the local feature detector, and the object Pixel detector (851); If there is no feature object, step 852 temporarily stores information of all objects.

For reference, the object detection and object information calculation program 700 uses a higher contrast of eyes, nose, and mouth in thumbnails extracted from the video editing program 600 based on WPF, which is capable of video editing and thumbnail extraction. Extract the average contrast setting the candidate area, smooth the histogram to increase the average contrast, detect the primary image outline using the Sobel algorithm, detect the secondary image outline using the Laplacian algorithm, and use the Canny algorithm An outline detector 810 for detecting a tertiary object outline; Recognize faces using Haar Wavelet, identify faces, extract facial feature points using Delaunay triangulation, measure mutual distance between feature points using Delaunay triangulation, and use third-order face outlines within face outlines A face detector 820 for calculating an area and a pixel; Object feature detection unit that detects local features in the outline using the image identification area as an object, checks local features, measures the mutual distance between local features, and calculates the area and pixel in the object outline using the tertiary object outline 830; Object pixel detection unit 840 that designates the rectangular area of the object by connecting the global descriptor to the x, y coordinates of the end point of the object using a cubic outline, extracts pixel information of the boundary of the object outline, and calculates an area within the object outline. ; After detecting the first object information detected by the face detector, the local feature detector, and the object pixel detector, the object detection information storage unit 850 checks whether there are other feature objects and if there is no feature object, temporarily stores information of all objects. It is made to include.
The outline detector 810 sets the average contrast level and smooths the histogram smoothly through the average contrast level extraction 811 using a raster intensity contrast test method to catch the feature point candidate areas of the face such as eyes, nose and mouth. . The edge of the primary image is detected using the Sobel algorithm (812), and the edge of the secondary image is detected using the Laplacian algorithm. Finally, the tertiary outline is detected using the Canny algorithm.
The face detector 820 recognizes a face using the Haar Wavelet (821), when the face is identified (822), sets the face area as the ROI and extracts the facial feature point (823) using Delaunay triangulation. The distance between them is measured 824, and the area and pixel information in the outline of the face are measured.
The object local feature detection unit 830 detects a local feature using a Harris coner detection algorithm in the object outline (831) and checks whether the local feature exists (832) and generates an LFH if the local feature exists. The mutual distance is measured 833, and the area and pixel information in the object outline are measured 834.
The object pixel detection unit 830 designates an object rectangular region using x, y coordinates of an end point of an object using a distribution map of an edge type (831), and extracts pixel information of a boundary portion of an object outline (832). The area in the object outline is calculated (833).
The object detection information storage unit 840 checks whether there are other feature objects after the analysis of the primary object information detected by the face detector, the object region feature detector, and the object pixel detector 841, and if there are no feature objects, The information is temporarily stored 842 and if there are other objects, the apparatus returns to the face detector 820 again to detect the primary objects.
In other words, the similar video search system using the object detection information according to the present invention, the WPF-based video editing program 600 capable of video editing, video information extraction and thumbnail extraction, the object by using the extracted thumbnail Client 300 having an object detection and object information calculation program 700 that calculates object information, uploads video and video information, object information, and video and video information to video information database, and object information to object information database. After receiving the object information and the video information from the storage server 200 and the client 300 to store the similar object inspection 900 to compare with the information stored in the video information database 400 and the object information database 500 It includes a Web & Streaming server 100 to create and show the same video list (1000),
The object detection and object information calculation program 700 selects a candidate area using the higher contrast of eyes, nose, and mouth in the thumbnail extracted from the WPF-based video editing program 600 capable of video editing and thumbnail extraction. Extract the average contrast setting, smooth the histogram that increases the average contrast, detect the primary image outline using the Sobel algorithm, detect the secondary image outline using the Laplacian algorithm, and use the Canny algorithm to determine the tertiary object. Recognize the face using the outline detection unit 810, Haar Wavelet for detecting the outline, identify the face, extract the facial feature points using Delaunay triangulation, measure the mutual distance between the feature points using Delaunay triangulation, Face detector 820, which calculates the area and pixel in the face outline using the third face outline, in the outline Detects local features, checks local features, measures mutual distances between local features, and uses object tertiary feature detector 830 and tertiary outlines to calculate area and pixel within object outline using tertiary object outlines. The object's rectangular area by connecting the x and y coordinates of the object's end point, extracting the pixel information of the boundary of the object's outline, and calculating the area within the object's outline, the face detector, the feature detector, After the analysis of the primary object information detected by the object pixel detection unit, the object detection information storage unit 850 for checking whether there is any other feature object, and if there is no feature object temporarily stores the information of all objects, characterized in that it comprises a .

3 is a flowchart of a similar video search according to an embodiment of the present invention. As shown in FIG. 3, a similar video search according to an embodiment of the present invention may include: searching 901 using a category of data stored in a video information database 400; Searching for using the name of the video stored in the video information database 400 (910); Searching (920) using a media position stored in the video information database 400; Searching 930 using a media position of a thumbnail stored in the video information database 400; searching 940 using information of an object stored in the object information database 500; Calculating a likelihood probability using a probability estimation method in the classified image (941); Retrieving only the first thumbnail object information among the object information stored in the object information database 500 (950); A media item source check step 951 of checking if there is a media item source added at the client; Extracting by increasing the number of thumbnails of the edited image (953); Detecting the object using the newly extracted thumbnail and calculating information 960; And calculating a probability of whether there is no similar image and transmitting a message to the client 300 (961).

In more detail, the index word is generated by using the part-of-speech tag, the old chunker, and the P-norm model from the video keyword transmitted from the client 300, and the index word is compared with the data of the video information database 400 and edited. In the category search step 901 for classifying a video including the same keyword as the video, an index word is generated using a chart parsing algorithm, a part-of-speech tag, an old chunker, and a P-norm model from the video title transmitted from the client 300. And a similar video name checking step 910 for classifying a video including the same title as the edited video by comparing the index word with the data of the video information database 400, the video playing time transmitted from the client 300, Compare the scene with the edited video by comparing the scene change point with the data 400 of the movie information database. The media position check step 920 of classifying an image such as a play time and a scene change point, and compares the media position of the thumbnail transmitted from the client 300 with the data of the moving image information database 400 to be the same as the edited movie. In a thumbnail position inspection step 930 of classifying a video having a thumbnail media position, the object information transmitted from the client 300 is compared with the data of the object information database 500 and has the same thumbnail object as the edited video. In the image classified in the object information retrieval step 940, the object information retrieval step 940, and the additional object information inspection step 960, a video similar to the image edited using Bayes estimation probability estimation method is classified. A similar image probability calculation step 941 for listing the classified images, to the client 300 An initial thumbnail inspection step 950 of classifying a video having the first thumbnail object identical to the edited video by comparing the object information of the first thumbnail with the data of the object information database 500 among the transmitted object information, and the client 300 A media item source checking step 951 for classifying a video having the same edited video and the same media item source by comparing the media item source transmitted from the data with the data of the video information database 400, and the media item source checking step 951 The image thumbnail extraction step 952 of re-extracting thumbnails by transmitting the scene change points of the images classified in the step C) to all the scene change points as thumbnail positions, and re-extracting the thumbnails of the thumbnails re-extracted by the client 300. Detects an object and detects the object's area, distance between feature points, outline, P Additional object information detection step 953 for storing object information such as ixel as an XML file and retransmitting it to the Web & Streaming server 100, and storing the object information retransmitted by the client 300 in the object information database 500. Additional object information checking step (960) and object information retrieving step (940) for classifying a video having the same thumbnail object as the edited video by comparing the thumbnail object information of the classified video in the media item source checking step (951). If the image is not classified in the additional object information checking step 960, a step of calculating the probability of whether there is no similar image for transmitting a message that there is no similar image to the client 300 is characterized in that it comprises a step (961).

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

The configuration and operation as described above are not limited to the claims of the present invention as an embodiment, and various changes and modifications are possible within the scope of not changing the technical spirit of the present invention. It is obvious to those who are engaged.

100: Web & Streaming Server 200: Storage Server
300: Client 400: Video Information Database
500: object information database 600: WPF-based video editing program
700: object detection and object information calculation program

Claims (3)

In the video search system,
Client with WPF-based video editing program 600 capable of video editing, video information extraction and thumbnail extraction, object detection and object information calculation program 700 that calculates object information by detecting objects using extracted thumbnails ( 300), video and video information and object information are uploaded and the video and video information is stored in the video information database, the object information is stored in the object information database from the storage server 200, the client 300 receives the object information and video information It includes a Web & Streaming server 100 to create and show the same image list 1000 through the process of similar object inspection 900 to compare the information stored in the video information database 400 and the object information database 500, ,
The object detection and object information calculation program 700 selects a candidate area using the higher contrast of eyes, nose, and mouth in the thumbnail extracted from the WPF-based video editing program 600 capable of video editing and thumbnail extraction. Extract the average contrast setting, smooth the histogram that increases the average contrast, detect the primary image outline using the Sobel algorithm, detect the secondary image outline using the Laplacian algorithm, and use the Canny algorithm to determine the tertiary object. Recognize the face using the outline detection unit 810, Haar Wavelet for detecting the outline, identify the face, extract the facial feature points using Delaunay triangulation, measure the mutual distance between the feature points using Delaunay triangulation, Face detector 820, which calculates the area and pixel in the face outline using the third face outline, in the outline Detects local features, checks local features, measures mutual distances between local features, and uses object tertiary feature detector 830 and tertiary outlines to calculate area and pixel within object outline using tertiary object outlines. The object's rectangular area by connecting the x and y coordinates of the object's end point, extracting the pixel information of the boundary of the object's outline, and calculating the area within the object's outline, the face detector, the feature detector, After the analysis of the primary object information detected by the object pixel detection unit is finished, the object detection information storage unit 850 for checking whether there is any other feature object, and if there is no feature object temporarily stores the information of all objects, characterized in that it comprises a Similar Video Retrieval System Using Object Detection Information
delete WPF-based video editing program 600 capable of video editing and thumbnail extraction, outline detection unit 810, face detection unit 820, object region feature detection unit 830, object pixel detection unit 840, object detection information storage unit ( The client 300 having the object detection and object information calculation program 700 for detecting the object and calculating the object information using the extracted thumbnail including the 850, uploads the video and the video, the object information, and receives the video information database ( 400 and object information and video information received from the storage server 200 and the client 300 stored in the object information database 500 and compared with the information stored in the video information database 400 and the object information database 500 In the method using a video search system including a Web & Streaming server 100 to create and show the same video list,
From the video keyword transmitted from the client 300, an index word is generated by using a part-of-speech tag, an old chunker, and a P-norm model, and the index word is compared with the data of the video information database 400 to include the same keyword as the edited video. In the category search step 901 for classifying a moving image, an index term is generated by using a chart parsing algorithm, a part-of-speech tag, an old chunker, and a P-norm model from the moving image title transmitted from the client 300, and the index term is used as the video information. Similar video name checking step 910 for classifying a video including the same title as the edited video by comparing the data of the database 400, the video playing time and the scene change point transmitted from the client 300 to the video information. Same playback time and transition time as the edited image compared to the data 400 of the database Media position check step (920) for classifying an image, such as a video having the same thumbnail media position as the video edited by comparing the media position of the thumbnail transmitted from the client 300 with the data of the video information database 400 Thumbnail position inspection step 930 of classifying the object, the object information sent from the client 300 is compared with the data of the object information database 500, the object information search to classify the video having the same thumbnail object as the edited video In the image classified in the step 940, the object information retrieval step 940, and the additional object information inspection step 960, a similar image is classified using the Bayes estimation probability estimation method and the classified image is listed. Similar image probability calculation step 941, the first of the object information transmitted from the client 300 Initial thumbnail inspection step 950 for classifying a video having the first thumbnail object equal to the edited video by comparing the object information of the thumbnail with data of the object information database 500, the media item source transmitted from the client 300 Scenes of the images classified in the media item source inspecting step 951 and the media item source inspecting step 951 to classify the video having the same edited video and the same media item source by comparing the data of the video information database 400 with the data. Image thumbnail extraction step 952 of re-extracting thumbnails by transmitting the transition point to the client 300 as all the scene transition points as thumbnail positions, and detecting the object of the thumbnail re-extracted by the client 300 to obtain an area of the object. XML file for object information such as distance between each feature point, outline, Pixel, etc. Additional object information detection step 953 storing and retransmitting to Web & Streaming server 100, and media item source checking step 951 storing object information retransmitted by the client 300 in object information database 500 An additional object information checking step 960, an object information searching step 940, and an additional object information checking step 960 for classifying a video having the same thumbnail object as the edited video by comparing the thumbnail object information of the classified images in And calculating a probability (961) of whether there is no similar image, which transmits a message indicating that there is no similar image to the client 300 when the image is not classified, in the similar video search method using the object detection information.

Images