KR20130122262A - System for searching video and server for analysing video - Google Patents

Abstract

The present invention relates to a video retrieval system and a video analysis server and, more particularly, to a video analysis server for analyzing a video, storing a subject in the video and features of the subject, and managing the subject and features, and a video retrieval system for retrieving a video using analysis data of the video analysis server. The video analysis server according to an embodiment of the present invention comprises: a subject detection module for searching a subject by analyzing video data and extracting a subject recognition video of which the background is removed from the video data; a feature extraction module for extracting feature data of the subject by analyzing the subject recognition video, generating subject feature data by converting the feature data into a text form or a binary form, and obtaining reproduction time of the video data having the subject therein as subject retrieval time; a storing unit for storing the subject retrieval time and the subject feature data as video analysis data; and a control unit for operating the subject detection module and the feature extraction module. [Reference numerals] (1) Image analyzing data;(AA) Image analyzing;(BB) Searching server;(CC) Searching client;(DD) Warning server (control)

Description

Image Search System and Video Analysis Server {SYSTEM FOR SEARCHING VIDEO AND SERVER FOR ANALYSING VIDEO}

The present invention relates to an image retrieval system and an image analysis server. More specifically, an image retrieval system using an image analysis server and an image analysis server for storing and managing objects and attributes of an image by analyzing the image and retrieving the image using the image analysis server Relates to a search system.

In order to prevent crimes and arrest criminals, CCTVs are used to photograph and record the entrances and boundary areas of facilities. The captured video and recorded video are used for criminal investigations, and the recorded video is inconvenient to be checked by a person, and thus, the video is analyzed more efficiently.

As the risk of terrorism and crime has increased, the demand for surveillance systems that can simultaneously monitor multiple locations with fewer people has increased. Traditional surveillance systems simply record video on behalf of the human eye. The advent of this surveillance system has made it possible for a small number of people to monitor large areas.

In this regard, the prior art is an NVR system including an event function and a control method according thereto (Patent Application No. 10-2010-0018113).

1 relates to a conventional NVR system. Referring to FIG. 1, the conventional NVR system 10 includes a receiver 12, a controller 14, a communication unit 16, and a storage unit 18.

The receiver 12 receives image information of the surveillance zone from a plurality of network cameras installed in the surveillance zone to be monitored by the user.

The controller 14 determines whether an event is generated from the received image information, and generates an event generation signal when determining the event occurrence.

The communication unit 16 transmits the generated event generation signal to an external terminal using a network transmission or a wireless LAN.

The storage unit 18 stores the received image information.

However, the conventional system is only to determine whether the event occurs through the monitoring camera and informs, there was a problem that can not detect people or vehicles.

An object of the present invention is to provide an image analysis server for searching for subjects and abnormal behavior from the image data.

Another object of the present invention is to provide an image analysis server for searching for a subject area and an abnormal behavior area from image data.

In addition, an object of the present invention is to obtain the feature data of the subject and the search time of the subject from the image data.

In addition, an object of the present invention is to search for an object in the analysis data by using the object and the feature of the object to be searched, and to search for the movement of the object, moving lines, and the like.

According to another embodiment of the present invention, an image analysis server may include: an object extraction module configured to analyze image data, search for a subject, and extract an object recognition image from which a background is removed from the image data; Extracts feature data of the subject by analyzing the object recognition image, converts the feature data into a text or binary format, generates object feature data, and obtains a reproduction time of the image data in which the object is found as an object search time; Extraction module; A storage unit which stores the object search time and the object feature data together as image analysis data; And a controller for driving the object detection module and the feature extraction module.

If the object recognition image is a person, the feature extraction module extracts feature data of the person, converts the feature data into a text or binary format, generates object feature data, and displays a reproduction time of the image data found by the object. The apparatus may further include a feature extraction module that is acquired at an object search time, wherein the feature extraction module generates feature data of a face using a Gabor wavelet.

The feature extraction module extracts license plate information and feature data of the vehicle when the object recognition image is a car, converts the license plate information and feature data into a text or binary format, and generates object feature data, and retrieves the object. The apparatus may further include a vehicle feature extraction module configured to obtain a reproduction time of the image data as an object search time.

The vehicle extraction module may extract a license plate and feature data of a vehicle through a preprocessing process, gray image conversion, SCW-based binarization, license plate verification, and post-processing.

The object detecting module may extract a color of the subject and re-analyze the image data using the color of the subject, thereby preventing a detection error of the subject.

The feature extraction module may generate feature data of an object by fusion of all feature data of a subject recognized as the same person.

The person feature extraction module may determine whether the subject is the same person by using a face clustering result.

The image retrieval system according to an embodiment of the present invention receives a retrieval request from an image server and a user terminal storing an image captured by an image analysis server and a plurality of imaging devices as an original image, and analyzes the retrieval request. The image search server may be configured to determine a condition, search for analysis data or an image corresponding to the search condition, and transmit a search completion message to a user terminal.

The image search server may search for analysis data corresponding to the search condition, and search for an image by using a time stamp value of the searched analysis data.

When the search condition is an image or an image, an object and a search condition to be searched may be extracted using an object detection module and a feature extraction module.

The present invention can provide an image analysis server for searching for a subject and an abnormal behavior from the image data.

In addition, the present invention may provide an image analysis server for searching for a subject region and an abnormal behavior region from the image data.

In addition, the present invention may provide an image analysis server for acquiring the feature data of the subject and the search time of the subject from the image data.

In addition, the present invention is to search for an object in the image data using only the object and the feature of the object to be searched, and to search for the movement of the object, moving lines, and the like.

1 is a block diagram showing a conventional search system.
2 illustrates an image retrieval system according to an embodiment of the present invention.
3 is a block diagram of another image retrieval system according to an embodiment of the present invention.
4 is a flowchart illustrating a video retrieval method according to an embodiment of the present invention.
5 is a block diagram of an image analysis server according to an embodiment of the present invention.
6 is a block diagram of a feature extraction module of an image analysis server according to an embodiment of the present invention.
7 is a flowchart illustrating a method of analyzing an image of an image analysis server according to an embodiment of the present invention.
8 illustrates a facial feature extraction method according to an embodiment of the present invention.

Brief Description of the Drawings The advantages and features of the present invention, and how to achieve them, will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Meanwhile, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

2 illustrates an image retrieval system according to an embodiment of the present invention.

As shown in FIG. 2, metadata (feature information) output of the image analysis server is transmitted to the search server, and the search server stores the data in the analysis information DB. The user instructs the search using a specific image or text in the search client. In this case, when the request of the search client is an image, the same operation as the image analysis performed by the image analysis server is performed to extract feature information. Deliver the feature information to the search server. The search server searches for an image or video from the analysis information DB using the received text (time stamp, that is, time, or specific color, car number, etc.) or feature information.

At this time, if the request of search client is text, it is the same as general DB search.

If the request of the search client is the feature information, since the feature information is binary data, the corresponding image or image is searched by searching for matching feature information using a separate method of matching the feature information.

In addition, by analyzing the image in the image analysis server, it is possible to detect the event. Event detection information can be delivered to a separate monitoring (warning) server, and can be notified to a video surveillance person or security company.

3 is a block diagram of an image retrieval system according to an embodiment of the present invention.

As illustrated in FIG. 3, the image search system 100 may include an image analysis server 110, an image server 120, an image search server 130, and user terminals 140.

The search request may be input through the user terminal 140. The search request may be divided into attributes corresponding to the object to be searched and characteristics of the object, and may include an image. The search target may be a creature such as a person or an animal and an inanimate object such as a car.

The user terminal 140 may input an image or text to be searched by the user. In addition, the user terminal may be equipped with a camera, or may include a separate storage unit for storing the image to be searched.

The user terminal 140 may be connected to an input device such as a mouse, a USB port, or a keyboard.

If the user inputs an image and a video to be searched for, the feature information about the object (person, car, etc.) and the characteristic (gender, color, model, height, head shape, etc.) to search from the image and image It can be extracted, and in the case of a human face, it can be evicted in the same manner as in FIG.

The user terminal 140, the image analysis server 110, the image search server 130, and the image server 120 may communicate by using a wired or wireless communication method.

The image server 120 may receive an image captured in real time from a remotely installed imager, or may receive a recorded image. In addition, the image server 120 may store and manage images together with identification information for identifying an image pickup device.

The image server 120 may store the images in the photographed time order, and store the images according to the image capturing apparatus.

Here, the imaging device may be a CCTV camera, a smart camera, an IP camera, a DVR, an NVR. The image server and the image pickup device may communicate using a wired or wireless communication method.

When the image search server 130 receives the user's search request, the image search server 130 may analyze the search request and convert the search request into a search condition. In addition, the image search server 130 may search for a corresponding image by using a search condition.

When the image search server 130 searches for an image by using a search condition, the image search server 130 may approach the image analysis server 110 and search by dividing the object into attributes and attributes.

The image analysis server 110 may analyze the image input from the image server 120, and store and manage analysis data (feature information data) configured of a target and a property in a text format and a binary format. The image analysis server 110 may not generate an object in the image or generate no data for the image in which there is no change.

The analysis data, which is a result of analyzing the image, may be configured in a text format and a binary format, and may include an object, an attribute, a time stamp, an identification number of the image pickup apparatus, and the like.

The image analysis server 110 may classify and store analysis data according to an object, and the analysis data according to an object may be sorted based on attributes. In addition, the object analyzed by the image analysis server 110 and features of the object may be stored as text or binary data.

The search result of the image search server 130 may be a time stamp list. Then, the image search server 130 may search the image list from the image server 120 using the time stamp list.

The image search server 130 does not search while analyzing the image at the time of search, but searches using the result data (feature information data) pre-analyzed by the image analysis server 110. Get search results faster and faster. The analyzed result data is stored in the image analysis server managed by the image search server 130.

4 is a diagram illustrating a search method according to an embodiment of the present invention.

As shown in FIG. 4, in the search method, a search request is transmitted from a user terminal to a search server (S420), the search server analyzes the search request to obtain a search condition (S430), and the search server is connected to the search condition. The method may include searching for corresponding image analysis data (S440), and searching for an image through time stamps of the image analysis data (S450).

In S420, the image search server receives a search request from a user terminal using a wired or wireless communication method.

In operation S430, the image search server analyzes the search request to determine a search condition.

The image search server may evict the search subject and the search condition by analyzing the search request received from the user terminal. Here, the search request may be an image, an image, a text, a keyword.

If the search request is an image or an image, the object of the search and the condition of the search may be extracted as in the analysis method of the image analysis server of FIG. 7.

In S440, the search server may search the image analysis server for image information and image information corresponding to the search target and the search condition.

In the case of using a search condition that is text, the search can be performed using the object of the search and the features of the search. At this time, the characteristics of the search may also vary according to the search target.

For example, when the search target is a human, the search condition may be a feature of a face, a gender, a shape of an eye nose, a color of clothes, and the like.

In addition, when the search target is a vehicle, the license plate may be used as an identifier, or the model, the color of the vehicle, or the like may be used as the search condition.

The image analysis server may analyze the image and store the object, the feature of the object, a time stamp, and an identification number of the image pickup device together with the image. In addition, the search result of the image search server may include a time stamp, an identification number of the image capturing apparatus, and the like as the format of the image analysis data. The original image may be obtained from the image server using the retrieved time stamp value.

In addition, an identification number of the imaging apparatus is obtained through the image analysis data, and the identification number makes it possible to know the photographed position.

Therefore, the retrieval system and operation procedure according to an embodiment of the present invention can be used for the occurrence of a crime and the suspect arrest for the crime.

The search result of the image may be provided by a thumbnail view, a list view, a statistical report view, or the like. In addition, the image search system may provide a combination search utilizing a profile function.

5 is a block diagram of an image analysis server 110 according to an embodiment of the present invention.

As illustrated in FIG. 5, an image analysis server according to an exemplary embodiment may include an object detection module 510, a feature extraction module 520, a storage unit 530, and a controller 540. In addition, the image analysis server 110 may further include a communication unit and a display unit.

The object detection module 510 analyzes the input image data to search for a subject and extracts an object recognition image from which the background is removed from the image data.

The input image data may be image data captured by a CCTV camera, an IP camera, a smart camera, or a DVR.

The object detection module 510 extracts only the object region image in which the subject is located by removing the background from the image data. In this case, the object detection module may use a technique of using a difference image, a technique of analyzing a motion, a meanshift and an application technique, and a local binary pattern technique.

If the subject is a person, the object detection module scans the entire area of the image data and finds candidate areas that may be human faces or vehicles. The object detection module then sequentially passes through the classifiers connected in series to remove the non-human face area in the middle. After performing the above steps repeatedly, the remaining areas may be determined as the human face.

The object detection module 510 may perform a face verification process using skin color information in order to prevent an error of incorrectly discriminating an area not corresponding to a face as a face.

The feature extraction module 520 analyzes the object recognition image to extract feature data of the subject, converts the feature data into a text format or a binary format, generates object feature data, and retrieves a playback time of the image data from which the subject is found. Obtained in time.

The object feature data may consist of the object's target and attribute values.

If the object of the object is a person, the attribute value may be the color of the skin, the shape of the eyes, the nose, the mouth, the height, the gender, the color of the clothes, the age, the body shape, and the like. If the object of the object is a vehicle, the attribute value may be a license plate, a color of the vehicle, a vehicle model, and the like.

In order to extract the feature data of the person, the feature extraction module 520 may first normalize the object region image based on both eye positions to extract the feature data of the person.

The feature extraction module 520 will be described with reference to FIG. 6.

The storage unit 530 bundles object search time and object feature data into image analysis data to store and manage the image analysis data.

The object characteristic data may include information corresponding to who, when, what and how.

The controller 540 drives the object detection module and the feature extraction module.

6 is a block diagram of a feature extraction module of an image analysis server according to an embodiment of the present invention.

As illustrated in FIG. 6, the feature extraction module of the image analysis server according to an exemplary embodiment may include a person feature extraction module 610, a vehicle feature extraction module 620, and an event extraction module 630. .

The person feature extraction module 610 detects a person from the input image and extracts a feature of the person.

When the feature feature extraction module 610 extracts feature information from the recognized face, the person feature extraction module 610 extracts texture feature information of the face based on wrinkles, eye, nose, and mouth outline information of the face in the recognized face image.

If face detection in CCTV images is difficult to pinpoint the location of both eyes due to low resolution and wearing glasses, the feature extraction module uses the gabor wavelet texture that resolves the face alignment error as the feature information of the recognized face. Can be. In addition, in order to improve the performance of the feature extraction module to recognize the feature, gabor wavelet can be used for the color information.

In addition, when face expression changes and posture changes frequently occur, the person feature extraction module may fuse color texture face feature information extracted from the same person.

Here, when fusion of color texture facial feature information extracted from the same person, the face of the same person may be distinguished using face clustering results, and then unnecessary information may be removed to recognize the face.

At this time, the face facing the front has the highest recognition rate in extracting feature information from the image.

The vehicle feature extraction module 620 detects the vehicle from the object region image, and extracts the license plate of the vehicle and the feature of the vehicle.

The license plate detection process is performed through preprocessing, gray image conversion, SCW-based binarization, license plate verification, and postprocessing.

The preprocessing process is a series of preprocessing operations for processing an input image, and includes image resizing and histogram smoothing processes.

The gray image conversion process converts a color image into a gray image for binarization.

SCW-based binarization binarizes the converted gray image based on SCW. The license plate verification process extracts objects that meet the license plate conditions from the binarized image.

The post-processing process adjusts the extracted license plate candidate images to the same size and corrects the images through slope compensation.

The event extraction module 630 extracts abnormal behavior of a person or a vehicle.

The extracted movement may be one of roaming, escape, intrusion, designated area intrusion, garbage dumping, illegal parking, and the like. The extracted motion may be represented by a preset ID value.

A prowl is an act of hanging around or wandering around. If the object must be repeatedly present in the same area or if the object exists in the image for a predetermined time or more, it can be detected as a wandering action.

Escape is a sudden run. In the case of appearance, if the person who maintains the average speed and suddenly detects a sudden change in speed, it can be regarded as an escape action.

In order to detect behavior for a designated area, a designated area (region of interest) may be set in advance. The region of interest may be set on a point basis or on a line basis.

If an object appears and exists for more than a predetermined time, it can be detected as an abnormal behavior.

7 is a flowchart illustrating a method of analyzing an image of an image analysis server according to an embodiment of the present invention.

As shown in FIG. 7, the method of analyzing an image of an image analysis server according to an exemplary embodiment of the present disclosure includes detecting and tracking an object (S710) and analyzing an detected object to detect an event (S720). Determining whether the detected object is a person, extracting and storing a feature of a face by recognizing a face (S730), determining whether the detected object is a vehicle, and extracting and storing a license plate and a feature of the vehicle (S740). It may include.

In operation S710, the background is extracted from the image data to extract only the object region image in which the subject is located. In this case, the object detection module may use a technique of using a difference image, a technique of analyzing a motion, a meanshift and an application technique, and a local binary pattern technique.

When detecting an object from the image data, the object can be detected based on the movement. All moving objects can be detected as objects and tracked until they disappear.

If the wind is blowing, the plant can also have movement, so it can be detected as an object.

S720 is a step of detecting the object, tracking the object, and detecting the movement of the object.

The detected movement may be one of roaming, escaping, invading, specifying area invading, garbage illegal dumping, illegal parking, and the like. The detected motion may be represented by a preset ID value.

A prowl is an act of hanging around or wandering around. If the object must be repeatedly present in the same area or if the object exists in the image for a predetermined time or more, it can be detected as a wandering action.

Escape is a sudden run. When an object first appears, and the object maintains an average speed and suddenly has a sudden speed change, it can be regarded as an escape action.

In order to detect behavior for a designated area, the system operator may designate the area as a designated area (region of interest) in advance. The designated area can be set on a point basis or on a line basis.

If an object appears and exists for more than a predetermined time, it can be detected as an abnormal behavior.

S730 determines whether the detected object is a person, and extracts person characteristic data.

If it is difficult to accurately locate the eyes of both eyes due to low resolution and wearing glasses, Gabor wavelet texture that solves the face alignment error is used as the feature information of the recognized face. In addition, Gabor wavelets can be used for color information to improve recognition performance.

If facial expression changes or posture changes frequently occur, color texture facial feature information extracted from the same person may be fused.

Here, when fusion of color texture facial feature information extracted from the same person, the face of the same person may be distinguished using face clustering results, and then unnecessary information may be removed to recognize the face.

In this case, the face facing the front may be most advantageous for extracting feature information from the image.

S740 is a step of extracting the license plate and features of the vehicle, if the detected object is a vehicle.

The license plate detection process is performed through preprocessing, gray image conversion, SCW-based binarization, license plate verification, and postprocessing.

The preprocessing process may include image resizing and histogram smoothing as a series of preprocessing operations for processing the input image.

The gray image conversion process converts a color image into a gray image for binarization. SCW-based binarization binarizes the converted gray image based on SCW. The license plate verification process extracts objects that meet the license plate conditions from the binarized image. The post-processing process adjusts the extracted license plate candidate images to the same size and corrects the images through slope compensation.

8 illustrates feature detection of a person according to an embodiment of the present invention.

The image analysis server according to an embodiment of the present invention may analyze the image data and store the analysis result in the form of text, and the stored analysis data may include a time stamp value and an identification number of the imaging apparatus. The time stamp value of the analysis data and the identification number of the imaging device allow access to the original image.

In addition, the image analysis server according to an embodiment of the present invention may include a module for analyzing a vehicle, a person, an abnormal behavior, and the like.

The vehicle feature extraction module preferentially extracts the license plate of the vehicle and then extracts the feature of the vehicle.

The person feature extraction module may manage all the feature data of the object recognized as the same person.

The event extraction module may extract the abnormal behavior from the image data by tracking an object appearing in the image and determining whether a specific condition is satisfied.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

100: video search system
110: video analytics server
120: video server
130: video search server
140: User terminal
510: object detection module
520: feature extraction module
530: storage unit
540:
610: person feature extraction module
620: vehicle feature extraction module
630: event extraction module

Claims (10)

An object detection module analyzing the image data to search for a subject and extracting an object recognition image from which the background is removed from the image data;
Extracting feature data of the subject by analyzing the object recognition image, converting the feature data into a text format or a binary format to generate object feature data, and obtaining a reproduction time of the image data in which the object is found as an object search time; Feature extraction module;
A storage unit which stores the object search time and the object feature data together as image analysis data; And
And a controller configured to drive the object detection module and the feature extraction module. The method of claim 1,
If the object recognition image is a person, the feature extraction module extracts feature data of the person, converts the feature data into a text format or a binary form, generates object feature data, and a playback time of the image data from which the object is found. Further comprising a character feature extraction module for obtaining the object search time,
The character feature extraction module generates the feature data of the face by using the Gabor wavelet.
The method of claim 1,
The feature extraction module extracts license plate information and feature data of a vehicle when the object recognition image is a car, converts the license plate information and feature data into a text format or a binary format, and generates object feature data. And a vehicle feature extraction module for obtaining a reproduction time of the retrieved image data as an object search time.
The method of claim 3, wherein
The vehicle feature extraction module extracts a license plate and feature data of a vehicle through a preprocessing process, gray image conversion, SCW-based binarization, license plate verification, and post-processing.
The method of claim 1,
The object detection module extracts the color of the subject and re-analyzes the image data by using the color of the subject, thereby preventing a detection error of the subject.
3. The method of claim 2,
And the character feature extraction module generates object feature data by fusing feature data of a subject recognized as the same person.
The method of claim 5, wherein
And the character feature extraction module determines whether a subject is the same person based on a result of face clustering.
An image analysis server according to any one of claims 2 to 3,
An image server for storing and managing an original image, which is an image photographed by a plurality of imaging devices; And
An image search server receiving the search request from a user terminal, analyzing the search request to determine a search condition, searching for analysis data or an image corresponding to the search condition, and transmitting a search completion message to the user terminal; Image retrieval system comprising.
9. The method of claim 8,
And the image search server searches for analysis data corresponding to the search condition and searches for an image using a time stamp value of the found analysis data.
10. The method of claim 9,
If the search condition is an image or an image, the image search system, characterized in that to extract the target and search conditions to search using the object detection module and feature extraction module.

Images