KR101513180B1 - System for a real-time cashing event summarization in surveillance images and the method thereof - Google Patents

Abstract

The present invention provides a video image processing apparatus comprising: a video management unit for receiving a counter image generated by shooting toward a counter of a store from a camera and transmitting the counter image to an image analysis unit; An image analyzer for separating and extracting the moving object from the background image in the counter image received from the image managing unit and tracking the extracted moving object; An event management unit that divides and manages the counter image into a customer area, an employee area, and a counter area preset by a user; The moving image tracked by the image analyzing unit is checked whether the moving object exists in the customer area or the employee area divided by the event management unit and if the moving object exists in both the customer area and the employee area, And when a motion is detected in the image of the counter region, the frame of the corresponding counter image is recorded as the occurrence of a customer-employee interaction event; And a data management unit for storing a frame in which the customer-employee interaction event is generated in the video content analysis unit, in a database; . According to the present invention, by using a low-resolution surveillance camera, images that interact between a customer and an employee are classified and stored in a database, so that a user can monitor payment behavior even with an inexpensive surveillance camera, A real-time settlement event summarizing system using the surveillance camera image and a method thereof can be provided.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system and a method for summarizing real-

More particularly, the present invention relates to a system and method for real-time settlement event summarization using surveillance camera images, and more particularly to a system and method for real- The present invention relates to a system and method for real-time settlement event summarization using a surveillance camera image that enables a problem image such as cheating caused by a customer or an employee to be easily searched even with an inexpensive low resolution surveillance camera.

Intelligent surveillance systems have received much attention from the security camera research community in order to automatically detect and monitor irregular object movements and events [1]. In addition, video analytics systems have been considered as a very efficient means for monitoring intentional or unintentional employee misconduct related to employees' cash registers in the retail and settlement areas of retail stores and stores [2, 3, 4]. In addition, Geron Healthcare Co., Ltd. has developed intelligent surveillance solutions for hospitals, stores, and general applications [5].

Following this trend, researchers have developed a retail behavior awareness model based on hand movements [3, 4]. However, even if such a model provides good performance within a certain limited range, such a model is based on a hand detector that detects hands in the skin color, and thus such a system is not suitable for environments where there is a lot of actual camera distortion, It is difficult to apply.

It is an object of the present invention, which is devised to solve the above-mentioned problems, to provide a low-resolution surveillance camera, in which images interacting between a customer and an employee are classified into a database and stored, And to provide a real-time settlement event summarizing system and a method thereof, which can easily detect a problem image such as an illegal act generated by a customer or an employee.

It is a further object of the present invention to provide a method and system for displaying a customer presence event, an employee presence event, a customer-employee interaction event, a cash register open event, and a card settlement event, The present invention provides a real-time settlement event summarization system and a method for providing real-time settlement event summarization using a surveillance camera image, which provides a history list to a user terminal so that a user can easily perform image retrieval and classification for each event type.

According to another aspect of the present invention, there is provided a system for summarizing real-time settlement events using surveillance camera images, comprising: a video management unit for receiving a counter image generated by shooting toward a counter of a store, An image analyzer for separating and extracting the moving object from the background image in the counter image received from the image managing unit and tracking the extracted moving object; An event management unit that divides and manages the counter image into a customer area, an employee area, and a counter area preset by a user; The moving image tracked by the image analyzing unit is checked whether the moving object exists in the customer area or the employee area divided by the event management unit and if the moving object exists in both the customer area and the employee area, And when a motion is detected in the image of the counter region, the frame of the corresponding counter image is recorded as the occurrence of a customer-employee interaction event; And a data management unit for storing a frame in which the customer-employee interaction event is generated in the video content analysis unit, in a database; .

The image management unit may further include: a camera management unit for receiving a counter image from a camera; An image format unit for converting the counter image received by the camera management unit into RGB image data and transmitting the RGB image data to the image analysis unit; .

In addition, the event management unit divides and manages an image corresponding to a cash register area preset by a user among the counter images, and the image content analyzing unit divides the image of the cash register divided by the event management unit into an image When the similarity is equal to or greater than a predetermined reference value, the frame of the corresponding counter image is recorded as a cash register open event.

In addition, the event management unit divides and manages an image corresponding to a cash register area preset by the user among the counter images, and the image content analyzing unit divides the image corresponding to the cash register area into a predetermined movement direction And calculating a number of the extracted feature points and an average motion distance, if the number of the feature points is equal to or greater than a predetermined number, and the average motion distance is equal to or greater than a predetermined movement distance, And the frame of the image is recorded as a cash register open event has occurred.

In addition, the event management unit divides and manages an image corresponding to a card receipt area preset by the user among the counter images, and the image content analyzing unit analyzes the image of the card receipt area divided by the event management unit, When the similarity is equal to or less than a preset reference value, the frame of the corresponding counter image is recorded as a card settlement event has occurred.

The motion picture content analyzing unit may divide the frame difference image obtained by subtracting the counting region image of the previous frame from the counting region image of a specific frame into a plurality of subblocks, The frame difference image brightness (that is, the brightness of the corresponding pixel in the frame difference image) for each pixel is changed by processing the corresponding brightness value when the brightness value is larger than a certain value, or 0 when the brightness value is smaller than the predetermined value, If the average value of the changed frame difference image brightness for each pixel in the block is set to the brightness value of each sub-block and the average value of each brightness value of all the sub-blocks is equal to or larger than the first threshold value, And a counter movement detection unit.

In addition, the image content analyzing unit may calculate the number of frames in which motion in the K frame periods preceding the specific frame is detected as the number of frames in which no motion is detected, with respect to a specific frame detected as having motion in the counter movement detecting unit And a counting unit motion error compensating unit for detecting the specific frame as a frame with motion when the divided value is equal to or larger than the second threshold value and for detecting the specific frame as a motionless frame when the divided value is equal to or less than the second threshold value.

In addition, if the image analysis unit determines that the moving object exists in the customer area, the data management unit stores the customer presence event identifier in the frame of the corresponding counter image and stores the same in a database. In the image analysis unit, If it is determined that a customer-employee interaction event has occurred in the image analyzing unit, it is determined that the customer-employee interaction event has occurred in the frame of the corresponding counter image If the cash register open event is judged to have occurred in the image analysis unit, the cash register open event identifier is written in the frame of the corresponding counter image and stored in the database,If it is determined to have occurred, the card payment event in seokbu, characterized in that the title of a payment card event identifier to a frame of the image counter, which are stored in the database.

Also, the data management unit may search for and use a customer presence event, an employee presence event, a customer-employee interaction event, a cash register open event, and a card settlement event by each event type, using an event identifier related to the counter image stored in the database. The event history list generated to be classifiable is provided to the user terminal, and when a specific event is selected from the user terminal, a counter image corresponding to the event is provided to the user terminal.

According to another aspect of the present invention, there is provided a method for summarizing a real-time settlement event using a surveillance camera image, the method comprising: (a) receiving a counter image generated by a video management unit toward a counter of a store, ; (b) separating and extracting the moving object from the background image in the counter image received from the image managing unit, and tracing the extracted moving object; (c) The image content analyzing unit checks whether a moving object tracked by the image analyzing unit exists in a customer area or an employee area divided by the event management unit. If the moving object exists in both the customer area and the employee area, Detecting whether there is movement in the image of the counter region, and recording a frame of the counter image as a customer-employee interaction event when a motion is detected in the image of the counter region; And (d) storing, in the database, a frame in which the data management unit generates the customer-employee interaction event in the video content analysis unit; .

In the step (c), the image content analyzing unit compares the image of the cash register divided by the event management unit with the reference template image in the open state of the cash register, and if the similarity is equal to or greater than a preset reference value, And the frame of the image is recorded as a cash register open event has occurred.

In the step (c), the image content analyzing unit may extract the feature points satisfying the predetermined movement direction and the motion distance from the image of the cash register divided by the event management unit, and calculate the number of the extracted feature points and the average motion When the number of the feature points is equal to or greater than a predetermined number and the average motion distance is equal to or greater than a predetermined motion distance after the distance is calculated, the frame of the corresponding counter image is recorded as a cash register open event.

In the step (c), the image content analyzing unit compares the image of the card receipt area divided by the event management unit with the reference template image in which the card receipt is not in power state, and when the similarity is equal to or less than a preset reference value, And the frame of the image is recorded as the occurrence of the card settlement event.

The step (c) includes the steps of: (c1) dividing a frame difference image obtained by subtracting the counting region image of the previous frame from the counting region image of a specific frame to detect a motion in the image of the counter region, The frame difference image brightness (i.e., the brightness of the corresponding pixel in the frame difference image) for each pixel in each sub-block is changed to a corresponding brightness value when the brightness value is larger than a certain value, And the average value of the changed frame difference image brightness for each pixel in each sub-block is set as the brightness value of each sub-block, and when the average value of each brightness value of all sub-blocks is equal to or greater than the first threshold value, And detecting that there is a defect.

The step (c) may further comprise: (c2) detecting, for a specific frame detected as having motion in the step (c1), the number of frames in which motion in K frames before the specific frame is detected, Detecting a specific frame as a motion-based frame when the value divided by the number of frames not equal to the second threshold is greater than or equal to a second threshold; and detecting the specific frame as a motionless frame when the value is equal to or less than the second threshold.

In the step (d), if the data management unit determines that the moving object exists in the customer area, the data management unit displays the customer presence event identifier in the frame of the counter image and stores the same in a database. When the controller determines that the moving object exists in the employee area, the employee presence event identifier is displayed in the frame of the counter image and stored in the database. When it is determined that the customer-employee interaction event has occurred in the image analysis unit, The customer-employee interaction event identifier is written in the frame of the counter image and stored in the database. If it is determined that the cash register open event has occurred in the image analysis unit, the cash register open event identifier is displayed in the frame of the corresponding counter image Save to Database And, when in the image analysis unit is determined to have occurred, the card payment event, the title of a payment card event identifier to a frame of the image counter is characterized in that stored in the database.

In addition, the present invention provides a method for summarizing real-time settlement events using surveillance camera images, comprising the steps of: (e) using the event identifier of the counter image stored in the database to generate a customer presence event, A cash register event, a cash register open event, and a card settlement event for each event type, and when a specific event is selected from the user terminal, a counter image corresponding to the event is provided to the user terminal To the user terminal.

According to the present invention, by using a low-resolution surveillance camera, images interacting between a customer and an employee are classified and stored in a database, so that a user monitors the settlement behavior with an inexpensive low-resolution surveillance camera, A real-time settlement event summarizing system using the surveillance camera image and a method thereof can be provided.

In addition, according to the present invention, the history of the customer, the presence event of the employee, the customer-employee interaction event, the open event of the cash register, and the card settlement event are displayed, The present invention can provide a real-time settlement event summary system and a method for real-time settlement event summarization using a surveillance camera image, by which a user can easily search and classify an image for each event type by providing a list to a user terminal.

1 is a block diagram of a real-time settlement event summary system according to an embodiment of the present invention.
FIG. 2 is a surveillance camera image screen in which four areas-customer, employee, cash register, and cash register area for payment event detection of the present invention are displayed.
3 is a frame difference image processing screen for detecting a customer-employee interaction event of the present invention.
4 is an image similarity check screen for detecting a cash register open event of the present invention.
5 is a XeronISS-SDK screen for developing the system of the present invention.
FIG. 6 is a screen showing experimental results of all the processes of the system according to the present invention.
7 is an event history list screen provided to a user terminal according to the present invention.
8 is a screen showing the motion-based cash register open state detection of the present invention.
9 is a screen showing an example of card settlement event detection of the present invention.
10 is a flowchart of a method of summarizing real-time settlement events according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings.

The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a real-time settlement event summary system according to an embodiment of the present invention.

1, a real-time settlement event summarizing system 100 according to an embodiment of the present invention includes an image management unit 110, an image analysis unit 120, an event management unit 130, an image content analysis unit 140, And a data management unit 150.

The image management unit 110 receives the counter image generated by photographing toward the counter of the store from the camera 200, and transmits the counter image to the image analysis unit 120.

The image analyzer 120 separates the moving object from the background image and extracts the moving object from the counter image received from the image managing unit 110, and tracks the extracted moving object.

The event management unit 130 divides and manages the counter image into a customer area, an employee area, and a checkout area preset by the user.

The video content analysis unit 140 checks whether the moving object tracked by the video analysis unit 120 exists in the customer area or the employee area divided by the event management unit 130. If the moving object exists in the customer area and the employee If motion is detected in the image of the counter region, the frame of the corresponding counter image is recorded as the occurrence of the customer-employee interaction event.

The data management unit 150 stores the frame in which the customer-employee interaction event is generated in the video content analysis unit 140 in the database 300.

The image management unit 110 includes a camera management unit 112 and an image format unit 114. The camera management unit 112 receives a counter image from the camera 200 and the image format unit 114 is a camera management unit 112 converts the received counter image into RGB image data, and transmits the RGB image data to the image analysis unit 120.

In addition, the event management unit 130 may divide and manage an image corresponding to a cash register area preset by the user among the counter images. In this case, the image content analysis unit 140 may determine The image of the divided cash register is compared with the reference template image of the open state of the cash register, and if the similarity is equal to or greater than the preset reference value, the frame of the corresponding counter image is recorded as the cash register open event.

Alternatively, the image content analysis unit 140 extracts feature points that satisfy predetermined motion direction and motion distance from the image of the cash register divided by the event management unit 130, and calculates the number of the extracted feature points and the average motion If the number of the feature points is equal to or greater than a predetermined number after calculating the distance and the average motion distance is equal to or greater than a predetermined motion distance, the frame of the counter image may be recorded as a cash register open event.

The event management unit 130 divides and manages an image corresponding to the card receipt area preset by the user among the counter images and the image content analysis unit 140 analyzes the image data of the card receipt area The image is compared with the reference template image in which the receipt of the card is not in a power state and if the similarity is equal to or less than a preset reference value, the frame of the corresponding counter image is recorded as a card settlement event.

In addition, the video content analysis unit 140 includes a counter movement detection unit 142 and a counter movement error compensation unit 144.

When detecting motion in the image of the counter region, the counter movement detecting unit 142 divides the frame difference image obtained by subtracting the counting region image of the previous frame from the image of the counter region of the specific frame into a plurality of sub-blocks, The frame difference image brightness for each pixel is calculated by processing the corresponding brightness value when the brightness value is larger than a certain value, or to 0 when the brightness value is smaller than the predetermined value, and calculating the average value of the frame difference image brightness of each pixel in each sub- Is a brightness value of each sub-block, and when the average value of brightness values of all sub-blocks is equal to or greater than a first threshold value, the specific frame is detected as having motion.

The counting unit motion error compensating unit 144 calculates the number of frames in which motion in the K frame periods preceding the specific frame is detected as a motion detected in the counter movement detecting unit 142 When the value divided by the number of frames is equal to or larger than the second threshold, the specific frame is finally detected as a frame having motion, and when the value is equal to or smaller than the second threshold value, the specific frame is detected as a frame without motion.

When it is determined in the image analysis unit 120 that the moving object exists in the customer area, the data management unit 150 displays the customer presence event identifier in the frame of the corresponding counter image and stores it in the database 300, If it is determined that the mobile object is present in the employee area, the controller 120 displays an employee presence event identifier in the frame of the counter image and stores the employee presence event identifier in the database 300. In the image analysis unit 120, If it is determined that the action event has occurred, the customer-employee interaction event identifier is displayed in the frame of the counter image and stored in the database 300. If it is determined that the cash register open event has occurred in the image analysis unit 120 , The cash register open event identifier is written in the frame of the corresponding counter image, 00). If it is determined that the card settlement event has occurred in the image analysis unit 120, the card settlement event identifier is displayed in the frame of the corresponding counter image and is stored in the database 300.

In addition, the data management unit 150 uses the event identifier related to the counter image stored in the database 300 to determine whether the customer presence event, the employee presence event, the customer-employee interaction event, the cash register open event, The event history list generated so as to be able to be searched and classified according to the event type is provided to the user terminal 400 and when a specific event is selected from the user terminal 400, a counter image corresponding to the event is provided to the user terminal 400 do.

The features of the present invention will be described in more detail as follows.

1. Overview

The present invention relates to a real-time settlement event summarizing system for monitoring and checking deliberate or unintentional employee fraud in a surveillance camera image. In order to summarize the settlement events in the recorded images, the present invention proposes a region-based customer-cashier interaction event (CCIE) and a cash register opening event (CROE). In order to detect these events, the present invention considers the customer or employee presence event using the information of the tracked object and its user defined area. In addition, in order to detect the CCIE and CROE, the present invention considers motion and pattern matching analysis methods in each of these areas, along with sensed customer or employee presence events. In addition, in order to detect a customer or employee presence event within these areas, the present invention considers an adaptive background-based moving object detection and tracking approach while addressing slow moving and stationary object problems.

2. The present invention In one embodiment  A description of the real-time billing event summary system

The system 100 of the present invention includes a camera management unit 112, an image format unit 114, an image analysis unit 120, an image content analysis unit 140, a data management unit 150, and an event management unit 130 It includes important components.

In the system 100 of the present invention, the camera management unit 112 and the image format unit 114 manage a video input terminal and convert an input image into RGB image data. Next, the event management unit 130 manages rule generation, deletion, modification, and scheduling of the user-defined image for the image analysis unit 120 and the image content analysis unit 140. The image analysis unit 120 extracts the object moving area together with a unique number (ID) and its attributes, for example, size, speed, color, and shape in a sequential input image. The video content analysis unit 140 analyzes events such as settlement events using not only user defined event rules but also information of moving objects from the input video. The data management unit 150 converts output data from the image analysis unit 120 and the image content analysis unit 140 into a metadata format and stores it in the database 300 for later use.

3. Billing event detection

To detect a payment event, the present invention defines four areas-customer, employee, checkout, and cash register area, as shown in FIG. In a first step for detecting the CCIE, the system 100 of the present invention uses tracking information to detect customers or employees in these areas. The CCIE is then determined by the motion analysis in the counter zone. To determine the CROE, the present invention detects an employee presence event in the employee area and then performs pattern matching analysis in the cash register area. The sensed event results are then stored in the database 300. Accordingly, the settlement event can be summarized by searching the database 300 for the detected event time and corresponding recorded images.

3.1. Moving object detection and tracking

In order to obtain the information of the moving object from the sequential input image in order to detect the presence events of customers and employees, the present invention can be reviewed with an adaptive background model based object detection and tracking or learning based human detection model (X. Wang, TX Han, and S. Yan, "An HOG-LBP Human Detector with Partial Occlusion Handling", ICCV09) [6, 7, 8].

In the case of the adaptive background model, the background is learned using Gaussian Mixture Model (GMM) for the pixels of the input image, and the background pixel or the moving object pixel is classified by comparing the input image with the GMM background model. To remove misclassified pixels from classified moving object images, we apply postprocessing techniques such as morphology and then separate the moving object regions using connected components labeling. [6, 7, 8, 9, 10, 11]. Then, the minimum / maximum size filter is applied to remove the noise object area that is not the size of the person. Finally, it can be traced through simply performing a matching pair procedure on the detected human region (see, for example, WJ Won and et al., "Simplified Multiple Objet Tracking Model for Real-Time Intelligent Surveillance System" , 2013. [12, 13].

Staffer and Grimson proposed a GMM-based adaptive background model using a K-Gaussian mixture [6, 8]. For each pixel, the composite weight at time t is updated as shown in Equation 1 below based on the weight at time t-1.

는 학습 비율(learning rate)이다. here,

Is the learning rate.

의 하향 순서로 정렬된 된다. 또한, 입력 영상과 매칭되는 가우시안 모델의 수식인 B는 다음 수학식 2와 같다.Also, in order to update the background of the input image, it is checked whether or not there is a matched model using the average value and the variance value for the previously learned K Gaussian, and if there is a matched model, the corresponding model is updated, . Also, after generating / updating the background Gaussian model for the pixel, the Gaussian models of the pixel

As shown in FIG. B, which is a mathematical expression of the Gaussian model matched with the input image, is expressed by the following equation (2).

는 매칭되지 않은 분산을 위해 동일하게 남아있게 된다.T is the minimum amount of data that should be considered by the background. here,

≪ / RTI > remain the same for unmatched variance.

Although this model works robustly in real-world environments, such as changes in object reflections due to illumination and background, it is problematic in detecting slow moving or stationary objects [6]. If the object moves very slowly or stops, pixels in the object area are updated as background with the same learning rate. As a result, the model can not detect slow moving or stopped objects. To solve this problem, the present invention introduces an adaptive learning rate concept such as Equation (3) using tracking object domain feedback from a tracker.

은 고정된 초기

변수이고,

는 각 추적된 객체에 대한 추적 이력(count)이고, TM은 추적 영역에 포함된 픽셀들에 대해서는 1로 설정되어 있고 그렇지 않은 영역에서는 0으로 설정되어 있는 추적 맵 표시자이다. 또한, 본 발명에서는

가 특정된 경계를 벗어나지 않도록 상한 및 하한 경계로서

을 설정하였다. 이에 따라,

는

로서 계산되고,

는 업데이트 시간 변수이다.here,

Is a fixed initial

Variable,

Is a tracking history (count) for each tracked object, and TM is a tracking map indicator set to 1 for pixels included in the tracking area and to 0 for areas not included. In the present invention,

As upper and lower bounds so as not to deviate from the specified boundary

Respectively. Accordingly,

The

Lt; / RTI >

Is an update time variable.

For this object tracking, the present invention introduces a feature matching method for generating a matching pair between the sensing object area and the tracking object area [12, 13]. In the feature matching step, the candidate matching Pair is selected by using the distance between images and the size ratio between the objects for the tracking object list and the sensing object lists. The binary matching pair table is regarded as a final candidate for the matching detection and tracking object when the histogram similarity is larger than the tracking threshold set by the user. After finding the best matching pair between the sensing object and the tracking object using the scoring method for the distance between the binary matching table and the detection / tracking object region, the matching tracking object determines the position and characteristics of the sensing object Lt; / RTI > However, if the matching pair is recognized as a merging object, the tracking object property is not updated. Otherwise, for an unmatched, overlapped tracking object, the tracking object position is updated by the expected position from the previous frame. Unmatched detection objects are recorded as new tracking objects after checking a specific match.

3.2. Customer-employee interaction event detection

To detect the CCIE, the system 100 of the present invention checks whether a customer and an employee are present by comparing the tracking object area with a user defined area. Then, if there are customers and employees, perform a motion analysis to determine if there is a CCIE in the customized checkout area. For this motion analysis, the present invention introduces a simple frame difference method using the following equation (4).

는 프레임 비율(frame rate)에 의해 결정되는 프레임 간격(frame interval)이다.here,

Is the frame interval determined by the frame rate.

에서의 프레임 이미지에서

에서의 프레임 이미지를 뺀 이미지를 얻은 후, 이 프레임 차이 이미지는 도 3에 도시된 바와 같이 N x M 서브 블록에 의해 분할된다. 그 다음, 각 서브 블록 내 개개의 픽셀(x,y 좌표상의 픽셀)에 대하여 프레임 차이 이미지의 밝기 D(x, y)를 Dth보다 크면 해당 밝기값(d), Dth보다 작으면 0으로 처리하여 산출하고, 각 서브 블록 내 개개의 픽셀의

의 평균값을 구하여, 그 평균값을 각 서브 블록의 밝기값 S(m,n)으로 한다.A frame difference image, i.e., a specific time

In the frame image at

The frame difference image is divided by NxM subblocks as shown in FIG. Then, if the brightness D (x, y) of the frame difference image is greater than Dth for each pixel (x, y coordinate) in each sub-block, the brightness value d is set to 0 And each pixel in each sub-block is calculated

, And the average value is set as the brightness value S (m, n) of each sub-block.

)에 의해 결정된다.The frame level interaction event TE, which is an interaction event at the frame level, is calculated using the sub-block brightness value S (m, n) and the threshold value

).

)와 감지되지 않은 이벤트(

)의 비율과 임계치(

)에 의해 다음 수학식 6을 사용하여 결정된다.However, image distortion and noise can make erroneous decisions in frame level interaction detection. In order to reduce transient event detection errors, the present invention introduces an error compensation method using a temporal analysis method. Each frame level detection result is stored for K periods. Then, the final event detection result (FE)

) And undetected events (

) And the threshold value (

) Using the following equation (6).

That is, in the above Equation (6), an event may occur due to instantaneous image distortion (change of light, image distortion caused by shadow, etc.) in the image. To overcome this, And it is necessary to observe whether or not the event is detected during the set time by the user and to confirm the ratio of the event detection result and the event non-detection result during the set time, thereby preventing the event detection due to the instantaneous image distortion.

3.3. Cash register open event detection

As shown in FIG. 4, if the employee is within the employee area, the system of the present invention checks the open status of the cash register in the user defined cash register area to detect the CROE based on the pattern matching analysis. To check the cash register open status, the present invention simply introduces a template matching method [14]. For an input image, a similarity map is generated by the following Equation (7).

The cash register open state is then determined by searching for a local maximum point that exceeds the threshold in the similarity map for each frame. The final detection result is determined by Equation (6), which is an error compensation algorithm.

4. Experimental results

Using the Xeron Intelligent surveillance system-software development kit (XeronISS-SDK), we implemented a payment event summary system as a store monitoring application called XeronISS Client. As shown in FIG. 5, the XeronISS-SDK supports a C ++ library-based parameter setting, a terminal management unit, an image format unit, a data management unit, an image processor, a VA, a VCA and an alarm / event management module in order to easily develop a surveillance system do. XeronISS Client also has 16 channel monitoring, VA, VCA, smart search, event statistical analysis, video recording and summary and active auto tracking.

Figure 6 shows the experimental results for all processes of the system according to the invention. When the moving object appears on the screen, the system of the present invention detects and tracks the moving object. Then, as shown in Fig. 6 (a), the customer and employee presence events are determined using the tracking object area. Finally, the system of the present invention senses CCIE and CROE using motion and pattern matching analysis, as shown in Figure 6 (b).

As shown in FIG. 7, the system of the present invention can arrange a list of event histories during a user-defined period and display the history. The user can then view a summary image and a snapshot of the image by clicking on the event history list.

FIG. 8 is a screen showing the motion-based cash register open state detection of the present invention, and FIG. 9 is a screen showing an example of card settlement event detection of the present invention.

Hereinafter, the operation and operation of the present invention will be described in detail with reference to FIG.

10 is a flowchart of a method of summarizing real-time settlement events according to an embodiment of the present invention.

First, the image management unit 110 receives the counter image generated by photographing toward the counter of the store from the camera 200, and transmits the counter image to the image analysis unit 120 (step S110).

Then, the image analysis unit 120 separates the moving object from the background image and traces the extracted moving object in the counter image received from the image management unit 110 (step S120).

Then, the video content analysis unit 140 checks whether the moving object traced in the video analysis unit 120 exists in the customer area or the employee area divided by the event management unit 130, Area and the employee area, it is detected whether there is motion in the image of the counter zone. When motion is detected in the image of the counter zone, the frame of the corresponding counter image is recorded as the occurrence of the customer-employee interaction event (step S130 ).

Then, the data management unit 150 stores the frame in which the customer-employee interaction event is generated in the video content analysis unit 140 in the database 300 (step S140).

The data management unit 150 then uses the event identifier for the counter image stored in the database 300 to determine whether the customer presence event, the employee presence event, the customer-employee interaction event, the cash register open event, When a specific event is selected from the user terminal 400, the event history list generated so as to be searched and classified for each event type is provided to the user terminal 400, and a counter image corresponding to the selected event is transmitted to the user terminal 400 (Step S150).

Here, in step S130, the video content analyzer 140 compares the video of the cash register divided by the event manager 130 with the reference template video in the open state of the cash register, and if the similarity is not less than a predetermined reference value , The frame of the corresponding counter image can be recorded as the occurrence of the cash register open event.

Alternatively, in step S130, the image content analysis unit 140 extracts feature points that satisfy predetermined motion direction and motion distance from the image of the cash register divided by the event management unit 130, and extracts the extracted feature points If the number of the feature points is equal to or greater than a predetermined number after calculating the number and the average motion distance and the average motion distance is equal to or greater than the predetermined motion distance, the frame of the corresponding counter image may be recorded as a cash register open event.

In step S130, the image content analysis unit 140 compares the image of the card receipt area divided by the event management unit 130 with the reference template image in which the card receipt is in a low power state, , The frame of the corresponding counter image is recorded as the occurrence of the card settlement event.

Step S130 includes the following steps S132 and S134.

First, in step S132, a frame difference image obtained by subtracting the counting region image of the previous frame from an image of the counting region of a specific frame is divided into a plurality of subblocks, The frame difference image brightness for the pixel is calculated by processing the corresponding brightness value when the brightness value is larger than a certain value, or to 0 when the brightness value is smaller than the predetermined value. The average value of the frame difference image brightness of each pixel in each sub- Block, and when the average value of each brightness value of all the sub-blocks is equal to or larger than the first threshold value, the specific frame is detected as having motion.

In the next step S134, a value obtained by dividing the number of detected frames in the K frame periods before the specific frame by the number of frames in which no motion is detected is calculated for a specific frame detected as having motion in step S134 If the difference is equal to or greater than the second threshold value, the specific frame is detected as a frame with motion, and if the specific frame is below the second threshold value, the specific frame is detected as a frame without motion.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. Such modifications are within the scope of the claims.

Bibliography

[1] Y. Tian, L. Brown , A. Hampapur, M. Lu, "IBM Smart Surveillance System (S3): Event Based Video Surveillance System with an Open and Extensible Framework," Machine Vision and Applications , vol. 19, Sep. 2008.

[2] AW Senior, "Video analytics for retail," in AVSS , pp. 423-428, 2007.

[3] H. Trinh, Q. Fan , J. Pan, P. Gabbur, S. Miyazawa, and S.Pankanti, "Detecting Human Activities in Retail Surveillance Using Hierarchical Finite State Machine," in ICASSP , pp. 1337-1340, 2011.

[4] H. Trinh, Q. Fan, P. Gabbur, and S. Pankanti, "Hand Tracking Binary Quadratic Programming and Its Application to Retail Activity Recognition," in CVPR , pp. 1902-1909, 2012.

[5] http://www.xeron-healthcare.com/

[6] Y. Tian, A. Senior, and M. Lu, "Robust and Efficient Foreground Analysis in Complex Surveillance Video," Machine Vision and Application , vol. 23, no. 7, pp. 967-983, Sep. 2012.

[7] T. Bouwmans, F. El Baf, and B. Vachon, "Background Modeling using Mixture of Gaussians for Foreground Detection-A Survey," Recent Patents on Computer Science , vol. 1, no.3 pp.21-237, 2008.

[8] C. Stauffer, and WEL Grimson, "Adaptive background mixture models for real-time tracking," in CVPR , 1999.

[9] Z. Zivkovic, and F. Heijden, "Efficient adaptive density estimation per image pixel for the task of background substraction," Pattern Recognition Letter , vol. 27, pp. 773-780, 2006.

[10] F. Porikli, and J. Thornton, "Shadow flow: a recursive method to learn moving cast shadows" in ICCV , 2005.

[11] S. Nadmi and B. Bhanu, "Physical models for moving shadow and object detection in video," IEEE Trans . Pattern Analysis and Machine Intelligence , vol. 26, no. 8, pp. 1079-1087, 2004.

[12] Q. Pan, J. Li , and SZ Li, "Real-time multiple objects tracking with occlusion handling in dynamic scenes," in CVPR , pp. 970-975, 2005.

[13] S. Johnsen, and A. Tews, "Real-Time Object Tracking and Classification Using a Static Camera," in People and Detection workshop of ICRA , 2009.

[14] F. Jurie, and M. Dhome, "Real Time Robust Template Matching," in BMCV , pp. 123-132, 2002.

100: real-time settlement event summary system 110:
112: camera management unit 114: image format unit
120: Image analysis unit 130: Event management unit
140: video content analysis unit 142:
144: counter movement error compensating unit 150:
200: camera 300: database
400: User terminal

Claims (17)

A video management unit for receiving a counter image generated by shooting toward a counter of a store from a camera and transmitting the counter image to an image analysis unit;
An image analyzer for separating and extracting the moving object from the background image in the counter image received from the image managing unit and tracking the extracted moving object;
An event management unit that divides and manages the counter image into a customer area, an employee area, and a counter area preset by a user;
The moving image tracked by the image analyzing unit is checked whether the moving object exists in the customer area or the employee area divided by the event management unit and if the moving object exists in both the customer area and the employee area, And when a motion is detected in the image of the counter region, the frame of the corresponding counter image is recorded as the occurrence of a customer-employee interaction event; And
A data management unit for storing, in a database, a frame in which the customer-employee interaction event is generated in the image content analysis unit; Real - time settlement event summarization system using surveillance camera images.
The image processing apparatus according to claim 1,
A camera management unit for receiving a counter image from a camera; And
An image format unit for converting the counter image received by the camera management unit into RGB image data and transmitting the RGB image data to the image analysis unit; Wherein the real-time settlement event summarization system uses the surveillance camera image.
The method according to claim 1,
Wherein the event management unit divides and manages an image corresponding to a cash register area preset by a user among counter images,
The image content analyzing unit compares the image of the cash register divided by the event management unit with the reference template image of the open state of the cash register, and if the similarity is equal to or greater than a preset reference value, Is recorded as the occurrence of the real-time settlement event.
The method according to claim 1,
Wherein the event management unit divides and manages an image corresponding to a cash register area preset by a user among counter images,
The image content analyzing unit extracts feature points that satisfy predetermined motion direction and motion distance from the image of the cash register divided by the event management unit, calculates the number of the extracted feature points and the average motion distance, When the number of frames is equal to or greater than a predetermined number and the average motion distance is equal to or greater than a preset motion distance, the frame of the corresponding counter image is recorded as a cash register open event.
The method according to claim 1,
The event management unit divides and manages an image corresponding to a card receipt area preset by a user among counter images,
The image content analyzing unit compares the image of the card receipt area divided by the event management unit with a reference template image in which the card receipt is in a low output state and if the similarity is not more than a preset reference value, And the real-time settlement event summary system using the surveillance camera image.
The image processing apparatus according to claim 1,
A frame difference image obtained by subtracting a counting region image of a previous frame from an image of a counter region of a specific frame is divided into a plurality of sub-blocks, and a frame difference image The brightness (i.e., the brightness of the corresponding pixel in the frame difference image) is changed by processing the corresponding brightness value when the brightness value is larger than a certain value, or 0 when the brightness value is smaller than the predetermined value, And a counter movement detecting unit for detecting that the specific frame has motion when the average value of brightness of the frame difference image is a brightness value of each sub-block and the average value of each brightness value of all the sub-blocks is equal to or greater than a first threshold value Real - time settlement event summary system using camera images.
The image processing apparatus according to claim 6,
If the value obtained by dividing the number of frames detected as the motion in the K frame periods before the specific frame by the number of frames in which no motion is detected is greater than or equal to the second threshold value for a specific frame detected as motion in the cash- And a calculation unit motion error compensator for detecting the specific frame as a motion frame and finally detecting the specific frame as a motionless frame if the specific frame is below a second threshold value. system.
The method according to claim 3 or 4,
The event management unit divides and manages an image corresponding to a card receipt area preset by a user among counter images,
The image content analyzing unit compares the image of the card receipt area divided by the event management unit with a reference template image in which the card receipt is in a low output state and if the similarity is not more than a preset reference value, Recorded,
The data management unit,
If it is determined that the moving object exists in the customer area in the image analysis unit, the customer presence event identifier is displayed in the frame of the corresponding counter image and is stored in the database,
If it is determined that the moving object exists in the employee area in the image analysis unit, the employee presence event identifier is written in the frame of the corresponding counter image and is stored in the database,
If it is determined that a customer-employee interaction event has occurred in the image analysis unit, the customer-employee interaction event identifier is displayed in the frame of the counter image and stored in the database,
If it is determined that the cash register open event has occurred in the image analysis unit, the cash register open event identifier is written in the frame of the counter image and stored in the database,
Wherein the card settlement event identifier is displayed in a frame of the counter image and stored in the database when the image analyzing unit determines that a card settlement event has occurred.
9. The data processing apparatus according to claim 8,
An event history generated so that the customer presence event, the employee presence event, the customer-employee interaction event, the cash register open event, and the card settlement events can be searched and classified for each event type, using the event identifier related to the counter image stored in the database Providing a list to the user terminal, and when a specific event is selected from the user terminal, providing a counter image corresponding to the selected event to the user terminal.
(a) receiving, from a camera, a counter image generated by capturing an image of a store toward a counter of a store, and transmitting the received counter image to a video analyzer;
(b) separating and extracting the moving object from the background image in the counter image received from the image managing unit, and tracing the extracted moving object;
(c) The image content analyzing unit checks whether a moving object tracked by the image analyzing unit exists in a customer area or an employee area divided by the event management unit. If the moving object exists in both the customer area and the employee area, Detecting whether there is movement in the image of the counter region, and recording a frame of the counter image as a customer-employee interaction event when a motion is detected in the image of the counter region; And
(d) storing, in the database, a frame in which the data management unit generates the customer-employee interaction event in the video content analysis unit; A method for summarizing real time settlement events using surveillance camera images.
The method of claim 10, wherein the step (c)
The image content analyzing unit compares the image of the cash register divided by the event management unit with the reference template image in which the cash register is opened and if the similarity is equal to or greater than a preset reference value, Wherein the real-time settlement event summarization method records real-time settlement event summaries using surveillance camera images.
The method of claim 10, wherein the step (c)
The image content analyzing unit extracts the feature points satisfying the predetermined movement direction and the motion distance from the image of the cash register divided by the event management unit, calculates the number of the extracted feature points and the average motion distance, When the average motion distance is equal to or greater than a predetermined number and the average motion distance is equal to or greater than a preset motion distance, the frame of the corresponding counter image is recorded as a cash register open event.
The method of claim 10, wherein the step (c)
The image content analyzing unit compares the image of the card receipt area divided by the event management unit with the reference template image in which the card receipt is not in the output state and if the similarity is equal to or less than a preset reference value, And recording the real time settlement event using the surveillance camera image.
The method of claim 10, wherein the step (c)
(c1) dividing the frame difference image obtained by subtracting the counting region image of the previous frame from the counting region image of the specific frame, into a plurality of subblocks, The brightness of the frame difference image (i.e., the brightness of the corresponding pixel in the frame difference image) is changed by processing the corresponding brightness value if the brightness value is larger than a certain value, or 0 when the brightness value is smaller than the predetermined value, And determining that the specific frame has motion if the average value of the brightness values of all the subblocks is equal to or greater than the first threshold value Real - time settlement event summary method using surveillance camera images.
15. The method of claim 14, wherein step (c)
(c2) a value obtained by dividing the number of detected frames in K frames before the specific frame by the number of frames in which no motion is detected, for a specific frame detected as having motion in the step (c1) Detecting a specific frame as a frame in which motion is finally determined and detecting the specific frame as a motionless frame if the detected frame is below a second threshold; .
13. The method according to claim 11 or 12,
The step (c)
The image content analyzing unit compares the image of the card receipt area divided by the event management unit with the reference template image in which the card receipt is not in the output state and if the similarity is equal to or less than a preset reference value, ≪ / RTI >
The step (d)
If the data management unit determines that the moving object exists in the customer area, the data management unit stores the customer presence event identifier in the frame of the corresponding counter image and stores the same in the database, and if the moving object exists in the employee area An employee presence event identifier is displayed in a frame of the corresponding counter image and stored in the database, and when it is determined that a customer-employee interaction event has occurred in the image analysis unit, the customer- If the cash register open event is judged to have occurred in the image analysis unit, the cash register open event identifier is written in the frame of the counter image and stored in the database, If it is determined that the de billing event occurs, how to mark the card payment event identifier to a frame of the image counter SUMMARY real time billing event using a surveillance camera image, characterized in that for storing in the database.
17. The method of claim 16,
(e) The data management unit searches for a customer presence event, an employee presence event, a customer-employee interaction event, a cash register open event, and a card settlement event by each event type using an event identifier related to the counter image stored in the database Providing an event history list generated so as to be classifiable to a user terminal and providing a counter image corresponding to the selected event to a user terminal when a specific event is selected from the user terminal; The method comprising the steps of:

Images