KR20230085281A - Image analysis system and method based on image and lidar sensor - Google Patents

Abstract

The present invention relates to an image analysis system and method, the system being connected through a network to at least one lidar sensor, a surveillance camera, and a manager terminal installed in a branch office. The image analysis system comprises: a database part for storing collection information, which is generated by collecting lidar sensor information, surveillance camera information, sensor information and image information, analysis information and manager information; and a server for collecting image information and sensor information output from the lidar sensor and the surveillance camera, analyzing the collected image information and sensor information to calculate a congestion index for measuring the congestion degree of the branch office, and analyzing the collected image information and sensor information to calculate the satisfaction degree of a customer. In addition, an embodiment of the present invention enables a process in the branch office for reducing perceived congestion to be verified using actual/intuitive congestion degrees, which are calculated from customer data analyzed from the moment of entering the branch office until the time of handling tasks and leaving the branch office.

Description

Image analysis system and method based on image and lidar sensor}

The present invention relates to an image and lidar sensor-based image analysis system and method, and in particular, monitors branches of financial institutions through image and lidar sensor-based data collaboration, and multi-channel data Based on redundant processing, it is about a video and lidar sensor-based video analysis system and method that analyzes data by branch unit, improves face-to-face channel customer experience in response to the analysis result, and can preemptively respond to crime in the branch. .

In general, along with the rapid development of communication technology and computer fields, computer communication and Internet communication have become an essential means for modern people as a place of information acquisition and exchange, and their functions and roles are explosively increasing.

As the Internet is activated, the Internet is emerging as one of the important sources of information. In particular, the Internet is gradually becoming the center of information to the extent that a significant portion of information on law, medical, etc., which in the past had to be consulted by experts, can be provided through the Internet.

On the other hand, the congestion level based on the intelligent queuing system (IQS) currently being serviced is not separated from business customers and non-business customers and is statistically processed in batches, making it impossible to calculate actual and real-time congestion levels.

And since customers are currently counted from the point in time after the queue ticket is issued, there is a limit to reflecting the point in time after entering the branch.

In addition, in the related art, a customer waiting chair, a digital information display (DID), an IQS, and the like are traditionally or collectively arranged by a branch office.

Due to the arrangement where the characteristics of the branch and the customer's point of view were not considered, the facility and environment management of the branch depended on the police officer and the manager of the branch.

In addition, in order to arrest crimes such as voice phishing, conventional financial services that continuously track account transaction details are unreliable and discontinuous.

In the prior art, there has been a case in which an employee (including a police officer) accidentally detects a customer's abnormal behavior and prevents voice phishing, but this is not sufficient as a method for preventing voice phishing.

In this way, conventionally, in branches of financial institutions such as banks, it is required to improve and understand customer waiting experience in order to improve not only actual congestion but also perceived congestion due to increased congestion.

In addition, as crimes such as voice phishing are increasing, the need for real-time monitoring services using video and LIDAR sensors is emerging.

The technical problem to be achieved by the present invention is to improve the conventional problem, and to reduce the perceived congestion by using the actual/intuitive congestion calculated from customer data analyzed from the moment of entering the branch to the time of processing and leaving the branch. It is possible to verify my process??, to provide an image analysis system and method based on images and lidar sensors.

In addition, the technical problem to be achieved by the present invention is to improve the conventional problem, to provide an image and lidar sensor-based image analysis system and method that can rearrange furniture and DID according to the customer's moving line by branch situation. .

In addition, the technical problem to be achieved by the present invention is to improve the conventional problems, and it is possible to promptly cope with congestion in the branch with real-time congestion notification, it is possible to prevent crime through monitoring the operation of the branch, and facility safety To provide an image and lidar sensor-based image analysis system and method capable of monitoring customer safety and environmental management.

An image and lidar sensor-based image analysis method according to the characteristics of the present invention for solving these problems is,

An image analysis method of a video analysis system connected to at least one lidar sensor and surveillance camera installed in a branch through a network,

Collecting, by a server, image information and sensor information output from the lidar sensor and surveillance camera;

calculating, by the server, a congestion index for measuring the degree of congestion of the branch by analyzing the collected image information and sensor information;

Calculating customer satisfaction by analyzing the collected image information and sensor information by the server;

determining, by the server, whether there is an abnormal behavior of the customer;

When there is an abnormal behavior of the customer, the server makes an alarm.

The congestion index is

Calculate the average number of waiting customers and waiting time,

Standardize the calculated average number of waiting customers and waiting time,

After scoring the standardized waiting customer standardized value and waiting time standardized value as 100 points,

It is characterized in that the congestion index is calculated.

The customer satisfaction level is

Calculate customer satisfaction index through waiting time, facial expression and behavior.

As a result of customer experience management surveys within financial institutions, customer perceived congestion, which is a satisfaction index for the waiting situation when a customer visits the store, has an inverse correlation between customer perceived congestion and customer experience index. ) is characterized by an increase in

An image and lidar sensor-based image analysis system according to the characteristics of the present invention for solving these problems is,

A video analysis system connected through a network with at least one lidar sensor, surveillance camera, and staff terminal installed in a branch,

A database unit for storing lidar sensor information, monitoring camera information, sensor information and image information collection information, analysis information, and person in charge information;

Collect image information and sensor information output from the lidar sensor and surveillance camera, analyze the collected image information and sensor information to calculate a congestion index to measure the congestion level of the branch, and calculate the collected image information and sensor information It includes a server that analyzes and calculates customer satisfaction.

The server determines whether there is an abnormal behavior of the customer, and if there is an abnormal behavior of the customer, the server sends an alarm to a designated person in charge with a note.

The database unit,

Lidar sensor information DB that stores information on the location where the lidar sensor is installed and the branch office;

Surveillance camera information DB that stores information on locations where surveillance cameras are installed and branch offices;

a collection information DB for storing collection information obtained by collecting sensor information output from the lidar sensor and image information output from the monitoring camera;

an analysis information DB for storing analysis information obtained by analyzing the collection information;

A contact information DB for storing the contact information and terminal information of the person in charge is included.

The server,

an information collection unit that collects image information and sensor information output from the lidar sensor and surveillance camera;

an analysis unit that analyzes the collected image information and sensor information to calculate a congestion index for measuring the congestion level of the branch, and analyzes the collected image information and sensor information to calculate customer satisfaction;

and an alarm unit that determines whether there is an abnormal behavior of the customer and, when there is an abnormal behavior of the customer, notifies the terminal of the person in charge.

The congestion index is

Calculate the average number of waiting customers and waiting time,

Standardize the calculated average number of waiting customers and waiting time,

After scoring the standardized waiting customer standardized value and waiting time standardized value as 100 points,

It is characterized in that the congestion index is calculated.

The customer satisfaction level is

Calculate customer satisfaction index through waiting time, facial expression and behavior.

As a result of customer experience management surveys within financial institutions, customer perceived congestion, which is a satisfaction index for the waiting situation when a customer visits the store, has an inverse correlation between customer perceived congestion and customer experience index. ) is characterized by an increase in

In the embodiment of the present invention, it is possible to verify the process in the branch for reducing the perceived congestion using the actual/intuitive congestion calculated with the analyzed customer data from the moment of entering the branch to the time of processing and leaving the branch. A lidar sensor-based image analysis system and method may be provided.

In addition, in the embodiment of the present invention, it is possible to provide an image and lidar sensor-based image analysis system and method capable of relocating furniture and DID according to the customer's movement line for each branch situation.

In addition, in the embodiment of the present invention, it is possible to promptly cope with congestion in the branch with real-time congestion notification, it is possible to prevent crime through monitoring the operation of the branch, and it is possible to monitor facility safety, customer safety and environmental management, An image and lidar sensor-based image analysis system and method can be provided.

1 is a diagram illustrating an image analysis system based on an image and lidar sensor according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating the configuration of a manager terminal of FIG. 1 .
3 is a diagram illustrating an image analysis method based on an image and lidar sensor according to an embodiment of the present invention.
4 to 4 are diagrams for explaining the final operation of an image and lidar sensor-based image analysis system according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

1 is a diagram illustrating an image analysis system based on an image and lidar sensor according to an embodiment of the present invention.

Referring to FIG. 1, an image analysis system 100 based on an image and lidar sensor according to an embodiment of the present invention,

As an image analysis system 100 connected to at least one lidar sensor 210, a monitoring camera 220, and a person in charge terminal 300 installed in a branch office through a network 400,

A database unit 120 for storing lidar sensor information, surveillance camera information, sensor information and image information collection information, analysis information, and person in charge information;

The image information and sensor information output from the lidar sensor 210 and the surveillance camera 220 are collected, and the collected image information and sensor information are analyzed to calculate the congestion index for measuring the congestion of the branch, and the collected and a server 110 that analyzes image information and sensor information to calculate customer satisfaction.

The server 110 determines whether there is an abnormal behavior of the customer, and if there is an abnormal behavior of the customer, the server 110 notifies the terminal 300 of the person in charge.

The database unit 120,

LiDAR sensor information DB (121) for storing information on the location where the lidar sensor 210 is installed and the branch office;

Monitoring camera information DB (122) for storing information on the location where the monitoring camera 220 is installed and the branch office;

a collection information DB (123) for storing collection information obtained by collecting sensor information output from the lidar sensor and image information output from the monitoring camera 220;

Analysis information DB (124) for storing the analysis information obtained by analyzing the collected information;

It includes a contact information DB 125 for storing the contact information and terminal information of the person in charge.

The server 110,

an information collection unit 111 that collects image information and sensor information output from the lidar sensor 210 and the monitoring camera 220;

an analysis unit 112 that analyzes the collected image information and sensor information to calculate a congestion index for measuring the congestion level of the branch, and calculates customer satisfaction by analyzing the collected image information and sensor information;

An alarm unit 113 that determines whether there is an abnormal behavior of the customer and, if there is an abnormal behavior of the customer, notifies the terminal 300 of the person in charge.

The server 110 detects customers after learning environmental data in the branch and provides real-time congestion notification service.

The server 110 determines the number of customers for each zone in the branch and calculates the congestion level of the branch through visualization, and separates and detects employees and customers.

In addition, the server 110 solves customer problems that are difficult to detect by customizing the same customer in multiple cameras 220 through calibration of multiple surveillance cameras 220 and lidar sensors 210, and The purpose of the visit can be analyzed.

In addition, the server 110 learns branch environment data to define abnormal behavior patterns in the branch and prevent crime. In addition, facility safety, customer safety, environmental management monitoring, and voice phishing crime detection notification system prevent crime prevention and real-time damage.

In addition, the server 110 may intuitively monitor the environment of the branch by utilizing 3D visualization technology and display it on the terminal 300 in charge.

In addition, the server 110 monitors the congestion status of the branch office and the pattern of customers visiting the branch office in real time, and enables real-time monitoring through notification for each customer congestion level, movement line, and abnormal behavior.

The congestion index is

Calculate the average number of waiting customers and waiting time,

Standardize the calculated average number of waiting customers and waiting time,

After scoring the standardized waiting customer standardized value and waiting time standardized value as 100 points,

It is characterized in that the congestion index is calculated.

The customer satisfaction level is

Calculate customer satisfaction index through waiting time, facial expression and behavior.

As a result of customer experience management surveys within financial institutions, customer perceived congestion, which is a satisfaction index for the waiting situation when a customer visits the store, has an inverse correlation between customer perceived congestion and customer experience index. ) is characterized by an increase in

FIG. 2 is a diagram showing the configuration of a manager terminal of FIG. 1 .

Referring to FIG. 2 , the manager terminal 300 includes an input unit 320, a storage unit 360, a camera unit 370, a communication unit 350, a control unit 330, and a display unit 310.

The display unit 310 displays various screen information according to selection of the input unit 320 .

The input unit 320 may include an input device such as a touch panel and a keypad, and is used by a user to select a menu.

The storage unit 360 may store an app for using the service, and may also store photographing information captured by the camera unit.

The camera unit 370 takes pictures according to the selection of the input unit 320 .

The control unit 330 displays the contents received from the server 110 on the display unit 310, transfers the information input by the input unit 320 to the server 110, and executes an app.

In addition, the control unit 330 receives an alarm signal from the server 110, displays it on the display unit 310, and allows a person in charge to take action accordingly.

The communication unit 350 transmits and receives information under the control of the control unit 330, and may use various wireless Internet or other wireless communication networks such as Bluetooth, Wi-Fi, 3G, and 4G.

The person in charge terminal 300 may be a computer, smart phone, laptop, tablet PC, or the like.

The operation of the image and LIDAR sensor-based image analysis system according to an embodiment of the present invention having such a configuration will be described as follows.

3 is a diagram illustrating an image analysis method based on an image and lidar sensor according to an embodiment of the present invention.

Referring to FIG. 3 , at least one lidar sensor 210 and surveillance camera 220 installed in a business branch monitor a corresponding area and output sensor information and video signals. At this time, the lidar sensor 210 monitors and outputs the movement of customers in the branch, and the monitoring camera 220 photographs and outputs images of employees and customers in the branch.

Then, the information collection unit 111 of the server 110 collects sensor information and image information output from the lidar sensor 210 and the monitoring camera 220, and collects information DB 123 of the database unit 120. stored in (S310).

At this time, the information collection unit of the server 110 may refer to the lidar sensor information DB 121 and the monitoring camera information DB 122 to index and store sensor information and image information at which location of which branch.

Next, the analyzer 112 of the server 110 analyzes the collected image information and sensor information to calculate a congestion index for measuring the degree of congestion of the branch (S320).

Here, the congestion index is an index for measuring the degree of congestion of a branch office and is calculated by separating customers waiting for work (customers with IQS issued) and customers waiting for non-work (customers without IQS issued).

Business customer congestion index = (business) scored waiting customers_standardized value * 0.5 + scored waiting time_standardized value * 0.5

Non-business customer congestion index = (non-business) scored waiting customers_standardized value * 0.5 + scored waiting time_standardized value * 0.5

At this time, non-business customers are calculated when they are customers who have not issued an IQS and are in the waiting space (Zone area).

First, using Equation 1 below, the average number of (business ㅇ non-business) waiting customers and average waiting time are calculated.

Next, the calculated average (business ㅇ non-business) number of waiting customers and average waiting time are each standardized as shown in Equation 2.

Next, the standardized average number of waiting customers (ZVL average waiting time) and average waiting time (ZVL average waiting time) are scored.

First, the scoring of the standardized value of average waiting customers is as follows.

1) When ZVL average waiting customers > 3, 100

2) When ZVL average waiting customers < -3, 0

3) And except for the above two cases, it is as follows.

Next, first, the scoring of the average waiting time standardized value is as follows.

1) When ZVL average waiting time > 3, 100

2) When ZVL average waiting time < -3, 0

3) And except for the above two cases, it is as follows.

Next, the analyzer 112 of the server 110 analyzes the collected image information and sensor information to calculate customer satisfaction (S330).

Here, the customer satisfaction index is calculated through waiting time, facial expression, and behavior. In particular, the satisfaction level of business customers is measured once after work is over and for non-business customers at the last waiting area before leaving the store.

Customer satisfaction is calculated in the following five stages.

In the first step, calculate the waiting time for each (business/non-business) customer.

Calculate the customer waiting time_normalized value by performing Min-Max Scaling (normalized scaling on a daily basis) in the range of 0 to 1 for the waiting time of business/non-business customers per person (per person).

In step 2, the emotion reliability value is calculated as shown in Equation 3 below with reference to the reliability score of FIG. 4 .

Then, Min-Max Scaling (normalized scaling on a daily basis) is performed on the calculated emotion reliability value in the range of 0 to 1 to calculate the emotion score.

In the next three steps, the abnormal behavior score is calculated as shown in Equation 4.

Here, an example of the abnormal behavior is shown in FIG. 5, and there are abnormal behavior such as voice phishing for crime prevention, abnormal behavior such as falling down for customer safety, and abnormal behavior such as finger pointing for employee safety.

In the next 4 steps, work customer satisfaction and non-business customer satisfaction are calculated respectively.

Business customer satisfaction = ((1-(Work) customer waiting time_normalized value) * 0.4 + emotional score * 0.3 + abnormal behavior score * 0.3) ?? 100

Non-business customer satisfaction = ((non-business) customer waiting time_normalized value) * 0.4 + emotion score * 0.3 + abnormal behavior score * 0.3) ?? 100

In the following five steps, customer satisfaction and employee satisfaction are calculated as shown in Equations 5 and 6, respectively.

The server 110 may display the congestion index calculated in this way on the person in charge terminal 300, or it may be stored in the analysis information DB 124 of the database unit 120 and provided upon request.

In addition, the server 110 calculates the customer's perceived congestion as needed.

Here, customer perceived congestion is a satisfaction index for the waiting situation when a customer visits the store. As a result of the customer experience management survey in the row, congestion and customer experience index have an inverse correlation, and the less congested the customer experience (satisfaction index) is. An increasing correlation can be observed. At this time, as shown in FIG. 6 or 7, a customer satisfaction index for the waiting experience may be measured through a customer experience survey related to visiting or waiting in a branch, and the corresponding satisfaction index may be defined as a customer perceived congestion.

In addition, as shown in FIG. 8, the server 110 may send a survey to mobile (KakaoTalk) on the next business day after visiting the branch to conduct a customer satisfaction survey.

At this time, the index for each question item can be calculated as in Equation 6 below.

The table in FIG. 9 summarizes the customer experience congestion level survey, evaluation reflection, questionnaire, and other matters.

Then, the alarm unit 113 of the server 110 determines whether there is an abnormal behavior of the customer (S340).

For example, it may be determined that the abnormal behavior score is greater than or equal to a reference score.

When there is an abnormal behavior of the customer, the alarm unit 113 of the server 110 refers to the person in charge information DB 125 and sends an alarm to the person in charge terminal 300 (S350). In other words, since the customer's abnormal behavior is detected by voice phishing, etc., an alarm is issued to take action.

In an embodiment of the present invention, object detection and monitoring through branch image analysis may be utilized. In other words, big data analysis generated by the branch video analysis system can be used for existing financial services, and the accuracy of video analysis can be verified in preparation for queue tables in which actual customers cannot accurately count, and insights to improve branch congestion through convergence analysis video security and management system through branch operation monitoring.

In addition, in the embodiment of the present invention, it is possible to utilize object tracking monitoring through branch image analysis. In other words, it can be used for existing financial services through the analysis of big data generated by the branch image analysis system, and it is possible to analyze the effects of PG2.0 and digital self-stores by deriving the purpose of visiting the branch by identifying the customer movement, and by identifying the customer movement You can analyze the types of customers who churn (e.g., customers who churn with an average wait time of more than a few minutes).

In addition, in an embodiment of the present invention, behavior recognition monitoring through branch image analysis may be utilized. In other words, it can be used for existing financial services through analysis of big data generated by the branch office video analysis system, and crime prevention, facility safety, customer safety, and environmental management monitoring are possible by utilizing the branch office CCTV analysis and monitoring system.

In addition, in the embodiment of the present invention, it is possible to verify the process in the branch for reducing perceived congestion by using the actual/intuitive congestion calculated from the analyzed customer data from the moment of entering the branch to the time of processing and leaving the branch.

In addition, in the embodiment of the present invention, it is possible to rearrange furniture and DID according to the customer's moving line for each branch situation.

In addition, in the embodiment of the present invention, it is possible to promptly respond to congestion in the branch with real-time congestion notification, it is possible to prevent crime through monitoring the operation of the branch, and it is possible to monitor facility safety, customer safety and environmental management. .

The embodiments of the present invention described above are not implemented only through devices and methods, and may be implemented through a program that realizes functions corresponding to the configuration of the embodiments of the present invention or a recording medium on which the program is recorded. Implementation can be easily implemented by an expert in the technical field to which the present invention belongs based on the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. that fall within the scope of the right.

Claims (7)

An image analysis method of a video analysis system connected to at least one lidar sensor and surveillance camera installed in a branch through a network,
Collecting, by a server, image information and sensor information output from the lidar sensor and surveillance camera;
calculating, by the server, a congestion index for measuring the degree of congestion of the branch by analyzing the collected image information and sensor information;
Calculating customer satisfaction by analyzing the collected image information and sensor information by the server;
determining, by the server, whether there is an abnormal behavior of the customer;
An image analysis method based on an image and a lidar sensor comprising the step of making an alarm by the server when there is an abnormal behavior of the customer.
According to claim 1,
The congestion index is
Calculate the average number of waiting customers and waiting time,
Standardize the calculated average number of waiting customers and waiting time,
After scoring the standardized waiting customer standardized value and waiting time standardized value as 100 points,
Image and lidar sensor-based image analysis method, characterized in that for calculating the congestion index.
According to claim 2,
The customer satisfaction level is
An image analysis method based on image and lidar sensor that calculates customer satisfaction index through waiting time, facial expression and behavior.
A video analysis system connected through a network with at least one lidar sensor, surveillance camera, and staff terminal installed in a branch,
A database unit for storing lidar sensor information, monitoring camera information, sensor information and image information collection information, analysis information, and person in charge information;
Collect image information and sensor information output from the lidar sensor and surveillance camera, analyze the collected image information and sensor information to calculate a congestion index to measure the congestion level of the branch, and calculate the collected image information and sensor information An image analysis system based on image and lidar sensors including a server that analyzes and calculates customer satisfaction.
According to claim 4,
The server determines whether there is an abnormal behavior of the customer, and if there is an abnormal behavior of the customer, the server alarms the terminal in charge of the video and LIDAR sensor-based image analysis method.
According to claim 5,
The database unit,
Lidar sensor information DB that stores information on the location where the lidar sensor is installed and the branch office;
Surveillance camera information DB that stores information on locations and branches where surveillance cameras are installed;
a collection information DB for storing collection information obtained by collecting sensor information output from the lidar sensor and image information output from the monitoring camera;
an analysis information DB for storing analysis information obtained by analyzing the collected information;
An image analysis system based on images and lidar sensors including a contact information DB for storing contact information and terminal information of a person in charge.
According to claim 6,
The server,
an information collection unit that collects image information and sensor information output from the lidar sensor and surveillance camera;
an analysis unit that analyzes the collected image information and sensor information to calculate a congestion index for measuring the congestion level of the branch, and analyzes the collected image information and sensor information to calculate customer satisfaction;
An image and LIDAR sensor-based image analysis system including an alarm unit that determines whether there is an abnormal behavior of the customer and, if there is an abnormal behavior of the customer, an alarm unit that notifies the terminal of the person in charge.

Images