JP6744652B1 - Congestion information notification system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To complete the privacy protection of a user and to easily grasp the congestion degree intuitively in a system for informing the user of the congestion degree of various facilities such as stores, restaurants, amusement parks, public offices and banks. To achieve both. SOLUTION: An image capturing unit 10 that captures an area to be monitored, a person identifying unit 21 that identifies a person 101 in a captured image 100 captured by the image capturing unit 10, and skeletal information of the person 101 identified by the person identifying unit 21. The skeleton information extraction unit 22 that extracts 101A, the image processing unit 24 that performs image processing so that the person 101 in the captured image 100 is displayed with only the skeleton information 101A, and the privacy that has been subjected to image processing by the image processing unit 24. The congestion information notification system 1 includes the information transmission unit 25 that transmits the protection-processed image 100A as congestion information. [Selection diagram] Figure 1

Description

The present invention relates to a congestion information notification system. More specifically, the present invention relates to a system for notifying a user of the degree of congestion of various facilities such as stores, restaurants, amusement parks, public offices, and banks.

In order to contribute to measures against infectious diseases and the like, there is a demand for a means for easily informing, in real time, facility users and prospective users of congestion information in various facilities used by the public such as stores.

There is already known a system that detects the degree of congestion in a monitoring area from a monitoring image and notifies the monitoring person (for example, Patent Document 1 ). It is possible to notify the congestion status by transmitting numerical information relating to the congestion degree obtained by the system described in Patent Document 1 to the customer.

In order for the users of the facility to intuitively understand the degree of congestion, it is effective to show not only numerical information but also an image showing the congestion inside the facility. Is. In the conventional congestion information notification system, the observer can also check the image of the actual monitoring target area, but disclosing such an image to an unspecified number of users is a privacy protection request. It is unforgivable because it is against it.

To solve this problem, a process of reducing the distinguishability of the background image in order to protect the privacy of an individual, a process of generating a mask image, and a three-step image process of synthesizing the background image and the mask image are obtained. There has been proposed a system (Patent Document 2 ) for exposing the captured image to the user.

JP, 2018-42049, A JP, 2017-98879, A

The image for notifying the degree of congestion provided by Patent Document 3 requires complicated image processing, and therefore an expensive image processing device is essential. In addition, even if the masking process is performed, many users who actually use the facility feel a sense of resistance that the video of themselves is disclosed through public lines, etc. In many cases, it is practically difficult to publish such a video against the user's will.

It is a system that can be introduced at a low cost using existing surveillance cameras, etc., and allows users of the facility to intuitively grasp the degree of congestion of the facility with images, and Regarding the aspect of protecting the privacy of the user of the facility, which may also be a shooting target in the image, a means for notifying the congestion degree that can dispel the anxiety of the user has been sought.

The present invention is a system for informing a user of the congestion degree of various facilities such as a store, a restaurant, an amusement park, a public office, a bank, etc., and complete integrity of the privacy protection of the user and an intuitive grasp of the congestion degree The purpose is to achieve both sex.

The present inventor has thought that by using the skeletal information extraction technology from images, which has been developed in recent years, it is possible to achieve both complete privacy protection of the user and ease of intuitive grasp of the degree of congestion. The present invention has been completed. The present invention specifically solves the above problems by the following solving means.

(1) An image capturing unit that captures an area to be monitored, a person identifying unit that identifies a person in a captured image captured by the image capturing unit, and a skeleton that extracts skeleton information from an image of the person identified by the person identifying unit. An information extraction unit, an image processing unit that performs image processing so as to display a person in the captured image only with skeleton information, and generates a privacy-protected processed image, and a privacy protection-processed image generated by the image processing unit. A congestion information notification system, comprising: an information transmission unit that transmits an image as congestion information.

According to the congestion information notification system of (1), in a system that notifies users of the congestion levels of various facilities such as stores, restaurants, amusement parks, public offices, and banks, the integrity of the privacy protection of users and It is possible to achieve both ease of intuitive grasp of the degree of congestion.

(2) The congestion information notification system according to (1), further including a congestion degree calculation unit that calculates the congestion degree of the monitoring target area from the number of persons identified by the person identification unit.

According to the congestion information notification system of (2), the monitoring image is processed into the congestion information including the privacy-protected processed image that can be intuitively grasped easily, and at the same time, the monitoring image is used to objectively measure the congestion degree. Numerical data etc. can be obtained.

(3) The congestion information notification system according to (2), wherein the information transmission unit transmits, as the congestion information, information obtained by combining the privacy protection processed image and the character information indicating the congestion degree. ..

According to the congestion information notification system of (3), the privacy-protected image and the objective numerical data obtained in the congestion information notification system of (2) are combined to be intuitively easy to grasp, and It is possible to send congestion information with objectivity.

(4) In the privacy-protected image, grid lines are displayed that indicate the correspondence between the length in the captured image and the actual distance in the monitoring target area. (1) to (3) The congestion information notification system according to any one.

According to the congestion information notification system of (4), the system user can easily grasp the actual distance between people from the privacy-protected processed image forming the congestion information.

(5) An image capturing step in which the image capturing section captures an image of a monitoring target area, a person identifying section identifies a person in a captured image captured by the image capturing section, and a skeleton information extracting section identifies the person. A skeleton information extraction step of extracting skeleton information from the image of the person identified by the section, and the image processing section performs image processing so as to display only the skeleton information of the person in the photographed image, and the privacy protection processed image And a congestion information transmitting step in which the information transmitting unit transmits the privacy protection processed image as congestion information, the congestion information notifying method.

According to the congestion information notifying method of (5), in a system for notifying the user of the degree of congestion of various facilities such as stores, restaurants, amusement parks, public offices, and banks, the integrity of the privacy protection of the user and It is possible to achieve both ease of intuitive grasp of the degree of congestion.

(6) The congestion information notification method according to (5), wherein the congestion degree calculation unit further includes a congestion degree calculation step of calculating the congestion degree of the monitoring target area from the number of persons identified by the person identification unit.

According to the congestion information notification method of (6), the monitoring image is processed into the congestion information including the privacy-protected processed image that can be intuitively grasped easily, and at the same time, the monitoring image can be used to objectively measure the congestion degree. Numerical data etc. can be obtained.

(7) In the congestion information transmitting step, the congestion information notification according to (6), which transmits, as the congestion information, information obtained by combining the privacy protection processed image and the character information indicating the congestion degree. Method.

According to the congestion information method of (7), the privacy-protected image is combined with the objective numerical data obtained in the congestion information notification method of (6) to be intuitively easy to understand and objective. It is possible to send congestion information that has a certain nature.

(8) From (5) to (7), a grid line is displayed in the privacy-protected image, the grid line indicating the correspondence between the length in the captured image and the actual distance in the monitoring target area. ) The congestion information notification method according to any one of 1).

According to the congestion information notification method of (8), the user of the method can easily grasp the actual distance between people from the privacy-protected processed image forming the congestion information.

(9) In the congestion information notification method according to any one of (5) to (8), the person identification step, the skeleton information extraction step, and the image processing step are performed by the person identification unit and the skeleton information extraction. And a program for notifying congestion information, which is executed by an arithmetic processing unit including the image processing unit.

According to the program of (9), in the system that informs the user of the congestion degree of various facilities such as stores, restaurants, amusement parks, public offices, and banks, the integrity of the privacy protection of the user and the congestion degree It is possible to achieve both intuitive graspability.

According to the present invention, in a system that informs the user of the congestion degree of various facilities such as stores, restaurants, amusement parks, public offices, banks, etc., the completeness of the privacy protection of the user and the intuitive grasp of the congestion degree It is possible to achieve both ease of use.

It is a block diagram which shows the structure of the congestion information alerting|reporting system of this invention. It is a figure which shows an example of the captured image which concerns on the congestion information alerting|reporting system of this invention. It is a figure which shows an example of the privacy protection processed image which concerns on the congestion information alerting|reporting system of this invention. It is a figure which shows an example of the congestion information displayed on a user's information receiving terminal by the congestion information alerting|reporting system of this invention.

Hereinafter, the best mode for carrying out the present invention will be described.

<Congestion information notification system>
FIG. 1 is a block diagram showing a configuration of a congestion information notification system 1 of the present invention. It should be noted that each of the following drawings including FIG. 1 is a schematic diagram, and the size and shape of each portion are appropriately changed and shown for easy understanding. Further, in the following description, a specific configuration is illustrated as an example, but these can be appropriately changed.

[overall structure]
The congestion information notification system 1 includes a photographing unit 10 and a calculation processing unit 20. The information transmitted from the congestion information notification system 1 is transmitted to the information receiving terminal 30 owned by the user of the system.

In the following, the congestion information notification system 1 of the present invention will be described in detail as an example of a preferred embodiment of the congestion information notification system 1 having a configuration in which the imaging unit 10 and the arithmetic processing unit 20 are separately arranged at separate positions. explain. However, in the present invention, the image capturing unit 10 and the arithmetic processing unit 20 may be integrated into a single congestion information notification system, or only part of the arithmetic processing unit 20 may be captured. It can also be configured to be installed in.

The congestion information notification system 1 is an information processing system in which the above-described components are connected to each other so that they can communicate with each other. The connection between the respective units may be a wired connection using a dedicated communication cable or a wired LAN connection. Further, the congestion information notifying system 1 can be configured not only by wired connection but also by connection using various wireless communication such as wireless LAN, short-distance wireless communication, and mobile phone line.

[Photographing section]
The image capturing unit 10 is a so-called surveillance camera that captures an area to be monitored for monitoring the degree of congestion in various facilities such as stores, restaurants, amusement parks, public offices, and banks. The image capturing unit 10 can be configured by various conventionally known monitoring cameras capable of continuously capturing the monitoring target area. Further, the surveillance camera constituting the imaging unit 10 can be constituted by a 3D camera which can directly obtain the three-dimensional position information in the space to be the surveillance target area, or the three-dimensional space to be the surveillance target area is two-dimensional. It can also be configured with a monocular camera that captures an image of.

In the congestion information notification system 1, when the image capturing unit 10 is a monocular camera capable of capturing a monitoring image having only two-dimensional information, the congestion information notification system 1 determines the position in the monitoring image as a three-dimensional monitoring target area. It is preferable to further include a coordinate setting unit (not shown in the block diagram) that sets identifiable coordinates in association with the actual position in the space. If the image capturing unit 10 is a 3D camera that can directly obtain three-dimensional position information in the space to be monitored, the coordinate setting unit in the above aspect is not essential. The coordinate setting unit may be incorporated in the image capturing unit 10 or may be additionally provided as a part of the arithmetic processing unit 20 as a separate device.

Here, FIG. 2 is a diagram illustrating an example of a captured image 100 captured by the capturing unit 10. The photographing unit 10 has a function of processing the photographed image 100 into image data in a digital format so that the arithmetic processing unit 20 can perform arithmetic processing, and outputting the image data to the arithmetic processing unit 20. If so, various existing digital cameras can be used without particular limitation.

[Arithmetic processing section]
The arithmetic processing unit 20 performs various arithmetic processes necessary for creating congestion information on the image data output and transmitted from the photographing unit 10. The arithmetic processing unit 20 can be configured using, for example, a personal computer, a tablet terminal, a smartphone, or the like. Alternatively, the arithmetic processing unit 20 can be configured as a dedicated device specialized for the operation according to the present invention. Further, the arithmetic processing unit 20 can be configured such that the information processing apparatus having the above functions is configured as a server on the cloud and is shared by a large number of clients.

In any of the configurations described above, the arithmetic processing unit 20 includes hardware such as a CPU, a memory, and a communication unit. The arithmetic processing unit 20 having such a configuration executes the congestion information notification program, which is the computer program according to the present invention, to concretely perform the various operations described below and the congestion information notification method. Can be run to.

The arithmetic processing unit 20 is connected to the photographing unit 10 so that the image data can be received from the photographing unit 10. This connection may be a wired connection using a dedicated communication cable or a wired LAN connection. Further, the connection is not limited to the wired connection, but may be a connection using various kinds of wireless communication such as wireless LAN, short-range wireless communication, and mobile phone line. It should be noted that the arithmetic processing unit 20 may be arranged not in the vicinity of the photographing unit 10 but in a remote place apart from the photographing unit 10.

The arithmetic processing unit 20 includes at least a person identifying unit 21, a skeletal information extracting unit 22, an image processing unit 24, and an information transmitting unit 25. Moreover, it is preferable that the arithmetic processing unit 20 further includes a congestion degree calculation unit 23. A coordinate setting unit may be provided as necessary.

(Person identification part)
The person identifying unit 21 automatically identifies and detects a person from the captured image 100 captured by the capturing unit 10. Specifically, this person can be identified by any of various conventionally known methods or a combination thereof. For example, an object other than the background can be extracted from the captured image 100 based on the background difference. This background difference is a known technique, and by taking the difference between the image data of the captured image captured by the capturing unit 10 and the background image of the capturing target area acquired in advance, a moving object Is a technology for extracting.

In addition, regarding the identification of the extracted monitoring target, a machine learning type image analysis apparatus having a neural network (a so-called “deep learning technology It is preferable to use a conventional image recognition device”).

The image recognition technology using deep running is disclosed, for example, in the following.
"Deep Learning and Image Recognition, Operations Research"
(Http://www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf)

By these various known image recognition techniques, it is possible to automatically identify a "person" in a captured image with an extremely high accuracy rate.

(Skeletal information extraction unit)
The skeleton information extraction unit 22 extracts corresponding skeleton information 101A, 102A,..., 105A from the persons 101, 102,..., 105 in the captured image 100. The skeletal information 101A, 102A,..., 105A is a plurality of characteristic points corresponding to the joint parts of the head, both shoulders, hips, hands, and legs of the people 101, 102,. Is image information composed of line segments connecting (see FIG. 3).

Specifically, the extraction of the human skeleton information as shown in FIG. 3 can be performed by any one of various conventionally known methods or a combination thereof. As an example, the skeleton information of “person” can be extracted from a two-dimensional image by using a technique called “OpenPose” disclosed in the following document. In addition, by analyzing the position and velocity of each of these feature points, it is possible to recognize the position and movement of a person "Zhe Cao et al. Realtime Multi-Person 2D Human Posse Estimating Part Affinity Fields, CVPR 2017".

(Crowding degree calculator)
The congestion degree calculation unit 23 calculates the congestion degree in the area based on the number of persons identified by the person identification unit 21. The degree of congestion, that is, the number of people in each arbitrary partial area, is, for example, the “actual position” occupied by the people 101, 102,..., 105 in the monitored area, and the skeleton information 101A, 102A,. ..By grasping the characteristic points constituting 105A as the positions (three-dimensional position information) occupied on the coordinates set by the coordinate setting unit described below, and analyzing the three-dimensional position information, objective numerical values can be obtained. It can be obtained as data.

(Image processing unit)
The image processing unit 24 performs image processing on the captured image 100 (FIG. 2) to generate a privacy protection processed image 100A (FIG. 3). As shown in FIG. 3, the privacy-protected processed image 100A corresponds to the skeleton information 101A, 102A,..., 105 extracted by the skeleton information extraction unit 22 from the persons 101, 102,..., 105 in the captured image 100. ·············································································S “105A”.

Further, it is more preferable that a grid line (not shown) that displays the correspondence between the length in the captured image and the actual distance in the monitored area is further displayed in the privacy-protected image. The grid line may be any line as long as the actual distance in the monitoring target area can be grasped by the length of the line segment shown in the image. Thereby, the user of the system can easily grasp the actual distance between the person displayed as the skeleton information there from the privacy protection processed image. In addition, by actually indicating a partial area occupying a predetermined unit area in the monitored area by a grid line, the congestion degree (density) of the partial area can be displayed as visual information that is easier to grasp. ..

(Information dispatch department)
The information transmitting unit 25 transmits the privacy-protected processed image 100A, which has been subjected to the above-described image processing by the image processing unit 24, as information for notifying the user of the system of the congestion information of the facility. It should be noted that, here, the assumed users of the system are the person in charge of management of the facility, and all potential users who are considering using the facility. However, if necessary, the congestion information may be disclosed only to a specific authenticated customer group.

The congestion information transmitted by the information transmission unit 25 is preferably information in which the privacy protection processed image 100A and the character information 200 indicating the congestion degree are combined. This textual information 200 relates to the information such as "I'm busy" and "I'm vacant" that sensually indicates the degree of congestion, and the actual number of people staying in a specific area (density). The information including numerical information can be generated based on the data obtained by the congestion degree calculation unit 23. Then, the congestion information generated in this way is preferably displayed on the information receiving terminal 30 of the user of the system such as a smartphone as shown in FIG. 4, for example.

(Coordinate setting section)
In addition, for example, by further providing a coordinate setting unit as disclosed in Japanese Patent No. 6581280 in the arithmetic processing unit 20, the position corresponding to the floor surface in the photographed image photographed by the photographing unit 10 is associated with the actual size. It is also possible to set identifiable coordinates. The coordinates set by the coordinate setting unit means that when a certain arbitrary position is specified in the captured image captured by the capturing unit 10 and the position is on the floor surface, the floor surface is the actual one. It is a coordinate with which it is possible to specify which position in the space of the monitoring area corresponds. That is, the set coordinate position is set in association with the actual size. By providing such a coordinate setting unit, even from an image having only two-dimensional information acquired only by a monocular camera that can be acquired inexpensively without introducing a distance measuring device or a 3D camera, etc. It is possible to efficiently identify a person with high accuracy only by performing simple processing.

[Information receiving terminal]
The information receiving terminal 30 is not particularly limited as long as it is a device or device that allows the user of the system to receive the congestion information transmitted by the information transmitting unit 25, and various devices possessed or lent by the user. Can be used. The information receiving terminal 30 is specifically a personal computer with a monitor, or a small portable information processing terminal that can be easily carried, and the like so that the user of the system can recognize the congestion information. It can be configured by various information processing terminals that can be appropriately displayed with various images, characters, sounds, or the like, or various information display devices including a monitor or the like that can respond to such requests.

<Congestion information notification method>
Preferably, the congestion information notification method of the present invention, which can be executed by the operation of the congestion information notification system 1 of the present invention, includes a photographing step executed by the photographing unit 10, a person identification step executed by the person identifying unit 21, The skeleton information extraction step executed by the skeleton information extraction unit 22, the congestion degree calculation step executed by the congestion degree calculation unit 23, the image processing step executed by the image processing unit 24, and the information transmission unit 25. The congestion information transmitting step is sequentially performed, so that the entire process is performed. However, in the case of the minimum configuration, the congestion information transmission step is not an essential configuration. The congestion information notification method of the present invention can also be implemented as an aspect in which only the privacy protection processed image 100A generated in the image processing step is transmitted as congestion information.

(Shooting step)
In the shooting step, the shooting unit 10 shoots the monitoring target area. Here, in this photographing, still images are continuously photographed at a predetermined interval, and a monitoring operation described later is performed as a sequence of images to be photographed. However, by substantially shortening the photographing interval, substantially, It can also be considered that a surveillance operation is being performed as video shooting.

(Person identification step)
In the person identifying step, the person identifying unit 21 identifies the person in the captured image captured by the capturing unit 10. If the person is identified, the process proceeds to the skeleton information extraction step or the congestion degree calculation step. If the person is not identified, the process returns to the imaging step and the monitoring operation is continued.

(Skeletal information extraction step)
In the skeleton information extraction step, the skeleton information extraction unit 22 extracts the corresponding skeleton information 101A, 102A,..., 105A from the persons 101, 102,..., 105 (see FIG. 2) identified in the person identification step. (FIG. 3) is extracted.

(Congestion degree calculation step)
In the congestion degree calculation step, the congestion degree calculation unit 23 specifies the number of persons identified in the person identification step, and calculates the congestion degree of the monitoring target area from the numerical data relating to the number of people staying per unit area. The criterion for determining the congestion degree here can be arbitrarily changed according to the business type, business type, place, etc. of the target facility to which the present method is applied. As an example, as an initial setting, when three or more persons are identified in an area having an arbitrary unit area (for example, 5 m ² ), it is determined that the area is congested. It is possible to exemplify an embodiment in which a judgment criterion as a setting is set and the unit area and the number of people are appropriately adjusted as needed .

(Image processing step)
In the image processing step, the image processing unit 24 sets the person 101, 102,..., 105 (see FIG. 2) in the captured image 100 to the corresponding skeleton information 101A, 102A,..., 105A (FIG. 3). The privacy-protected image 100A is generated by performing image processing so that it is displayed only.

(Congestion information transmission step)
In the congestion information transmitting step, the information transmitting unit 25 transmits the privacy protection processed image 100A generated in the image processing step as congestion information. In addition, it is preferable that the congestion information is information (see FIG. 4) obtained by combining the privacy-protected image 100A and the character information 200 indicating the congestion degree calculated in the congestion degree calculation step.

In addition, a machine learning type information analysis device (a so-called “information analysis device using a so-called “deep learning technology” (population) that has accumulated neural network congestion information for past months, days, days, hours, and weather) It is also possible to train the intelligence)) to obtain the congestion prediction data after the present point of time, and transmit this congestion prediction data together with the above-mentioned congestion data. Further, other sales promotion information and the like can be transmitted together with the congestion information.

Further, it is more preferable to display a grid line on the privacy-protected image 100A, which indicates the correspondence between the length in the captured image 100 and the actual distance in the monitored area.

After executing the congestion information transmitting step, the arithmetic processing unit 20 determines whether or not to end the monitoring. The termination of monitoring is determined to be terminated, for example, when a command to terminate monitoring is input. If it is ended, the monitoring operation is ended, otherwise, the process returns to the photographing step and the monitoring is continued.

DESCRIPTION OF SYMBOLS 1 Congestion information notification system 10 Imaging part 20 Arithmetic processing part 21 Person identification part 22 Skeleton information extraction part 23 Congestion degree calculation part 24 Image processing part 25 Information transmission part 30 Information receiving terminal 100 Photographed image 101, 102, 103, 104, 105 People 100A Privacy protection processed images 101A, 102A, 103A, 104A, 105A Skeleton information 111 Background 200 Character information

Claims (7)

An imaging unit that captures the monitored area,
A person identifying section for identifying a person in a captured image captured by the capturing section;
A skeleton information extraction unit that extracts skeleton information from the image of the person identified by the person identification unit;
A congestion degree calculation unit that calculates a congestion degree of a partial region occupying a predetermined unit area in the monitoring target region from the number of persons identified by the person identification unit,
Corresponding to the length in the photographed image and the actual distance in the monitoring target area, a grid line indicating the partial area is displayed in the photographed image, and the person in the photographed image is represented by skeleton information only. An image processing unit that performs image processing to display and generates a privacy-protected processed image,
An information transmitting unit that transmits, as congestion information, information in which the privacy protection-processed image generated by the image processing unit and text information indicating the congestion degree are combined ,
Congestion information notification system. The information transmission unit is the congestion after the present time obtained by making the machine learning type information analysis device having the neural network learn the data that accumulates the congestion information for each of the past month, day, day of the week, time, and weather. Transmitting prediction data together with the congestion information,
The congestion information notification system according to claim 1. The information transmission unit transmits sales promotion information together with the congestion information,
The congestion information notification system according to claim 1. A photographing step in which the photographing unit photographs the monitoring target area;
A person identifying section for identifying a person in a captured image captured by the capturing section;
A skeleton information extraction unit, a skeleton information extraction step of extracting skeleton information from the image of the person identified by the person identification unit;
A congestion degree calculation unit, a congestion degree calculation step of calculating the congestion degree of a partial region occupying a predetermined unit area in the monitoring target region from the number of persons identified by the person identification unit,
The image processing unit corresponds to the length in the captured image and the actual distance in the monitoring target area, displays grid lines indicating the partial areas in the captured image, and An image processing step of generating a privacy protection processed image by performing image processing so as to display only the skeleton information,
An information transmitting unit transmits, as the congestion information, information obtained by combining the privacy protection processed image and the character information indicating the congestion degree , the congestion information transmitting step,
Congestion information notification method. In the congestion information transmitting step, the information transmitting unit causes a machine learning type information analysis device having a neural network to learn data in which congestion information for each of past months, days, days of the week, time, and weather is accumulated. The congestion forecast data after the present obtained by sending together with the congestion information,
The congestion information notification method according to claim 4. In the congestion information transmitting step, the information transmitting unit transmits the sales promotion information together with the congestion information,
The congestion information notification method according to claim 4 or 5. The congestion information notification method according to any one of claims 4 to 6 ,
The person identification step, the skeleton information extraction step , the image processing step , and the congestion degree calculation unit step ,
A congestion information notification program to be executed by an arithmetic processing unit including the human identification unit, the skeleton information extraction unit , the image processing unit , and the congestion degree calculation unit .

Images