WO2020213058A1

WO2020213058A1 - Digital smart defense security system, method, and program

Info

Publication number: WO2020213058A1
Application number: PCT/JP2019/016324
Authority: WO
Inventors: 和夫三輪
Original assignee: 株式会社テクノミライ
Priority date: 2019-04-16
Filing date: 2019-04-16
Publication date: 2020-10-22
Also published as: JPWO2020213058A1

Abstract

Provided are a digital smart defense security system, method, and program capable of outputting a quick and appropriate alarm in accordance with the position of a person within a security zone. A digital smart defense security system (1000) comprises: a person detection unit (111) which detects a person present in a security zone in which a plurality of regions are set to correspond to degrees of risk; a reporting unit (115) which warns the person detected by the person detection unit (111), using a sound corresponding to the region in which the person is present, and reports the presence of that suspicious person; and a speech recognition unit (112) which acquires conversational speech and determines whether the content of the conversation is suspicious. If the speech recognition unit (112) determines that the content of the conversation is suspicious, the reporting unit (115) reports the presence of the suspicious person.

Description

Digital Smart Defense Security Systems, Methods and Programs

The present invention relates to a digital smart defense security system, a method and a program for taking a picture of a security area with a surveillance camera to prevent crime.

Patent Document 1 states, "As an alarm device for detecting a picking action, a picking action detection sensor (vibration sensor) detects a vibration sound due to the picking action, and when such a vibration sound is detected, an alarm is given. Was output to call the attention of those around (paragraph 0002). " The technique of Patent Document 1 relates to an alarm device that outputs an alarm through a built-in speaker when a picking action is detected.

Japanese Unexamined Patent Publication No. 2005-143004

In the alarm device described in Patent Document 1, even if a person in the security area is detected, it cannot be detected until there is a picking action, and an alarm cannot be output.
In addition, it is required to more accurately determine that a person is a suspicious person.
An object of the present invention is to provide a digital smart defense security system, method and program capable of outputting a prompt and appropriate alarm according to the position of a person in a security area.

The digital smart defense security system according to the present invention provides a person detecting means for detecting a person in a security area in which a plurality of areas are set according to the degree of danger, and a person detected by the person detecting means. It is characterized by providing a notification means for notifying the presence of a suspicious person while giving a warning by voice corresponding to the area.

With this configuration, a person in a security area set according to the degree of danger is determined and a voice warning is given according to the degree of danger, so a quick and appropriate alarm is output according to the position of the person in the security area. can do. For example, a suspicious person impersonating a courier, a salesman, a bank clerk, etc. may visit the house. In this case, since a voice corresponding to the degree of danger is given according to the position of the suspicious person, the suspicious person can hesitate the suspicious act or repel the suspicious person.

A voice recognition means for acquiring the voice of the conversation and determining that the content of the conversation is suspicious is provided, and the reporting means is determined by the voice recognition means that the content of the conversation is suspicious. In response to this, by reporting the existence of a suspicious person, it is possible to prevent the suspicious person from entering the house. Even if it cannot be determined that the person is a suspicious person only by determining the position of the suspicious person in the security area, it can be determined that the person is a suspicious person from the conversation with the suspicious person. Conversations are not limited to people within the security zone, as do telephone conversations.

The voice recognition means recognizes the voice of the conversation of the intercom installed at the entrance, and determines that the visitor impersonating a delivery person, a salesman, a bank employee, etc. is a suspicious person through the conversation of the intercom. It is possible to prevent suspicious persons from entering the house.

An ID terminal detecting means for detecting an ID (Identification) terminal possessed by a non-intruder is provided, and the reporting means does not report a person who possesses the ID terminal detected by the ID terminal detecting means, for example. It is possible to detect the smartphones of family members and related persons and exclude them from suspicious persons, reduce unnecessary reports, and effectively monitor them. In addition, resources for monitoring can be reduced to reduce costs.

The voice recognition means can determine a suspicious person from a telephone conversation by recognizing the voice of a telephone call, and can prevent apo-den robbery and transfer fraud.

A suspicious vocabulary storage means for storing a vocabulary used in a suspicious conversation is provided, and the voice recognition means collates the acquired word appearing in the voice with the vocabulary of the suspicious vocabulary storage means, and the content of the conversation is By determining that it is suspicious, the accuracy of suspicious person determination can be improved, and apo-electric robbery and transfer fraud can be reliably prevented.

Further, the digital smart defense security method of the present invention has a person detection step for detecting a person in a security area in which a plurality of areas are set according to the degree of danger, and a person detected by the person detection step. It is characterized by providing a notification step for notifying the presence of a suspicious person while giving a warning by voice corresponding to the area.

Further, the present invention relates a computer to a person detecting means for detecting a person in a security area in which a plurality of areas are set according to a degree of danger, and a person detected by the person detecting means in the area. It is a program to function as a digital smart defense security system equipped with a notification means for notifying the existence of a suspicious person while giving a warning with a corresponding voice.

According to the present invention, it is possible to output a prompt and appropriate alarm according to the position of a person in the security area.

It is a block diagram which shows the structure of the digital smart defense security system which concerns on embodiment of this invention. It is a block diagram of the control part of the digital smart defense security system which concerns on embodiment of this invention. It is a block diagram which shows the whole using the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the security application operation of the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the security mode setting process of the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the security mode of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example of the security in the private house of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example of the security in the apartment house of the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the warning by the automatic voice and the remote dialogue function by the image, voice according to the position of the intruder of the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the digital smart defense security processing of the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the digital smart defense security processing of the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the function of the voice recognition part of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example of the intruder detection in the plane of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example of the intruder detection in the elevation of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example of the intruder verification by time of the digital smart defense security system which concerns on embodiment of this invention. It is a figure which shows the example which the digital smart defense security system which concerns on embodiment of this invention maps a site boundary to a camera image. It is a figure explaining an example of registering the height of the digital smart defense security system which concerns on embodiment of this invention in advance in the site. It is a figure explaining the change of height by a human state in the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the change of height by a human state in the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the judgment by the change of the movement of the pupil and the neck in the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the temporary security release by BLE of the digital smart defense security system which concerns on embodiment of this invention. It is a figure explaining the application example 1 of the intrusion detection by the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the outline of the operation of a digital smart defense security system, a family, and a suspicious person in the application example 1 of FIG. It is a figure explaining the application example 2 of the intrusion detection by the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the operation of the digital smart defense security system in the application example 2 of FIG. It is a figure explaining the application example 3 of the intrusion detection by the digital smart defense security system which concerns on embodiment of this invention. It is a flowchart which shows the operation of the digital smart defense security system in the application example 3 of FIG. It is a flowchart which shows the process of notifying a user according to the degree of caution in the application example 4 of the digital smart defense security system which concerns on embodiment of this invention. In the application example 5 of the digital smart defense security system according to the embodiment of the present invention, the outline of the operation of the digital smart defense security system and the family / guardian of the automatic voice warning according to the person in the security area is shown. It is a flowchart. It is a flowchart which shows the outline of detection, registration and transfer of a suspicious call in application example 6 of the digital smart defense security system which concerns on embodiment of this invention. FIG. 5 is a flowchart showing a function of determining the attention level of a call partner by voice recognition in Application Example 7 of the digital smart defense security system according to the embodiment of the present invention.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.
(Embodiment)
FIG. 1 is a block diagram showing a configuration of a digital smart defense security system according to an embodiment of the present invention.
This digital smart defense security system is suitable for application to business establishments that require a high degree of management, such as residences, corporate offices, factories, research institutes, information processing rooms, and money collection rooms.

As shown in FIG. 1, the digital smart defense security system 1000 includes one surveillance camera 11 (photographing means) installed in each of the security areas, a human sensor 20, and Wi-installed in the security area. Fi (Wireless Fidelity) terminal (hereinafter referred to as "Wi-Fi master unit") 30, beacon master unit 40, mobile terminal device 50 (ID (Identification) terminal) carried by related parties (family members), and the entire system It includes a monitoring device 100 to control and an AI (Artificial Intelligence) accelerator (Accelerator) 200 (human detection means). The security area is a security area (security target area), and includes, for example, a site, a front door and a balcony for a residence, and an elevator hall, a balcony and a window for an office.

Although the monitoring device 100 is installed in the security area, it may be installed outside via a network (not shown). When the monitoring device 100 is installed on a server on the network, a plurality of security areas can be monitored.

<Surveillance camera 11>
The surveillance camera 11 captures an image (two-dimensional) of the security area specified in two or three dimensions.
A part or all of the surveillance camera 11 is a PTZ camera having a PTZ (pan / tilt / zoom) function, and is remotely controlled by the surveillance device 100. The surveillance cameras 11 are installed around the outer wall of the security target building, for example, at the entrance / exit where an intruder can enter / exit, the window opening, the outer periphery of the site boundary, and the corresponding site area. The image taken by the surveillance camera 11 is output to the surveillance device 100 and recorded in the recording unit 160.

<Human sensor 20>
The motion sensor 20 is a thermo camera, an infrared camera, or the like, and detects the temperature of a sensing object in the security area to detect a suspicious person in the security area.

<Wi-Fi master unit 30>
The Wi-Fi master unit 30 uses Wi-Fi to exchange information with the Wi-Fi slave unit 51 of the mobile terminal device 50. Further, the Wi-Fi master unit 30 can acquire position information by Wi-Fi positioning, that is, can acquire position information using a Wi-Fi access point and a predetermined position information service.

<Beacon master unit 40>
The beacon master unit 40 is a BLE transmitter installed at an entrance or the like and sends a beacon signal containing an individual identification code by BLE (Bluetooth Low Energy).
BLE is a wireless technology that detects proximity. BLE is composed of a combination of a beacon master unit 40, which is a beacon device on the transmitting side, and an application of a mobile terminal device 50 (corresponding to the beacon slave unit 52 described later) that supports radio wave reception from the beacon master unit 40. The BLE transmits unique ID information necessary for identification, and responds only to the application associated with the ID information of the mobile terminal device 50. The application of the mobile terminal device 50 registers the same identifier as the beacon master unit 40. The application (beacon slave unit 52) of the mobile terminal device 50 stands by in the background by executing an application equipped with a BLE function, and excites a predetermined action when approaching the beacon of the beacon master unit 40.

[Mobile terminal device 50]
The mobile terminal device 50 is carried by a family member or the like. The mobile terminal device 50 is, for example, a smartphone 50a, a tablet 50b, a notebook computer 50c, or the like. The mobile terminal device 50 is also a mobile phone, a PHS (Personal Handy-Phone System), a PDA (Personal Digital Assistants), a dedicated terminal, or the like. In the present embodiment, the mobile terminal device 50 can be used by a family member or the like in various places (that is, the current position), and receives an e-mail or a video including a moving image from the monitoring device 100 via a telephone line (not shown). It is possible.

In the present embodiment, the mobile terminal device 50 assumes the use of a smartphone 50a (ID terminal), and each individual can use it in various places (that is, the current position). One of the mobile terminal devices 50 is located at a security company (not shown).

The smartphone 50a has a digital smart defense security application (hereinafter referred to as "security application"). If the security application is started by the background processing of each application, for example, it is via the communication carrier network (fixed network), the web service cloud 300 (see Fig. 3 below), or the Internet 303 (see Fig. 3 below). , The cloud server 301 on the web service cloud 300 (see FIG. 3 below) can be connected, and the telop about the suspicious person can be received from the cloud server 301. The smartphone 50a can notify a telop in which a suspicious person is detected on a standby screen or the like.

The smartphone 50a includes a Wi-Fi individual identification device (hereinafter referred to as a "Wi-Fi slave unit") 51, and a GPS 53 that captures the positions of related parties.
The smartphone 50a may be provided with the beacon slave unit 52. Alternatively, the smartphone 50a may include any one of the Wi-Fi slave unit 51, the beacon slave unit 52, and the GPS 53.

<Wi-Fi handset 51>
The Wi-Fi slave unit 51 receives and individually identifies the radio waves of the Wi-Fi master unit 30 installed in the business facility. The monitoring device 100 stores the arrangement information of the Wi-Fi master unit 30 installed in the facility as safety-related information. When the Wi-Fi slave unit 51 approaches the Wi-Fi master unit 30, it is possible to determine the ID and position of the person who carries the mobile terminal device 50.

<Beacon slave unit 52>
The beacon slave unit 52 is an application of the mobile terminal device 50 that supports reception of radio waves from the beacon master unit 40. The beacon master unit 40 transmits a beacon (unique ID information necessary for identification), and the application (beacon slave unit 52) of the mobile terminal device 50 excites a predetermined action when approaching the beacon of the beacon master unit 40. To do.

<GPS53>
The GPS 53 receives radio waves of position information from GPS satellites and the like. The GPS 53 acquires the position information by calculating the current position information as three parameters of latitude, longitude and altitude from the information received via the GPS antenna. The acquired position information is timely transmitted to the monitoring device 100.

In the present embodiment, an example using GPS satellites is shown as a means for acquiring position information, but a method using a positional relationship with a base station other than GPS may also be used. For example, when an Android (registered trademark) smartphone or a high-performance mobile phone with a camera is used as the mobile terminal device 50, which is a mobile terminal, a base station and a mobile phone communication network (not shown) are used in place of or in combination with GPS53. It is also possible to send and receive information to and from the mobile phone company server via the mobile phone company and acquire the current location information of the own terminal from the proximity confirmation.

Further, location information acquisition by Wi-Fi positioning, that is, location information acquisition using a Wi-Fi access point and a predetermined location information service may be used.

[Monitoring device 100]
The monitoring device 100 is installed in the residence of a person concerned (for example, a family member) and centrally manages the security area. The monitoring device 100 may be a general server computer, a personal computer, or the like.

The monitoring device 100 includes a control unit 110, an input unit 120, a storage unit 130, a dictionary 135 with appearance frequency (suspicious vocabulary storage means), a display unit 140, an output unit 150, and a recording unit 160 (recording means). A registrant information DB (database) 165, an image processing unit 170, an interface (I / F) unit 180, and a communication unit 190 are provided, and each unit is connected by a bus 195.

After that, when the subject is described as "○○ part is", the control unit 110 shall read each program from the ROM as necessary, load it into the RAM, and execute each function (described later). Each program may be stored in the storage unit 130 in advance, or may be incorporated into the monitoring device 100 when necessary via another storage medium or communication medium.

The control unit 110 is composed of a CPU (Central Processing Unit) and the like, controls the entire monitoring device 100, executes a control program, and functions as a digital smart defense security system. The detailed configuration of the control unit 110 will be described later.

The input unit 120 is an input device for the user of the monitoring device 100 to input instructions and the like, such as a keyboard, a mouse, a touch panel provided on the screen of the display unit 140, and a microphone.

The storage unit 130 is composed of memories such as ROM (ReadOnlyMemory), RAM (RandomAccessMemory), and EEPROM (ElectricallyErasableProgrammableRead-OnlyMemory), and stores various data and programs used by the control unit 110. The storage unit 130 stores still images or moving images received from the surveillance camera 11, various data and programs used by the control unit 110, and the like.

The dictionary 135 with frequency of appearance is a dictionary with frequency of appearance of vocabulary used in special fraud, apoden robbery, foreclosure, foreclosure and burglary (hereinafter referred to as "special fraud / apoden robbery") (FIG. 12). reference). The frequency-appearing dictionary 135 stores vocabulary used in suspicious conversations such as special fraud and apo-den robbery.
The dictionary 135 with appearance frequency may use a part of the memory area of the storage unit 130.

The display unit 140 displays the operating status of the monitoring device 100, the image received from the monitoring camera 11, the GUI (Graphical User Interface) for operating the monitoring device 100, and the like.

The output unit 150 is, for example, an audio interface, and outputs an audio signal from the monitoring device 100 to the sound system 158 in the security area. As the audio signal output from the monitoring device 100 to the acoustic system 158, for example, an audio signal input from an audio input device such as a microphone provided in the input unit 120, or music data stored in the storage unit 130 is controlled by the control unit. 110 may be the reproduced audio signal. The sound system 158 includes an amplifier and a plurality of speakers arranged on the premises, and emits a signal input from the monitoring device 100 on the premises.

The recording unit 160 is composed of an external storage device such as an HDD (Hard Disk Drive), and records an image in the security area taken by the surveillance camera 11. The recording unit 160 records high image quality for a predetermined short time after shooting, converts it to low image quality after the predetermined short time elapses, and records for a predetermined long time.

The registrant information DB 165 stores registrant information such as family members, parents, managers, and visitors.
In addition, a face information DB (not shown) may be set up separately from the registrant information DB 165 to store face images (face information) of suspicious persons, store personnel, and the like. Basic data such as the face of a person registered in this face information DB is obtained from a headquarters, a head office, a security company, etc. (not shown) via the I / F unit 180, and a face information DB is constructed. In addition, mutual face information DB can be updated to the latest face image (face information) by exchanging information with the headquarters, head office, security company, etc. (not shown).

Further, it is provided with an image correspondence DB (not shown) that stores each position in two dimensions of the image taken by the surveillance camera 11 and each position in the three-dimensional space taken by the surveillance camera 11 in association with each other. The control unit 110 stores the shape defined in the three-dimensional space in the image correspondence DB in association with it by a projection transformation that projects it on a two-dimensional surface. The image-corresponding DB may use a part of the memory area of the storage unit 130.

The image processing unit 170 is composed of a DSP (Digital Signal Processor) or the like, and performs predetermined processing on the received image. Predetermined processing includes contour extraction, image resizing, resolution conversion processing, and the like.

When the image captured by the surveillance camera 11 in 1 second is, for example, a 5-frame image, the 1/5 second image, the 2/5 second image, the 3/5 second image, the 4/5 second image, and the 5/5 second image. In the case of 10 frames of images per second in motion, the size of the moving object can be known by inputting the outer shape line of the object in the motion of 10 images.
The image processing unit 170 processes the image data taken by the surveillance camera 11 and outputs the image in the security area.

The I / F unit 180 connects each surveillance camera 11 arranged in the security area with the surveillance device 100. Further, the I / F unit 180 is connected to a headquarters, a head office, a security company, etc. (not shown) by a network or a dedicated line.
The communication unit 190 transmits / receives data to / from the mobile terminal device 50 via the base station. The communication unit 190 has a wireless communication function, and is connected to a control board using, for example, a UART (Universal Asynchronous Receiver Transmitter).

[Control unit 110]
FIG. 2 is a block diagram of the control unit 110 of the digital smart defense security system according to the embodiment of the present invention.
As shown in FIG. 2, the control unit 110 is configured by a CPU (Central Processing Unit) or the like, controls the entire monitoring device 100, executes a control program, and functions as a digital smart defense security system.
The control unit 110 includes a person detection unit 111 (person detection means), a voice recognition unit 112 (voice recognition means), an ID terminal detection unit 113 (ID terminal detection means), and an ID terminal registration unit 114 (ID terminal registration means). ) And a reporting unit 115 (reporting means).

The control unit 110 determines whether or not the person has the ID terminal (mobile terminal device 50; smartphone 50a) detected by the ID terminal detection unit 113 (a person who has a predetermined ID such as a family member). ..

The control unit 110 excludes the person who possesses the ID terminal detected by the ID terminal detection unit 113 from the suspicious person.

The control unit 110 refers to the registrant information DB 165 to determine whether or not the person has the ID terminal (ID of the visitor or the like) registered by the ID terminal registration unit 114, and determines whether or not the person has the ID terminal (ID of the visitor or the like). ) Is excluded from suspicious persons.

The control unit 110 determines the risk of intrusion based on the position of a person in the security area. For example, the risk of intrusion is determined based on where the intruder is on the security belts, the yellow belt Y, the orange belt O, and the red belt R (see the upper parts of FIGS. 8 to 9).

The control unit 110 performs an image / voice dialogue or a voice warning according to the degree of danger.

The control unit 110 is based on the two-dimensional position of the person and the size of the person (particularly the size of the head of the person) detected by the person detection, and the three-dimensional space of the person by the image correspondence DB (not shown). Detect the position inside. Specifically, the control unit 110 reads the projected shape that has been defined and stored in advance as the shape projected in the three-dimensional space from the image correspondence DB. The control unit 110 performs the inverse conversion of the projection transformation that projects the shape defined in the three-dimensional space onto the two-dimensional plane based on the two-dimensional image of the person detected in the security area and the size of the person. The control unit 110 detects the three-dimensional position by performing the inverse conversion of the projection conversion from the image of one surveillance camera 11. In principle, it should not be possible to detect a 3D position in a 2D image, but it is assumed that the size of the person is known (assuming that there is almost no difference in the size of the head between people). The three-dimensional position can be detected based on the two-dimensional image taken by one surveillance camera 11 and the size information of the detected person.

The person detection unit 111 detects a person from the image taken by the surveillance camera 11 and detects the size of the person. Specifically, the person detection unit 111 detects a person in the security area by using the AI accelerator 200 (described later). The human detection unit 111 issues a human detection request to the AI accelerator 200, and the AI accelerator 200 executes an AI calculation other than the CPU and transmits the human detection result to the human detection unit 111. Since high speed is required for human detection, the AI accelerator 200 is used for human detection.

The digital smart defense security system 1000 detects an intruder in the security area by detecting a person using the AI accelerator 200.
The digital smart defense security system 1000 is extremely high in deep learning by the AI accelerator 200, in particular, by targeting only people, as compared with image recognition by a conventional motion detection type surveillance camera using image difference. Intruder can be detected with high accuracy.

In the present embodiment, the person detection unit 111 detects a person by using the AI accelerator 200, but the person may be detected by the thermo camera (or the motion sensor 20). That is, the motion sensor 20 detects the temperature in the security area. Then, the human detection unit 111 detects the presence of a person (suspicious person candidate) by detecting the body temperature of the person by the motion sensor 20 and detecting the change in the captured image by the surveillance camera 11.

The person detection unit 111 may combine the person detection using the AI accelerator 200 and the person detection using the thermo camera (or motion sensor). For example, in the security area, AI accelerator 200 is used for the entrance, veranda, and windows where high speed of human detection is required, and the site of the boundary between the site and the outside where high speed of human detection is not required is a thermo. Use a camera (or motion sensor).

The voice recognition unit 112 acquires the voice of the conversation and determines that the content of the conversation is suspicious. The voice recognition unit 112 collates the words appearing in the acquired voice with the vocabulary of the dictionary 135 with appearance frequency, and determines that the content of the conversation is suspicious.

The ID terminal detection unit 113 detects an ID (Identification) terminal possessed by a non-intruder.
The ID terminal registration unit 114 registers the ID terminal possessed by the visitor or the like in the registrant information DB 165.

The reporting unit 115 warns the person detected by the person detecting unit 111 with a voice corresponding to the area, and also notifies the existence of a suspicious person. The reporting unit 115 makes a report according to the degree of danger (degree of caution, degree of urgency).

The reporting unit 115 reports the existence of a suspicious person after the voice recognition unit 112 determines that the content of the conversation is suspicious.
The reporting unit 115 does not report the person who possesses the ID terminal detected by the ID terminal detecting unit 113.

[AI Accelerator 200]
The AI accelerator 200 is a dedicated processor that detects a person, and uses a computing resource other than the CPU. The AI accelerator 200 is, for example, an accelerator for image processing by a processor enhanced with a GPU (Graphics Processing Unit) and signal processing using an FPGA (Field Programmable Gate Array). Further, the AI accelerator 200 executes the calculation of AI (Artificial Intelligence) on a dedicated hardware (for example, GPU).

In the computer processing by a normal PC, it takes about 1.5 seconds to detect a person (human body) per digital image. Therefore, in the present embodiment, by using the AI accelerator 200, which is a human detection processor, the performance of the computer processing by the PC is about 10 times higher, and the intrusion detection is executed quickly. Further, in the present embodiment, the calculation of AI, which has a high calculation load, is left to the AI accelerator 200, which is a dedicated hardware. As a result, it was demonstrated that suspicious behavior can be detected in real time and suspicious persons can be registered even with a configuration using a commercially available camera and inexpensive equipment.

[Digital Smart Defense Security System]
FIG. 3 is a configuration diagram showing the whole using the digital smart defense security system according to the embodiment of the present invention.
As shown in FIG. 3, the digital smart defense security system 1000 actively cooperates with the cloud server (commercial server) 301 and the cloud server 301 that provide the digital smart defense security service on the web service cloud 300. It has a Push notification server 302 that acquires information and notifies the user's smartphone 50a (portable terminal device; ID terminal). The web service cloud 300 is connected to the Internet 303. The smartphone 50a can transmit texts and images to the cloud server 301 on the web service cloud 300 via the Internet 303. Further, the smartphone 50a receives the push notification from the push notification server 302 via the Internet 303. Further, the cloud server 301 and the push notification server 302 are connected to the smartphone 50a equipped with the security application via a communication carrier network (fixed network) (not shown) such as an LTE / 3G network.

As shown in FIG. 3, the digital smart defense security system 1000 detects an intrusion of a suspicious person into the premises and push-notifies the smartphone 50a owned by the person, his / her family, or the like.
When the user taps the notification that comes to the smartphone 50a, the security application is activated and the zoom screen of the suspicious person is displayed, and the telop "A suspicious person has been detected at the entrance on the 1st floor" is played. At the same time, the content is read aloud. In this way, the suspicious person is notified by the telop and voice on the screen of the smartphone 50a. Further, by the operation of the user of the smartphone 50a, for example, in an emergency, the related organizations (police, fire department) are notified. In this case, the security company and the head office of the affiliated company are also automatically notified. Also, if there is no urgency or if you want to confirm, only the security company will be notified.

[Security app operation]
FIG. 4 is a diagram showing a security application operation of the digital smart defense security system according to the embodiment of the present invention.
As shown on the left side of FIG. 4, a telop that detects a suspicious person is notified on the standby screen or the like of the smartphone 50a. By tapping the user, as shown in FIG. 4, the display of the smartphone 50a is switched to the security application operation display, the zoom screen of the suspicious person is displayed, and the position and situation of the suspicious person "suspicious person at the entrance on the 1st floor" is displayed. .. In addition, this telop is read aloud by automatic voice. Further, by tapping the user, the display of the smartphone 50a is displayed on four time-series screens as shown on the right side of FIG.

The operation of the digital smart defense security system configured as described above will be described below.
[Digital smart defense security processing]
First, the security mode setting of the digital smart defense security system will be described.

FIG. 5 is a flowchart showing a security mode setting process of the monitoring device 100 of the digital smart defense security system 1000. This flow is executed by the control unit 110 (see FIG. 2) of the monitoring device 100. FIG. 6 is a diagram illustrating a security mode.

In step S1, the security mode (home security, outing security, constant security, etc.) is defined. It is possible to provide a security function according to the security mode. The upper part of FIG. 6 shows the "always-on security mode". When set to the "always-on security mode", the digital smart defense security system 1000 can constantly monitor the status of the security area and prompt people to take appropriate measures. It also has the advantage of not requiring switching of security modes. On the other hand, the lower part of FIG. 6 shows the “outing / home security mode”. "Outing / home security mode" switches between outing security mode and home security mode. For example, the home security mode automatically turns on and off from 23:00 to 5:00. The outing security mode is turned on when going out and turned off when returning home. In the following, the case of "going out / home security mode" will be taken as an example.

Returning to the flow of FIG. 5, in step S2, the security mode is changed depending on the time.
In step S3, it is determined whether or not a change in the security mode has been received from the smartphone 50a (mobile terminal device 50).
When a change in the security mode is received from the smartphone 50a (mobile terminal device 50) (step S3: Yes), security is executed in the security mode instructed in step S4.
When the change of the security mode is not received (step S3: No), this flow is terminated and the security mode by time is executed in the defined security mode.
By defining the above security mode, unnecessary processing can be eliminated, and as a result, cost reduction and processing speed can be achieved by reducing calculation resources.

[Going out / home security mode]
The "outing / home security mode" of the digital smart defense security system 1000 will be described.
As shown in the lower part of FIG. 6, the “outing / home security mode” is set so that the user switches between the outing security mode and the home security mode depending on the time of day. For example, the home security mode automatically turns on and off from 23:00 to 5:00. The outing security mode is turned on when going out and turned off when returning home. By switching the security mode, the user can take measures such as increasing the security strength while going out and lowering the security strength while at home. The digital smart defense security system 1000 can execute security according to the actual situation of the user, and can effectively utilize the resources of the curity system.

[Always guard mode]
The "always-on security mode" of the digital smart defense security system 1000 will be described.
When a housewife or the like does household chores at home, a child or the like stays at home, or an elderly person or the like lives alone, a suspicious person who pretends to be a delivery service, a salesman, a bank clerk, etc. visits the house. May be done. It is dangerous for housewives, children, the elderly, etc. to enter such suspicious persons into their homes. In order to protect children and the elderly from such dangers, "guardians" such as children's parents and elderly sons, couples and grandchildren are set, and the guardians can confirm visitors. Provides "always guard mode". In addition, calls made by suspicious persons are often aimed at the time when the guardian is absent. By setting the security mode at all times, it is possible to protect elderly people from fraudulent calls at any time of the day. In addition, information by spoofing intrusion into important required facilities by outsiders such as stores, offices, research institutes, financial institutions, information materials rooms, and related parties such as third parties, and information by intercoms, camouflaged telephones, etc. Leakage can be prevented.

The "always-on security mode" of the digital smart defense security system 1000 has the following functions.
-A function to pre-register smartphone apps for notification and dialogue possessed by parents and managers of children's parents, elderly sons and couples, grandchildren, etc.
-Register the visitor in the registrant information DB 165 in advance by designating a schedule such as the date and time. In addition, a verification function using a QR code or the like to indicate to this system that the visitor is a regular visitor.
・ When a visitor (courier, postman, neighbor, salesman, bank clerk, etc.) enters the premises of the house, whether the visitor is a pre-registered visitor or neighbor A function to confirm by verification using a QR code or the like (see FIGS. 9 and 29).
-If the visitor is unregistered, a function to notify the smartphone of the guardian who has been registered in advance that an unregistered visitor has arrived with an image (see FIG. 29).

-A function for performing image / voice dialogue between a doorphone type dialogue system installed at the entrance and a smartphone dialogue system owned by a guardian (see FIGS. 9 and 29).
-A function to automatically detect and register unregistered or suspicious calls to the house, and a function to transfer to a guardian's smartphone when it is necessary to judge whether the other party can be trusted (Figs. 11 and 11). See 30).
-A function of determining the attention level of the other party by voice recognition (see FIGS. 10, 11, 30, and 31).
-A function that controls electronic locks at the entrance of a house, etc., and the system detects that a certain period of time, for example, 30 minutes has passed, and automatically locks the housewife when he / she goes out without locking.
・ When the electronic lock at the entrance of the house is controlled and unlocked from the smartphone owned by the housewife, or when the housewife does not carry the key or smartphone and cannot unlock the entrance at home, the guardian is remote. A function to unlock the lock remotely after confirming that the housewife, etc. is the person himself / herself and that no one has threatened him / her with a surveillance camera.

[Security example of digital smart defense security system]
An example of security for a digital smart defense security system will be described.
<Home &Building>
FIG. 7 is a diagram showing an example of security in a private house.
The digital smart defense security system provides security systems for individuals / apartments, offices, stores, offices, research institutes, information materials rooms, and financial institutions. The main functions are as follows.
A security mode (home security, outing security, etc.) is defined, and a security function is provided according to the security mode.
It has a function to change the security mode depending on the time.
An intruder is detected and registered from the image of the surveillance camera 11 installed around the outside of the house.
The detected intruder's risk (attention, urgency) is determined, and the user is notified according to the risk (attention, urgency).
The security mode is changed from the outdoors by the smartphone 50a.
A warning is notified to an intruder through a smartphone 50a, a speaker of a surveillance camera 11, or the like.

FIG. 8 is a diagram showing an example of security in an apartment house.
For example, it provides a security system for apartment houses. The main functions are as follows.
A security mode is defined and a security function is provided according to the security mode.
It has a function to change the security mode depending on the time.
An intruder is detected and registered from the image of the surveillance camera 11 installed on the outer periphery of the living area.
The detected intruder's risk (attention, urgency) is determined, and the user is notified according to the risk (attention, urgency).
The security mode is changed from the outdoors by the smartphone 50a.
A warning is notified to an intruder through a smartphone 50a, a speaker of a surveillance camera 11, or the like.

[Automatic voice warning according to people in the security area]
FIG. 9 is a diagram for explaining a warning by automatic voice and a remote dialogue function by image / voice according to the position of an intruder. FIG. 9 shows an "automatic voice warning" of the Digital Smart Defense Security System 1000. FIG. 9 shows a “doorphone type dialogue system” for explanation in comparison with the “automatic voice warning” of the digital smart defense security system 1000 on FIG. 9, and FIG. 9 bottom shows a “security camera type dialogue”. Indicates "system".

As shown in FIG. 9, the "doorphone type dialogue system" conducts two-way dialogue between the visitor and the guardian by the doorphone by image and voice. Alternatively, the "doorphone type dialogue system" executes a smartphone dialogue application of a mobile terminal (smartphone 50a) carried by a guardian via a cloud dialogue relay system to perform two-way dialogue by image and voice.

As shown in the lower part of FIG. 9, the "security camera type dialogue system" captures and collects sound of an intruder with a security camera equipped with a microphone, and the family engages in two-way dialogue with the intruder by image and voice. Alternatively, the "security camera type dialogue system" executes a smartphone dialogue application of a mobile terminal carried by a family member via a cloud dialogue relay system to perform two-way dialogue by speaker voice.

With respect to these dialogue systems, the digital smart defense security system 1000 accurately positions the position of an intruder who has invaded the security area as X, Y, and Z coordinates based on the image of the camera. Then, the digital smart defense security system 1000 determines that there is a person in the security area and warns with a voice according to the degree of danger (for example, according to the yellow belt Y, the orange belt O, and the red belt R). ..
In the above example of FIG. 9, it is determined that the intruder is on the yellow belt Y, the orange belt O, and the red belt R, and an automatic voice warning is given by sending an automatic voice message according to the degree of danger or the situation. Details will be described later in the flow of FIG. 29.
The digital smart defense security system 1000 can accurately position the position of an intruder who has invaded the security area as X, Y, and Z coordinates based on the image of the camera.

[Digital smart defense security processing]
Next, the digital smart defense security process of the digital smart defense security system 1000 will be described.
The digital smart defense security processing of the digital smart defense security system 1000 includes (I) "warning by automatic voice according to the position of the intruder, remote dialogue processing by image and voice", and (II) "conversation". There is "suspicious judgment processing based on the content". The above (I) and the above (II) may be executed individually or in combination.

<(I) "Remote dialogue processing according to the position of the intruder">
FIG. 10 is a flowchart showing (I) “remote dialogue processing according to the position of an intruder” of the digital smart defense security system. This flow is executed by the control unit 110 (see FIG. 2) of the monitoring device 100 in the “always-on security mode” set in FIG.

In step S11, an image of the security area (specific area) specified in three dimensions is taken. The surveillance camera 11 is, for example, a PTZ camera and is remotely controlled by the surveillance device 100. The image taken by the surveillance camera 11 is output to the surveillance device 100.
In step S12, the person detection unit 111 of the control unit 110 detects a person from the image captured by the surveillance camera 11. In the present embodiment, the person detection unit 111 requests the AI accelerator 200 to perform [human detection processing] by AI, and the person detection unit 111 waits for the detection result of the person from the AI accelerator 200. In addition, the person detection unit 111 may also use the person detection by the thermo camera (or the motion sensor 20).

In step S13, the control unit 110 determines whether or not a person has been detected in the image captured by the surveillance camera 11.
When a person is detected in the image captured by the surveillance camera 11 (step S13: Yes), the process proceeds to step S14. If no person is detected in the image captured by the surveillance camera 11 (step S13: No), this flow ends.

Whether or not the control unit 110 is a person (a person having a predetermined ID such as a family member) who has an ID terminal (mobile terminal device 50; smartphone 50a) detected by the ID terminal detection unit 113 in step S14. To determine. The control unit 110 excludes the person who possesses the ID terminal detected by the ID terminal detection unit 113 from the suspicious person (step S14: No), and proceeds to step S15.

In step S15, the control unit 110 refers to the registrant information DB 165 and determines whether or not the person has the ID terminal (ID such as a visitor) registered by the ID terminal registration unit 114. The control unit 110 excludes the person who has registered the ID terminal (ID such as a visitor) from the suspicious person (step S15: No), and proceeds to step S16.
In this way, when a family member or the like has an ID terminal or when an ID terminal such as a visitor is registered in advance, the subsequent processing is skipped to speed up the processing.

In step S16, the control unit 110 determines the risk of intrusion based on the position of a person in the security area. For example, the risk of intrusion is determined based on where the intruder is in the yellow belt Y, the orange belt O, or the red belt R (see the upper part of FIGS. 8 to 9). Specifically, the risk of intrusion is determined as follows.

(1) When a person is detected in the yellow belt Y as shown in FIGS. 8 to 9, the risk is moderate, and when a person is detected in the red belt R, the risk is high. .. Further, when a person is detected outside the yellow belt Y, it can be determined that the degree of danger is small.

(2) The degree of danger (attention, urgency) can be determined by the movement of a suspicious person. The movements of the suspicious person are, for example, the movement of the eyes and the movement of the neck (detailed later) and the movement of the arm (detailed later). If the movement of the suspicious person is unnatural, it can be determined that the degree of danger (attention, urgency) is high.

In step S17, the control unit 110 performs an image / voice dialogue or a voice warning according to the degree of danger.

In step S18, the reporting unit 115 makes a report according to the degree of danger (degree of caution, degree of urgency). The report transmits information about the suspicious person to, for example, the smartphone 50a of the person concerned. In this report, a message, a mark, highlighted letters, and a color code indicating the degree of danger should be attached according to the degree of danger of the intrusion crime (classification of the degree of danger). Also, change the report destination according to the degree of danger or urgency. For example, when an intruder (suspicious person) invades from the roof of the parking lot of the next house over the fence, the intruder is judged to be dangerous and urgent, and the most urgent telop about the intruder is notified. To do. In addition, the police will be included in the relevant organizations to be reported. And this report is given priority over the automatic registration process of the suspicious person. In addition, when a person is detected inside the red belt R, a telop with a high urgency of invasion is notified, and when a person is detected outside the yellow belt Y, a telop with a moderate urgency of invasion is notified (Fig. 4).

As described above, a telop for notifying the existence of a suspicious person is transmitted to the standby screen or the like of the smartphone 50a shown on the left side of FIG. By tapping the user, as shown in FIG. 4, the zoom screen of the suspicious person is displayed, the position and situation of the suspicious person "suspicious person at the entrance on the 1st floor" is displayed, and this telop is read out by automatic voice.

In step S19, the control unit 110 registers the information of the suspicious person in the security area in the storage unit 130 and ends the process of this flow. As a result, intrusion information of a suspicious person is recorded, which is useful information for crime prevention.
If no person is detected from the image taken by the surveillance camera 11 in step S13, if the person has an ID terminal such as a family member in step S14, or if the ID terminal such as a visitor is registered in step S15. If you are the person who did, end this flow.

<(II) "Suspicious judgment processing based on the content of the conversation">
FIG. 11 is a flowchart showing (II) “suspicious determination processing based on the content of conversation” of the digital smart defense security system. This flow is executed by the control unit 110 (see FIG. 2) of the monitoring device 100 in the “always-on security mode” set in FIG.
In step S21, the control unit 110 acquires the voice of the conversation between the intercom and the telephone (details will be described later).
In step S22, the voice recognition unit 112 (see FIG. 2) of the control unit 110 determines the degree of attention of the acquired conversation content by voice recognition. The voice recognition unit 112 determines the degree of attention based on the frequency of appearance of suspicious vocabulary in the contents of conversations on the intercom or telephone (see FIG. 12).

Specifically, the voice recognition unit 112 refers to the dictionary 135 with appearance frequency and determines the content of the conversation by collating words. The collated words are weighted according to their frequency of occurrence. When the frequency of occurrence of words is less than a predetermined threshold value, it is determined that the degree of attention is low. When the frequency of appearance of words is equal to or higher than a predetermined threshold value, it is determined that the degree of attention is high, and when the degree of attention is equal to or higher than a predetermined value, it is determined that the content of the conversation is suspicious.

When the caution level is equal to or higher than the predetermined threshold value (step S22: Yes), the control unit 110 shifts to the alert control in step S23. When the degree of attention is smaller than the predetermined threshold value (step S22: No), it is determined that it is normal, and the processing of this flow ends.

In step S24, the control unit 110 determines whether the acquired voice is that of the intercom. If the acquired voice is not that of the intercom (step S24: No), the process proceeds to step S27.
When the acquired voice is that of the intercom (step S24: Yes), in step S25, the control unit 110 refers to the registrant information DB 165 and determines whether or not the registered terminal exists.

When the registered terminal exists (step S25: Yes), in step S26, the control unit 110 releases the alert control and ends the process of this flow.
When the registration terminal does not exist (step S25: No), the control unit 110 determines in step S27 whether or not the acquired voice is a telephone.
When the acquired voice is that of a telephone (step S27: Yes), in step S28, the control unit 110 determines whether or not the content of the conversation is suspicious by collating the content of the conversation with words.
When the acquired voice is not that of a telephone (step S27: No), it is determined that the acquired voice is neither an intercom nor a telephone and is not subject to control, and the processing of this flow is terminated.

In the case of the determination result that the content of the conversation is suspicious in step S28 (step S28: Yes), the control unit 110 continues the alert control and saves this conversation in step S29.
In step S30, the control unit 110 notifies the guardian or the like that the content of the conversation is suspicious, and ends the process of this flow.

If the content of the conversation is not a suspicious determination result in step S28 (step S28: No), the alert control is released in step S26 and the present flow ends.

[Suspicious judgment of conversation content by voice recognition unit 112]
Next, the suspicious determination of the conversation content by the voice recognition unit 112 will be described.
FIG. 12 is a diagram illustrating the function of the voice recognition unit 112 of the digital smart defense security system 1000.
The digital smart defense security system 1000 acquires the voice of the telephone, and the voice recognition unit 112 refers to the dictionary 135 with the frequency of appearance and determines the content of the conversation by collating words.

The dictionary 135 with frequency of appearance is a dictionary with frequency of appearance of vocabulary used in special fraud, apoden robbery, etc. In the dictionary 135 with frequency of occurrence, words are associated with the frequency of occurrence of (good_freq / bad_freq). good_freq is the frequency of appearance of vocabulary that appears in a normal call, and the frequency of appearance of vocabulary that appears in a call of bad_freq: special fraud / apo-den robbery. For example, the dictionary 135 with frequency of appearance stores the following vocabulary and the like.

I (10,000 / 100,000)
Children (100 / 10,000)
Grandchildren (500 / 10,000)
Dad (500/50,000)
Mom (500/50,000)
Grandpa (100 / 40,000)
Grandma (100 / 60,000)
Cash (1,000 / 100,000)
Slip (10 / 1,000)
Deposit (700 / 80,000)
Counterfeit bill (10/500)
Card (100 / 90,000)
PIN (100 / 100,000)
Account (200 / 90,000)
Safe (10 / 20,000)
In addition, "general manager, general manager, technical general manager, managing director, managing director, headquarters, head office, head office, audit" may be included.
For example, in a normal call, vocabularies such as cash (1,000 / 100,000), PIN (100 / 100,000), and account (200 / 90,000) rarely appear. If these bad_freq vocabularies appear, special fraud and apo-den robbery are suspected. If they appear in combination, it can be judged that there is a higher possibility of fraud.

[Characteristics of Digital Smart Defense Security]
Next, the features of the digital smart defense security of the digital smart defense security system will be described.

The digital smart defense security system 1000 has a technology for determining intrusion in a plane, a technology for determining intrusion in an elevation, and a technology for detecting intrusion in a plane and an elevation as a site intrusion detection and registration function. , Use one or a combination of techniques to verify intrusion over time. The details will be described below.

To detect a suspicious person, first determine from the camera image whether a human has invaded the premises. After this judgment, it is verified whether the person is a suspicious person and registered / notified.

The warning area is inspected in 2D, 3D, and 4D (3D + time) to evaluate and register the degree of risk.

The area within the boundary of the site is categorized into the following three according to the degree of risk.
・ Yellow belt Y: For example, 30 cm to 4 m inside the boundary of the site
・ Orange belt O: Between yellow belt Y and red belt R ・ Red belt R: For example, 30 cm to 4 m from the wall surface of the house

<About security belt>
The yellow belt Y (hereinafter referred to as Y in the figure) is in the state of early warning level. Intrusion into this part is notified with a yellow risk, but by adding 3D and 4D additional information and verifying, and notifying if there is an intrusion as a result of verification, detection accuracy and early It becomes possible to deal with. In addition, the red belt R (hereinafter, indicated by R in the figure) is a level at which it is necessary to immediately notify when there is an intrusion into the area.

<Detection on a plane>
FIG. 13 is a diagram showing an example of intruder detection in a plane, in which the left side of FIG. 13 shows a normal state in a plane and the right side of FIG. 13 shows an abnormal state in a plane.
As shown on the left side of FIG. 13, a yellow belt Y is set inside the boundary of the site, and a red belt R is set so as to surround the wall surface of the house having the entrance door and the window with the orange belt O which is the security zone sandwiched. ..
The right side of FIG. 13 shows an abnormal state in a plane. An intruder has invaded beyond the yellow belt Y to the outside of the red belt R.

In the intruder detection on the plane, the inclusion relationship between the intruder and the boundary is evaluated. For example, as shown by the symbol a on the right side of FIG. 13, if the surveillance camera 11 shoots at 10 frames / second, an intruded moving object is detected from the images of 10 frames (see the frame on the right side of FIG. 13). Calculate its position on the plane. If the intruder is certainly detected within the 10 frames (5 frames for a surveillance camera with 5 frames per second), it is determined that the intruder may exist.

<Detection on elevation>
FIG. 14 is a diagram showing an example of intruder detection on an elevation. FIG. 14 left shows a normal state on an elevation, and FIG. 14 right shows an abnormal state on an elevation.
As shown on the left side of FIG. 14, a yellow belt Y is set inside the boundary of the site, and a red belt R is set so as to surround the wall surface of the house with the orange belt O sandwiched between them.
In the detection on the elevation, the height (see the frame on the right of FIG. 14) is evaluated from the image of the surveillance camera 11. Evaluate whether this height is a standard human height. For example, as shown by reference numeral b on the right side of FIG. 14, the number of images per second of the surveillance camera 11 is evaluated, and it is certainly determined that the intruder is a human being. In the example of FIG. 14, it is detected that an intruder invades beyond the yellow belt Y.
By detecting intruders on the elevation, it is possible to eliminate the invasion of animals and the false detection of movement due to the wind of planting.

<Verification by time>
FIG. 15 is a diagram showing an example of intruder verification by time, in which the left side of FIG. 15 shows a normal state in time and the right side of FIG. 15 shows an abnormal state in time.
As shown by the reference numeral c on the right side of FIG. 15, the digital smart defense security system 1000 repeats the evaluation every second for the detection of the intruder obtained by the intruder detection on the elevation. As shown on the right side of FIG. 15, an intruder is walking on the yellow belt Y. The digital smart defense security system 1000 determines that the intruder has certainly invaded after the intruder has been continuously detected on the yellow belt Y.

<Technology for determining intrusion from an elevation: Mapping>
A technique for determining intrusion from an elevation will be described.
As a technique for determining intrusion from an elevation, the plane layout information of the site boundary is mapped to the camera image.
FIG. 16 is a diagram showing an example of mapping the site boundary to the camera image, FIG. 16 left shows a plan view of the site boundary, and FIG. 16 right shows an example of mapping the site boundary to the camera image. Mapping is a technique for synthesizing an image generated by CG (Computer Graphics) with a photographic image. When compositing, the CG can be naturally superimposed and composited on the photographic image by making the background portion of the CG transparent.

As shown on the left side of Fig. 16, when installing the surveillance camera (see the x mark on the left side of Fig. 16), accurately create a plan view of the caution area (outline shape such as the boundary of the home site). In addition, the installation position, height and orientation (roll, pitch, yaw information) of the surveillance camera are also given. The roll, pitch, and yaw represent the rotation angle of the line of sight in a three-dimensional space. If the line-of-sight direction is the X-axis, the left-hand direction perpendicular to it is the Y-axis, and the upward direction of the viewpoint is the Z-axis, the rotation around the X-axis is called roll, the rotation around the Y-axis is called pitch, and the rotation around the Z-axis is called yaw.

The right side of FIG. 16 shows an example of mapping the image of the surveillance camera of reference numeral d on the left side of FIG.
The position information of the caution area is projected from the parameters of the camera and clipped by the viewport to find out what part of the image obtained from the surveillance camera is on the site (see the right side of FIG. 16). The viewport represents a two-dimensional limit when recognizing a three-dimensional space as two-dimensional. This corresponds to the range that appears in the picture when the picture is taken with a surveillance camera. The viewport represents the range that can be captured by the surveillance camera. Further, the clipping refers to a process of cutting out an object such as computer graphics represented by three-dimensional data to the size of a viewport.

<Technology for determining intrusion from an elevation: size comparison with a pre-registered object>
Known security areas such as gate height, car height, vegetation height, and human height (head size, upright height, crouching height, etc.) within the site, which is a mapped security area. Compare the size of a person).

FIG. 17 is a diagram illustrating an example of registering the height of something that is in advance on the site.
As shown in FIG. 17, the height of the gate, the height of the car, and the height of the vegetation in the site, which is the mapped security area, are registered. In addition, the standard size that a person has, that is, the height of the person (the size of a known person such as the size of the head, the height of standing upright, the height of crouching, etc.) is an image-compatible DB (not shown). Remember in advance. When a person invades, a person has a human height other than the height of the gate, the height of the car, and the height of the vegetation in the site, which is a pre-registered security area, and there are multiple shapes. The intruder (see the frame of FIG. 17) is determined by the image of.

FIG. 18 is a diagram illustrating a change in height due to a human condition. Reference numeral e in FIG. 18 indicates an abnormal state in which the height changes. Reference numeral f in FIG. 18 indicates an object (gate, etc.) of various heights registered in the image correspondence DB, and reference numeral g in FIG. 18 indicates the size of a person in a normal state registered in the image correspondence DB. Indicates the height.
As shown in FIG. 18, in order to determine the position and three-dimensional height of a human (a prone posture, a sitting posture, a mid-waist posture, a standing posture, etc.), five human bodies, that is, head and face parts. (To be able to correctly express the contrast with the thickness and height of parts, body, legs, etc.) Reflect. As a result, the accuracy of the attention level judgment is increased to nearly 100%.
The height of a human is determined by the height of a box in an area (primary line) (see the frame in FIG. 17) surrounding a moving object in an image obtained from a surveillance camera.

<Verification by time>
We will explain a technique for more accurately verifying intruder detection on an elevation by intruding over time.
FIG. 19 is a diagram illustrating a change in height depending on a human condition. Reference numeral h in FIG. 19 indicates a state in which the intruder is stopped, reference numeral i in FIG. 19 indicates a state in which the intruder is slowly moving, and reference numeral j in FIG. 19 indicates a state in which the intruder is moving quickly. Each is shown.

In the intruder detection on the elevation, it is possible to determine whether or not a human is invading the caution area (within the boundary of the site) at a certain moment, but it is not possible to determine whether or not the person is invading for some purpose. In other words, it is not possible to remove the case where the guard area (within the boundary of the site) has invaded temporarily. Therefore, in the digital smart defense security system 1000, temporal continuity is added.

As shown in Fig. 19, when the information that determines that a human has invaded the premises is continuously obtained in a short period of time. For example, if it is detected continuously for 2 seconds, it can be determined that the intruder has intentionally invaded. It is possible to determine the fact that "a person intentionally invaded the site" by using the verification technology for the intrusion due to this time, and to register the state information.

[Evaluation of alertness of detected intruders]
The digital smart defense security system 1000 evaluates the alertness of the detected intruder and notifies the user according to the alertness. Corresponds to the above-mentioned "report according to the degree of danger (degree of caution, degree of urgency)" in step S19 of the flow of FIG.
Specifically, a judgment technique based on the movement of the eyes and a judgment technique based on the movement of the neck are used. As a result, the digital smart defense security system can evaluate the psychological state of the target whose intrusion is detected in the caution area as a risk level.

<Judgment by eye movement>
The upper part of FIG. 20 is a diagram for explaining the judgment based on the movement of the pupil (left and right).
Using image processing technology, the face can acquire the position of the eyes, nose and mouth by "Facial Landmark Detection". The location of the pupil is determined by the following technique.
First, the first image is cut out. This is called a primary line. (2) Next, the image is binarized. The eyes turn black and the white eyes turn white. (3) Divide the eyes into three horizontal areas (right, center, left). The black pixels in each of these areas are counted. This most abundant area is called a secondary line. It is judged that there is a pupil in this secondary line. In other words, it is possible to determine whether the pupil is to the right, to the left, or in the center.

<Judgment by neck movement (left and right)>
The middle part of FIG. 20 is a diagram for explaining the judgment based on the movement of the neck (left and right).
People move their necks when they look to the left or right or look up. This movement of the neck is expressed in the form of the direction of the face. If the rectangle containing the points obtained by facial organ detection is the primary line and the rectangle containing the eyes, nose and mouth is the secondary line, the orientation of the face is determined by the distance between the primary line and the center of gravity of the area surrounded by the secondary line. It can be expressed numerically as a vector.

By extracting the contours of the eyes, eyebrows, nose, mouth, and chin with respect to the position of the entire face shown in the rectangular frame in the middle of Fig. 20, the face can be turned to the left from the neutral state in the center of Fig. 20. The state of moving the neck to the right and the state of moving the neck so as to turn the face to the right are determined step by step. It should be noted that this technique can also be used to generate a face image in a neutral state based on a face image taken from a certain angle.

<Judgment by neck movement (up and down)>
The lower part of FIG. 20 is a diagram for explaining the judgment based on the movement of the neck (up and down).
This neck movement (up and down) can be judged in the same way as the neck movement (left and right).

<Judgment over time>
Only the momentary situation can be understood only by the judgment based on the position of the pupil shown in the upper part of FIG. Therefore, the state in a short time is tracked. For example, when the position information of the eyes is acquired and recorded at 5 frames or 10 frames per second and the situations of right, center, and left are mixed, it can be determined that the eyes are moving violently. As a result, the digital smart defense security system can determine a state in which it is considered that "a person has moved his eyes violently due to malicious intent" and can register the state.

Similarly, regarding the position of the neck, the movement can be determined by using the change in the movement time of the neck. For example, when the vector changes from right to left or from top to bottom in a short time of 2 seconds, it can be determined that the neck has moved violently. As a result, the digital smart defense security system can determine a state in which it is considered that "a person has moved his or her neck violently due to malicious intent" and can register the state.
The determination process based on the change in the movement of the pupil and the neck in FIG. 20 will be described later in Application Example 4 in FIG. 28.

[Release security by BLE]
FIG. 21 is a diagram illustrating temporary security release by BLE.
BLE (Bluetooth Low Energy) is a technology that detects close proximity. The digital smart defense security system has a function of allowing the user and his / her family to invade the alert range by installing an application dedicated to the digital smart defense security system on the smartphone. The user has a predetermined unlock code (installed in the app), and when it detects proximity, this unlock code is sent to the Digital Smart Defense Security System to temporarily cancel the alert mode ( See step S3 in FIG. 5). As a result, it becomes possible to exclude the behaviors of the user and his / her family from the detection target, and it is possible to perform highly accurate intrusion detection.

[Application example]
An application example of intrusion detection by the digital smart defense security system will be described.
<Application example 1>
FIG. 22 is a diagram illustrating application example 1 of intrusion detection by the digital smart defense security system 1000.
As shown in FIG. 22, a yellow belt Y is set inside the boundary of the site, and a red belt R is set so as to surround the wall surface of the house having the entrance door and the window with the orange belt O sandwiched between them. FIG. 22 is a plan view, but in order to show the detection of an intruder by an elevation, the intruder is (1) a horizontal position (X-axis, Y-axis) and (2) a vertical position perpendicular to the horizontal position. (Z axis) is represented by a broken line. That is, the intruder detects the above-mentioned "position in the three-dimensional space by the control unit 110". Similarly, intruder detection on the elevation is executed for the suspicious persons (crouching suspicious person, prone suspicious person) in the security zones of the western space and the southern space of FIG. 22.

FIG. 23 is a flowchart showing an outline of the operations of the digital smart defense security system 1000, the family, and the suspicious person in the application example 1 of FIG. The flow of the family and the suspicious person in FIG. 23 is not CPU processing, but will be described with step numbers for convenience. The family has step F, and the suspicious person has step SP.

《Digital Smart Defense Security System》
In step S101, the digital smart defense security system 1000 starts security (step S1).
The security mode is changed according to the time in step S102 (step S2) (corresponding to step S2 in FIG. 5).

Arbitrary set condition determination is performed (step S103).
If the whole family has returned home, such as after 22:00, home security is automatically started in step S104.
At 23:00 at night, home security is automatically started in step S105.
In the case of 5:00 in the morning, the home security is automatically canceled in step S106.

An image is obtained from the surveillance camera (step S107).
Obtain a rectangle that surrounds a human in the image (step S108).
An intruder is detected and registered from the image of the surveillance camera 11 installed around the outside of the house (step S109).
Send a BLE signal to the family's smartphone (step S110).

It is determined whether or not the security is temporarily released by BLE (step S111).
When the security is temporarily released, the security is temporarily released (step S112).
If the security is not temporarily released, the horizontal position and the vertical position of the intruder are acquired in step S113.

It is determined whether or not a suspicious person has invaded the site (step S114).
When a suspicious person invades the site, the intruder's caution level detected in step S115 (corresponding to the “risk level determination” in step S17 in FIG. 10) is evaluated and used according to the caution level. Create information to notify the person.
In step S116, the family's smartphone is notified and notified (corresponding to the “report” in step S18 of FIG. 10), and the process returns to step S102.

"family"
A family member or a related person sets the security mode (step F1). When the family is absent, use out-of-home security, and when the family is at home, use home security (see the bottom of Fig. 6). The family security mode setting (step F1) information is transmitted to the digital smart defense security system 1000, and the digital smart defense security system 1000 starts security (step S1).

An intruder invades the yellow belt Y (step F2). For example, a suspicious person invades the yellow belt Y on the north side shown in FIG. The digital smart defense security system 1000 obtains an image from a surveillance camera (step S107).

It is determined whether the intruder is a family member (step F3).
If the intruder is a family member, the family member is inside the yellow belt Y (step F4). For example, a family is in the yellow belt Y on the north side shown in FIG.

The digital smart defense security system 1000 sends a BLE signal to the family's smartphone (step S110), and the family's smartphone receives the BLE signal (step F5).

In response to this, the family member or the person concerned changes the security mode from the outdoors using the smartphone (step F6). This is because even if the family is in Yellow Belt Y, they are not suspicious, so the security is temporarily released.
Act freely on the premises (step F7).

Receive a notification from the Digital Smart Defense Security System 1000 on your smartphone (step F8).
A warning is notified to an intruder from the speaker of the smartphone or the surveillance camera (step F9).

"invader"
If an intruder has invaded the yellow belt Y and the intruder is not a family member, a suspicious person has entered the yellow belt Y (step SP1).
A suspicious person takes a prone posture, a sitting posture, a mid-waist posture, a standing posture, etc. on the premises (step SP2).
The suspicious person's eyes wander left and right (step SP3).
A suspicious person quickly shakes his head from side to side (step SP4).
A suspicious person quickly shakes his head up and down (step SP5).
The movements of these suspicious persons are sent to the digital smart defense security system 1000, and the digital smart defense security system 1000 determines the degree of danger (attention, urgency) based on the movements of the suspicious person. The digital smart defense security system 1000 can notify family members and related organizations of the level according to the degree of danger (degree of caution, degree of urgency).

<Application example 2>
FIG. 24 is a diagram illustrating application example 2 of intrusion detection by the digital smart defense security system 1000. FIG. 24 shows an example of detecting and registering an intruder from the image of a surveillance camera installed around the outside of the house. FIG. 25 is a flowchart showing the operation of the digital smart defense security system 1000 in the application example 2 of FIG. 24.
The left side of FIG. 24 is an image in the case of 10 frames per second before the intrusion of a suspicious person (Before). As shown on the left side of FIG. 24, the suspicious person has not invaded the yellow belt Y.

The right side of FIG. 24 is an image of 10 frames per second after the intrusion of a suspicious person (After). As shown on the right side of FIG. 24, the suspicious person invaded the yellow belt Y. The person detection unit 111 of the control unit 110 (see FIG. 2) of the digital smart defense security system 1000 detects a person (suspicious person) from the image taken by the surveillance camera 11. The suspicious person detected on the right side of FIG. 24 is indicated by a rectangular frame, and the approximate two-dimensional distance to the red belt R is indicated by a broken line. As shown on the right side of FIG. 24, a suspicious person who has invaded beyond the yellow belt Y is approaching the red belt R.

In FIG. 25, an intruder into the red belt R is detected in step S201. As described above, the person detection unit 111 of the control unit 110 (see FIG. 2) of the digital smart defense security system 1000 detects a person from the image taken by the surveillance camera 11.
In step S202, detection of an intruder on the red belt R (for example, in the range of 30 cm to 4 m from the wall surface of the house) is started.

Hereinafter, the intruder is detected on each of the elevation and the plane by branching into the intruder detection on the elevation (step S203 to S209) and the intruder detection on the plane (step S210 to S214). In the present embodiment, the control unit 110 (see FIG. 2) refers to the image correspondence DB based on the two-dimensional position of the person and the size of the person detected by the person detection, and is in the three-dimensional space of the person. Detect the position. Then, the control unit 110 is characterized in that the presence of the suspicious person is notified in response to the fact that the position of the detected person in the three-dimensional space is within the three-dimensional security area. Therefore, intruder detection on the elevation is the main operation, and FIG. 24 also shows an image of the intruder on the elevation.

In step S203, intruder detection on the elevation is started (5 to 10 frames of images per second).
In step S204, an image for 1 second, for example, an image of 1/10 to 10/10 in the case of 10 frames / second is inspected on an elevation.

It is determined in step S205 whether or not the intrusion is detected.
If the intrusion is not detected in step S205, it is determined in step S206 that there is no intrusion, and in step S207 it is determined to be normal, and the processing of this flow is terminated.
When an intrusion is detected in step S205, it is determined in step S208 that there is an intrusion, and in step S209, it is determined that there is an abnormality (need attention), and the processing of this flow ends.

On the other hand, in step S210, intruder detection on a flat surface is started (5 to 10 frames of images per second).
In step S211, an image for 1 second, for example, an image of 1/10 to 10/10 in the case of 10 frames per second is inspected on a flat surface.

It is determined in step S212 whether or not the intrusion is detected.
If the intrusion is not detected in step S212, it is determined in step S213 that there is no intrusion, and in step S207 it is determined that the intrusion is normal, and the processing of this flow ends.
When an intrusion is detected in step S212, it is determined in step S214 that there is an intrusion, and in step S209, it is determined that there is an abnormality (need attention), and the processing of this flow ends.

In Application Example 2, since the intruder detection on the elevation and the intruder detection on the plane are used together, the detection accuracy can be improved as compared with the intruder detection alone on the elevation.

<Application example 3>
FIG. 26 is a diagram illustrating application example 3 of intrusion detection by the digital smart defense security system 1000. FIG. 26 left is a plan view showing intruder detection on a plane, and FIG. 26 right is an elevation view showing intruder detection on an elevation.

On the right side of FIG. 26, the plan view on the left side of FIG. 26 is projected and converted into a three-dimensional space for detecting an intruder by an elevation by the above-mentioned "technology for determining intrusion in an elevation (see FIGS. 16 to 17)". It is an elevation view. The right side of FIG. 26 shows an intruder inside the car gate. An intruder is represented by a broken line in a horizontal position (X-axis, Y-axis) and a vertical position (Z-axis).

FIG. 27 is a flowchart showing an operation in which the digital smart defense security system 1000 detects and registers an intruder from the image of a surveillance camera installed around the outside of the house in the third application of FIG. 26.
In step S301, intruder detection to the yellow belt Y (within the security applicable site, road boundary, adjacent land boundary, for example, in a range of 30 cm to 4 m) is started.

Hereinafter, the intruder is detected on each of the elevation and the plane by branching into the intruder detection on the elevation (step S302 to S311) and the intruder detection on the plane (step S210 to S214).

In step S302, intruder detection on the elevation is started (image of 5 to 10 frames / sec).
In step S303, an image for 1 second (for example, an image of 1/10 to 10/10 in the case of 10 frames per second) is inspected on an elevation.
In step S304, an image for 1 second (for example, an image of 10 frames) from 1 second to 2 seconds is inspected on an elevation.

After 1 second has elapsed in step S305, it is determined whether or not two or more frames are detected within 2 seconds.
If two or more frames are not detected within 2 seconds after 1 second elapses, it is determined in step S306 that there is no intrusion, and in step S307, it is determined to be normal, and the processing of this flow ends.

If two or more frames are detected within 2 seconds after 1 second has elapsed in step S305, it is determined that there is a possibility of intrusion in step S308, and temporal verification is started in step S309. The temporal verification on the elevation is as follows.
That is, in step S310, the next 1-second image (for example, 10 frames) is inspected on an elevation. Next, in step S311, the temporal verification of step S310 is repeated for 1 to 2 minutes to improve the accuracy.

In step S312, it is determined whether or not the image showing the intruder detection is continuously detected for a certain period of time.
If it is continuously detected for a certain period of time, it is determined in step S313 that it is abnormal (need attention), and the processing of this flow is terminated.

On the other hand, in step S314, intruder detection on a plane is started (image of 5 to 10 frames / sec).
In step S315, an image for 1 second (for example, an image of 1/10 to 10/10 in the case of 10 frames per second) is inspected on a flat surface.
In step S316, an image for 1 second (for example, an image of 10 frames) from 1 second to 2 seconds is inspected on a flat surface.

After 1 second has elapsed in step S317, it is determined whether or not two or more frames are detected within 2 seconds.
If two or more frames are not detected within 2 seconds after 1 second has elapsed, it is determined in step S319 that there is no intrusion, and in step S307, it is determined to be normal, and the processing of this flow is terminated.

If two or more frames are detected within 2 seconds after 1 second has elapsed in step S317, it is determined that there is a possibility of intrusion in step S319, and temporal verification is started in step S309. The temporal verification on the plane is as follows.
That is, in step S320, the next 1-second image (for example, 10 frames) is inspected on a flat surface. Next, in step S321, the temporal verification of step S320 is repeated for 1 to 2 minutes to improve the accuracy.

In the application example 3, since the intruder detection on the elevation and the intruder detection on the plane are used in combination as in the application example 2, the detection accuracy can be improved as compared with the intruder detection alone on the elevation.
Further, in Application Example 3, when there is a possibility of intrusion, time verification is performed, so that it is possible to more reliably detect the intrusion of an intruder and improve the accuracy of detection.

In this way, the digital smart defense security system 1000 is based on the technology of accurately positioning the position of an intruder who has invaded the security area as X, Y, and Z coordinates based on the image of the camera, and is based on the conventional security. It is possible to solve the problem of false positives in the system.

<Application example 4>
FIG. 28 is a flowchart showing a process of notifying the user according to the degree of attention of the digital smart defense security system 1000 in the application example 4. The flow of FIG. 28 shows the determination process based on the change in the movement of the pupil and the neck of FIG.
The facial organ is detected in step S401.

Hereinafter, the intruder is detected by branching into an intruder detection by the movement of the pupil, an intruder detection by the left / right movement of the neck, and an intruder detection by the up / down movement of the neck.
《Movement of eyes》
In step S402, the left and right positions of the pupil are determined based on the images of 5 to 10 frames per second.
In step S403, an image for 1 second, for example, an image of 1/10 to 10/10 in the case of 10 frames per second is inspected.
In step S404, an image (for example, 10 frames) for 1 second from 1 second to 2 seconds is inspected.

After 1 second has elapsed in step S405, it is determined whether or not two or more frames are detected within 2 seconds.
If two or more frames are not detected within 2 seconds after 1 second has elapsed, it is determined in step S406 that it is normal, and the processing of this flow is terminated.

If two or more frames are detected within 2 seconds after 1 second has elapsed in step S405, it is determined in step S407 that there is a possibility of abnormality, and the process proceeds to step S408.
Temporal verification is started in step S408. The temporal verification by the movement of the pupil is as follows.
That is, in step S409, the next 1-second image (for example, 10 frames) is inspected. Next, in step S410, the temporal verification of step S409 is repeated for 1 to 2 minutes to improve the accuracy.

In step S411, it is determined whether or not the image showing the intruder detection is continuously detected for a certain period of time.
If it is not continuously detected for a certain period of time, it is determined in step S412 that it is normal, and the processing of this flow ends. If it is continuously detected for a certain period of time, it is determined in step S413 that it is abnormal (need attention), and the processing of this flow is terminated.

《Left and right movement of the neck》
In step S414, the left and right positions of the neck are determined based on the images of 5 to 10 frames per second.
In step S415, an image for 1 second, for example, an image of 1/10 to 10/10 in the case of 10 frames per second is inspected.
In step S416, an image (for example, 10 frames) for 1 second from 1 second to 2 seconds is inspected.

After 1 second has elapsed in step S417, it is determined whether or not two or more frames are detected within 2 seconds.
If two or more frames are not detected within 2 seconds after 1 second has elapsed, it is determined in step S418 to be normal, and the processing of this flow is terminated.

If two or more frames are detected within 2 seconds after 1 second has elapsed in step S419, it is determined in step S419 that there is a possibility of abnormality, and the process proceeds to step S408.
Temporal verification is started in step S408. The temporal verification by the left and right movement of the neck is as follows.
That is, in step S420, the next 1-second image (for example, 10 frames) is inspected. Next, in step S421, the temporal verification of step S409 is repeated for 1 to 2 minutes to improve the accuracy.

《Neck up / down movement》
In step S422, the upper and lower positions of the neck are determined based on the images of 5 to 10 frames per second.
In step S423, an image for 1 second, for example, an image of 1/10 to 10/10 in the case of 10 frames per second is inspected.
In step S424, an image (for example, 10 frames) for 1 second from 1 second to 2 seconds is inspected.

After 1 second has elapsed in step S425, it is determined whether or not two or more frames are detected within 2 seconds.
If two or more frames are not detected within 2 seconds after 1 second has elapsed, it is determined in step S426 that it is normal, and the processing of this flow is terminated.

If two or more frames are detected within 2 seconds after 1 second has elapsed in step S425, it is determined in step S427 that there is a possibility of abnormality, and the process proceeds to step S408.
Temporal verification is started in step S408. The temporal verification by the vertical movement of the neck is as follows.
That is, in step S428, the next 1-second image (for example, 10 frames) is inspected. Next, in step S429, the temporal verification of step S428 is repeated for 1 to 2 minutes to improve the accuracy.

By judging the movement of the eyes, the left and right movement of the neck, and the up and down movement of the neck by the above flow, it is possible to increase the accuracy when judging whether the person is a well-meaning person or a malicious intruder, and the risk level can be increased. The determination can be made more accurately.

<Application example 5>
FIG. 29 is a flowchart showing an outline of the operation of the digital smart defense security system and the family / guardian of the automatic voice warning according to the person in the security area in the application example 5 of the digital smart defense security system 1000. .. This flow executes a function of performing dialogue and warning by image and voice.
The flow of << family / guardian >> in FIG. 29 is not CPU processing, but will be described with step number F for convenience.

《Digital Smart Defense Security System》
The digital smart defense security system 1000 detects that a visitor has invaded the yellow belt Y (see the upper part of FIG. 9) (step S501).
It is determined whether the security mode is the constant security mode or the outing / home security mode (see the bottom of FIG. 6) (step S502).
In the case of the constant security mode, steps S503 to S511 are executed. Further, in the case of the outing / home security mode, steps S512 to S519 are executed.

In the constant security mode, an automatic voice is used to tell the user, "Please show the QR code to the camera of the doorphone" (step S503).
The QR code is identified and collated with the registrant information DB 165 (see FIG. 1) (step S504).
The visitor determines whether or not the information has been registered by collating with the registrant information DB 165 (step S505).
If the visitor has already been registered (step S505: Yes), the visitor is notified by automatic voice that the visitor has been registered, and the flow process ends (step S506).

When the visitor is not registered in step S505 (step S505: No), the visit of the unregistered person is notified to the guardian's smartphone 50a (see FIG. 1) (step S507).
The dialogue with the doorphone is switched to the dialogue with a remote guardian (step S508).

It is detected that a visitor has invaded the red belt R (see the upper part of FIG. 9) (step S509).
A message such as "Register your image and notify related organizations" is sent by automatic voice (step S510).
Using the "doorphone type dialogue system" (see in FIG. 9), the response from the visitor is acquired by the microphone input and sent to the remote guardian's smartphone 50a (step S511).

On the other hand, when the security mode is the outing / home security mode (see the bottom of FIG. 6) in step S502, a message such as "This area is currently being guarded" is sent by automatic voice (step S512).
It is detected that a visitor has invaded the orange belt O (see the upper part of FIG. 9) (step S513).
A message such as "Please move out of this area immediately" is sent by automatic voice (step S514).
Notify the family's smartphone 50a that there is an intrusion (step S515).
It is detected that a visitor has invaded the red belt R (see the upper part of FIG. 9) (step S516).
A message such as "Register your image and notify related organizations" is sent by automatic voice (step S517).
Using the "security camera type dialogue system" (see the bottom of FIG. 9), the response from the intruder is acquired by the microphone input and sent to the remote family smartphone 50a (step S518).

In addition, as shown in the middle of FIG. 9 and the bottom of FIG. 9, both the "doorphone type dialogue system" and the "security camera type dialogue system" are remote from the local system by interposing the cloud dialogue relay system. The image and sound are relayed to and from the smartphone 50a of the above (step S519, step F13).

《Family / guardian》
The family / guardian is notified by the Digital Smart Defense Security System 1000 that an unregistered person has visited (step F11). For example, a family member / guardian receives a message from the Digital Smart Defense Security System 1000 stating that a visitor who has invaded Orange Belt O has been warned that "Please leave this area immediately" ( Step S514).

The family / guardian is notified by the Digital Smart Defense Security System 1000 that an intruder has entered the premises (step F12).
When a family member / guardian receives a notification that an unregistered person has visited the site, or a notification that an intruder has entered the premises, he / she uses the smartphone 50a app to "use the application of the smartphone 50a. Please say ", etc." and listen to the response from the other party (step F13).

The family / guardian determines whether the visitor is okay (step F14).
If there is no problem with the visitor, listen to the requirements and respond (step F15). If there is a problem with the visitor, attach an image and audio and notify the relevant organizations (step F16).

As shown in Application Example 5, the digital smart defense security system 1000 warns by automatic voice according to the position of an intruder, and realizes a remote dialogue function by image and voice.

In addition, the Digital Smart Defense Security System 1000 provides the following image privacy protection features in all security modes.
-It has a function to protect privacy by applying a mosaic to the images of detected humans such as passers-by and neighboring residents outside the boundary line of the site (road / adjacent land), public roads, adjacent land, etc.
-The function to remove the mosaic from the image when it is detected that the above passersby, neighboring residents, etc. have invaded the inside of the boundary line of the site (road, adjacent land), yellow belt Y, orange belt O, or red belt R. Have.

<Application example 6>
FIG. 30 is a flowchart showing an outline of detection, registration, and forwarding of a suspicious call in Application Example 6 of the Digital Smart Defense Security System 1000. This flow automatically detects suspicious calls, registers them, and forwards them to the guardian's smartphone when judgment is required.
The flow of << guardian >> in FIG. 30 is not CPU processing, but will be described with step number F for convenience.

《Digital Smart Defense Security System》
The digital smart defense security system 1000 confirms a call made by a suspicious person to a home telephone targeting an elderly person or the like (step S601).
The digital smart defense security system 1000 responds before the elderly or the like picks up the home phone (step S602).
It is determined whether or not the caller ID has been obtained (step S603).

If the caller ID can be obtained, the caller ID is collated with the telephone directory (step S604).
It is determined whether it is registered or unregistered (step S605).
In the case of normal registration, the home telephone is ringed (step S606).
When registered as a suspicious person, it is determined that the person is a suspicious person (step S607) and the telephone is hung up (step S608).
If not registered, the process proceeds to step S610.

On the other hand, if the caller ID cannot be obtained in step S603, it is determined that the other party is hiding the caller ID (there is a possibility of a suspicious person), and the process proceeds to step S610 (step S609).

In step S610, you must be a suspicious person with an automatic message such as "You are in wire fraud countermeasure mode. Record this call .... Please tell us your name and business or your institution." Send a message for those who are suspected of.
The caller ID (if obtained) is registered in a predetermined DB (not shown), and telephone recording is started (step S611).
It is determined whether or not the name, the matter, and the institution to which the person belongs have been mentioned (step S612).
If you do not say your name, requirements, or institution to which you belong, it is determined that you are an unregistered suspicious person (step S613), and you hang up (step S614).

When the name, requirements, and affiliated institution are stated, it is determined whether or not the affiliated institution is a police, government office, financial institution, or the like (step S615).
If the institution to which you belong is a police, government office, financial institution, etc., an automatic message such as "Call back" is sent (step S616) and the call is hung up (step S617).

When the affiliated institution is not the police, government office, financial institution, etc., the attention level of the other party is determined by voice recognition (step S618). Specific methods for determining the attention level of the other party by voice recognition have been described with reference to FIGS. 11 to 12.
Transfer to the parent's smartphone 50a (see FIG. 1) through a cloud-type IP phone transfer system (not shown) (step S619).
In addition, as a prerequisite for using this function, (1) a Hikari Denwa router, etc. is installed at home, and (2) a guardian installs an IP phone (VoIP: Voice over Internet Protocol) application on a smartphone. It is necessary to (3) contract services such as number display with the telephone company.

《Guardian》
Receive the incoming call transferred to the smartphone 50a (step F21).
Confirm the other party to call (step F22).
It is determined whether or not the other party of the call can be trusted (step F23).
If the other party on the phone can be trusted, it is judged to be normal (step F24) and the requirements are heard (step F25).

If the other party on the phone cannot be trusted, it is determined that the person is an unregistered suspicious person (step F26), the caller ID is registered as a suspicious person (step F27), and the call is hung up (step F28).

As shown in Application Example 6, when a suspicious person makes a call to a home phone aiming at an elderly person or the like, the digital smart defense security system 1000 responds before the elderly person or the like picks up the home phone and makes a suspicious call. Is automatically detected and registered, and when a judgment is required, it is transferred to the guardian's smartphone 50a. As a result, it is possible to prevent apo-den robbery and transfer fraud at the water's edge before the elderly or the like pick up the phone at home.

<Application example 7>
FIG. 31 is a flowchart showing a function of determining the attention level of the other party by voice recognition in the application example 7 of the digital smart defense security system 1000. This flow is a specific example of determining the attention level of the other party by voice recognition described in FIGS. 11 to 12.
The flow of << suspicious person >> in FIG. 31 is not CPU processing, but will be described with step number SP for convenience.

"Suspicious person"
A suspicious person tries to impersonate a child, grandchild, etc. by aiming at an elderly person (step SP11). For example, in the so-called "Oreore fraud," you call by pretending to be your son or grandson and make them believe that you are a true relative, saying that "the phone number has changed" and "the voice has changed because you have a cold." In some cases, they only tell them that their contact information has changed at the beginning, and after a while, they start talking about "actually, I need money ...". After that, explain the situation where you need money right now, such as "I have a traffic accident and have to pay the settlement fee", and tell them to transfer it to the designated account.

A suspicious person tries to find out the PIN code of a cash card from an elderly person or the like (step SP12). For example, in the so-called "refund fraud", the "social insurance secretariat" and "○○ city withholding section" are called while pretending to be employees of a public institution, and "there is a refund of the pension" and "a part of the insurance premium". Will be back. " Also, at this time, you can read the number of the target balance from the right by saying "It will be the number to receive", or ask for the account number.

A suspicious person tries to find out how much cash is in the house by aiming at elderly people (step SP13). For example, in so-called "loan insurance fraud" and "fictitious billing fraud", you send a postcard with a name and logo mark that resembles a major financial institution, and then say "a certain amount of deposit is required first" and so on. Request to transfer.
In this way, the suspicious person skillfully guides the elderly and the like to ask for the PIN of the cash card or transfer it to a designated account.

As mentioned above, all of them are triggered by a phone call from a suspicious person. As a security function against telephone fraud, the digital smart defense security system 1000 determines the degree of attention of the other party by voice recognition and determines a suspicious person from the telephone conversation to prevent transfer fraud and the like.

《Digital Smart Defense Security System》
The digital smart defense security system 1000 sends the voice of the other party to the voice recognition unit 112 (see FIG. 12) (step S701).
Time verification of special fraud, apo-den robbery, etc. is started (step S702).
Acquire the voice (step S703).

The acquired voice is collated with the dictionary 135 (see FIG. 12) with the frequency of appearance of vocabulary used in special fraud, apo-den robbery, etc. (step S704). The digital smart defense security system 1000 acquires the voice of the telephone, and the voice recognition unit 112 refers to the dictionary 135 with the frequency of appearance and determines the content of the conversation by collating words. For example, in a telephone conversation, the vocabulary such as "I", "phone number has changed", "settlement fee", "don't tell anyone", "transfer", "urgent", "refund", "account", etc. is normal. When it appears more frequently than the vocabulary that appears in, it is judged that the other party's attention is high. In particular, if a vocabulary that is directly linked to special fraud, such as a "password", is included, or if these vocabularies appear in combination, it is determined that the possibility of fraud is higher.

Find the probability of special fraud, apo-den robbery, etc. (step S705).
It is determined whether or not the probability is above a certain level (for example, 0.9 or above) (step S706).
If the probability does not reach a certain level, it is determined to be normal and this flow is processed (step S707).

If the probability is above a certain level, it is determined that there is a possibility of abnormality (step S708).
The next voice is acquired and a new probability is obtained (step S709).
It is determined whether or not the maximum determination time (for example, 10 seconds) has been reached (step S710).
If the maximum determination time has not been reached, the determination is continued and the process returns to step S707 (step S711).
When the maximum determination time is reached, it is determined whether or not the probability is at a certain level or higher (for example, 0.9 or higher) (step S712).
If the probability does not reach a certain level, it is determined to be normal and this flow is processed (step S713).
If the probability is equal to or higher than a certain level, it is determined that the person is an unregistered suspicious person, and this flow is processed (step S714).

As shown in Application Example 7, when a suspicious person makes a call to a home phone aiming at an elderly person, etc., the suspicious person can be determined from the telephone conversation, and apo-den robbery and transfer fraud can be prevented. Can be done.

As described in detail above, according to the present embodiment, the digital smart defense security system 1000 (see FIG. 1) detects a person in a security area in which a plurality of areas are set according to the degree of risk. The person detection unit 111 and the person detected by the person detection unit 111 are warned by the voice corresponding to the area, and the notification unit 115 that reports the existence of the suspicious person and the voice of the conversation are acquired and the conversation is performed. The reporting unit 115 includes a voice recognition unit 112 that determines that the content of the conversation is suspicious, and the reporting unit 115 receives that the voice recognition unit 112 determines that the content of the conversation is suspicious. Report the existence of.

In addition, it is possible to prevent a suspicious person from entering the house by reporting the existence of the suspicious person in response to the determination that the content of the conversation is suspicious. Even if the suspicious person cannot be determined only by determining the position of the suspicious person in the security area, it can be determined that the suspicious person is a suspicious person from the conversation with the suspicious person. Conversations are not limited to people in the security zone, as do telephone conversations.

In the present embodiment, the voice recognition unit 112 recognizes the voice of the conversation of the intercom installed at the entrance, and through the conversation of the intercom, the visitor who pretends to be a delivery person, a salesman, a bank employee, etc. is a suspicious person. It can be determined that there is, and it is possible to prevent a suspicious person from entering the house.

In the present embodiment, the digital smart defense security system 1000 includes an ID terminal detection unit 113 that detects an ID terminal possessed by a non-intruder, and the reporting unit 115 possesses an ID terminal detected by the ID terminal detection unit 113. By not reporting the person who is doing so, for example, the smartphones of family members and related persons can be detected and excluded from suspicious persons, unnecessary reports can be reduced, and monitoring can be effective. In addition, resources for monitoring can be reduced to reduce costs.

In the present embodiment, the voice recognition unit 112 can determine a suspicious person from a telephone conversation by recognizing the voice of a telephone call, and can prevent apoden robbery and transfer fraud. It should be noted that telephone conversations can be similarly applied to words in emails.

In the present embodiment, the digital smart defense security system 1000 includes a dictionary 135 with an appearance frequency that stores vocabulary used in suspicious conversations, and the voice recognition unit 112 uses the words that appear in the acquired voice with the appearance frequency. By collating with the vocabulary of the dictionary 135 and determining that the content of the conversation is suspicious, the accuracy of suspicious person determination can be improved, and apo-den robbery and transfer fraud can be reliably prevented. ..

In the present embodiment, the digital smart defense security system 1000 includes an AI accelerator 200 which is a computing resource other than the CPU, and the person detection unit 111 of the control unit 110 uses the AI accelerator 200 to detect a person in the security area. To detect. The AI accelerator 200 can detect a person existing in a wide security area in real time by executing a person detection process on a dedicated hardware separately from the CPU process. In addition, a person can be detected in real time even with a configuration using an inexpensive camera device.
In addition, the AI accelerator 200 can detect an intruder with extremely high accuracy as compared with image recognition by a moving object detection type surveillance camera using a conventional image difference.

The above description is an example of a preferred embodiment of the present invention, and the scope of the present invention is not limited thereto.
For example, in FIGS. 5 and 6, the security mode is changed from the outside by using the smartphone 50a, but the user may manually change the security mode.

As described above, the digital smart defense security process of the digital smart defense security system 1000 includes (I) "warning by automatic voice according to the position of the intruder and remote dialogue processing by image / voice" ( There are (FIG. 9, FIG. 10, FIG. 29, etc.) and (II) "suspicious determination processing based on the content of the conversation" (FIG. 11, FIG. 30, FIG. 31, etc.). The above (I) and the above (II) may be executed individually or in combination. In the above (I), a quick and appropriate alarm can be output mainly according to the position of the person in the security area, and in the above (II), it is mainly determined that the person is a suspicious person with higher accuracy. Can be done.

Further, in the above embodiment, the names of digital smart defense security system and method are used, but this is for convenience of explanation, and may be a monitoring system, a security system, a smart defense security method, or the like.

The digital smart defense security system and method of the present invention are also realized by a program for making a computer function as the digital smart defense security system or method of the present invention. The program may be stored on a computer-readable recording medium.

The recording medium on which this program is recorded may be the ROM itself of the Digital Smart Defense Security System, or a program reading device such as a CD-ROM drive is provided as an external storage device, and the recording medium is placed therein. It may be a CD-ROM or the like that can be read by inserting it.

Further, each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. In addition, each of the above configurations, functions, and the like may be realized by software for the processor to interpret and execute a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in memory, hard disks, recording devices such as SSDs (Solid State Drives), IC (Integrated Circuit) cards, SD (Secure Digital) cards, optical disks, etc. It can be held on a recording medium.

All publications, patents and patent applications cited herein are incorporated herein by reference only.

The digital smart defense security system, method and program according to the present invention are expected to be installed in business establishments requiring a high degree of management such as residences, corporate offices, factories, research institutes, information processing rooms, and money counting rooms. Will be done. In addition, residential, commercial facilities, offices, hospitals, hotels, financial institutions, factories, laboratories, power plants, air terminals, meeting places, stadiums, museums and other buildings indoors and outdoors, transportation trains, ferries, airplanes The inside of the car is also covered.

11 Surveillance camera (shooting means)
20 Motion sensor 30 Wi-Fi master unit 40 Beacon master unit 50 Mobile terminal device (ID terminal)
50a Smartphone (mobile terminal device; ID terminal)
51 Wi-Fi handset 52 Beacon handset 53 GPS
100 Monitoring device 110 Control unit 111 Person detection unit (person detection means)
112 Voice recognition unit (voice recognition means)
113 ID terminal detection unit (ID terminal detection means)
114 ID terminal registration unit (ID terminal registration means)
115 Reporting section (reporting means)
120 Input unit 130 Storage unit 135 Dictionary with frequency of appearance (suspicious vocabulary storage means)
140 Display unit 150 Output unit 160 Recording unit 165 Registrant information DB
170 Image processing unit 180 Interface (I / F) unit 190 Communication unit 200 AI accelerator (human detection means)
1000 Digital Smart Defense Security System Y, O, R Security Belt

Claims

A person detection means that detects people in security areas where multiple areas are set according to the degree of risk,
A digital smart defense security system characterized by providing a reporting means for notifying the existence of a suspicious person while giving a voice warning corresponding to the area to a person detected by the person detecting means.
A voice recognition means for acquiring the voice of the conversation and determining that the content of the conversation is suspicious is provided.
The digital smart defense according to claim 1, wherein the reporting means reports the existence of a suspicious person in response to the determination by the voice recognition means that the content of the conversation is suspicious. Security system.
The digital smart defense security system according to claim 2, wherein the voice recognition means recognizes the voice of conversation of an intercom installed at the entrance.
It is equipped with an ID terminal detecting means for detecting an ID (Identification) terminal possessed by a non-intruder.
The digital smart defense security system according to any one of claims 1 to 3, wherein the reporting means does not report a person who has the ID terminal detected by the ID terminal detecting means.
The digital smart defense security system according to claim 2, wherein the voice recognition means recognizes the voice of a telephone call.
Equipped with a suspicious vocabulary storage means that stores the vocabulary used in suspicious conversation
2. The voice recognition means is characterized in that the acquired words appearing in the voice are collated with the vocabulary of the suspicious vocabulary storage means to determine that the content of the conversation is suspicious. Described digital smart defense security system.
A person detection step that detects people in a security area where multiple areas are set according to the degree of risk,
A digital smart defense security method including a notification step for notifying the presence of a suspicious person while giving a voice warning corresponding to the area to a person detected by the person detection step.
Computer,
A person detecting means for detecting a person in a security area in which a plurality of areas are set according to the degree of danger and a person detected by the person detecting means are warned by a voice corresponding to the area. A program to function as a digital smart defense security system equipped with a means of reporting the existence of a suspicious person.