CN105894702B - Intrusion detection alarm system based on multi-camera data fusion and detection method thereof - Google Patents
Intrusion detection alarm system based on multi-camera data fusion and detection method thereof Download PDFInfo
- Publication number
- CN105894702B CN105894702B CN201610449982.1A CN201610449982A CN105894702B CN 105894702 B CN105894702 B CN 105894702B CN 201610449982 A CN201610449982 A CN 201610449982A CN 105894702 B CN105894702 B CN 105894702B
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- invader
- module
- control system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 36
- 230000004927 fusion Effects 0.000 title claims abstract description 18
- 238000007500 overflow downdraw method Methods 0.000 title 1
- 238000012545 processing Methods 0.000 claims abstract description 45
- 238000000034 method Methods 0.000 claims description 19
- 230000003068 static effect Effects 0.000 claims description 10
- 238000007689 inspection Methods 0.000 claims description 6
- 230000033001 locomotion Effects 0.000 claims description 5
- 230000004044 response Effects 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000003044 adaptive effect Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 claims description 2
- 239000013307 optical fiber Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 abstract description 27
- 238000004519 manufacturing process Methods 0.000 abstract description 2
- 238000004458 analytical method Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 239000011159 matrix material Substances 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 238000013519 translation Methods 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 3
- 206010021703 Indifference Diseases 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
- G08B13/19602—Image analysis to detect motion of the intruder, e.g. by frame subtraction
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
- G08B13/19639—Details of the system layout
- G08B13/19645—Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Closed-Circuit Television Systems (AREA)
- Burglar Alarm Systems (AREA)
Abstract
The invention discloses an intrusion detection alarm system based on multi-camera data fusion and a detection method thereof, belonging to the technical field of video monitoring. The system comprises a common camera, a pan-tilt camera, a zooming wide-angle multi-target tracking system, a PC type hard disk video recorder, a control system, a display, a multi-stage alarm processing module and an intruding object identification and processing module, wherein the multiple cameras are used for covering a required monitoring area in a full range, a shot video signal is digitized and compressed and then transmitted to the control system, the intruding object is identified and processed, and the corresponding alarm level is responded. The invention is suitable for scenes or passages with higher requirements on safety production, such as places of factories, tunnels, warehouses, airports and the like, and can monitor and alarm under different conditions.
Description
Technical field
The invention belongs to technical field of video monitoring, is that one kind is related to image procossing, video analysis and mould more specifically
The intrusion detection alarm method of the technologies such as formula identification.
Background technology
With economic level development and science and technology continuous progress, security protection consciousness is gradually rooted in the hearts of the people, people for
The demand of security protection is also improving constantly.An important technology application of the video monitoring as safety-security area is personal house and each
Industry field provides safety guarantee.In recent years, video monitoring system flourishes in each field in China, and video monitoring is deep
Enter each corner to social public security, place more especially higher to security requirement, as plant area, tunnel,
Warehouse, airport etc., played a significant role in terms of support personnel and property safety.
Intelligent, high Qinghua, networking are three main trend of video monitoring development, and the appearance of intelligent monitoring technology is exactly intelligence
The direct embodiment of this trend can be changed.On the one hand due to the raising of intelligent video analysis accuracy, the enhancing of anti-interference, intelligence
Video analysis is widely used in every field；On the other hand, current video intelligent compress technique and institute is needed for different
The various Intellectual Analysis Technologies of exploitation, can not only lift precision of analysis, while can also greatly promote detection effect
Rate, meet the intellectual analysis application of various special screnes.Meanwhile the concept of a large amount of concern intelligent monitorings is also begun in the industry, it is numerous
Producer starts intelligentized security product of researching and developing and put into production one after another, and patent associated therewith and scientific paper are also increasingly
It is more.For example, Chinese Patent Application No. is 201210294901.7, application publication date is in August, 2013 patent application of 14 days text
Part discloses a kind of monitoring method of the intelligent video monitoring system based on multi-cam data fusion, mainly includes two sides
Face：First, the operation of overall application program, second, video processnig algorithms API design and algorithm packaging, overall application program passes through
To video processnig algorithms API calling, video processing function is realized；The flow of overall application program is as follows：Monitoring system according to
The function of concrete configuration, into corresponding processing module, global monitoring and the area-of-interest monitoring to monitor area are realized, its
Middle configuration information can be configured by Man Machine Interface, communication or cable network；Monitoring system includes numeral
Video input and analog video input two kinds of monitoring system schemes.
However, problems also be present in current intelligent monitoring, it is main as follows：It is excessively simple to the analyzing and processing of monitoring objective
Singly, special messenger has been also needed to during the situation for complexity occur and has ceaselessly checked monitor, wasted time and energy, and easily miss important picture
Face information；Static object can not be identified and multiple mobile targets are tracked, while be unable to the particular location letter of feedback target
Breath；Shooting accuracy of the currently used wide angle cameras to abnormal object be not high, dead angle be present, can only often photograph one
Profile substantially, brings inconvenience to subsequent analysis；Monitoring objective indifference is treated, it is impossible to which classification is handled well, can not divided
The order of importance and emergency.Comparatively speaking, the present invention takes corresponding algorithm plan based on multiple-camera data for different doubtful situations
Slightly solves above mentioned problem well.
The content of the invention
1. to solve the problems, such as
Monitoring dead angle be present for existing monitoring system, need special messenger not stop to check monitor, to invasion to complex situations
The problems such as object indifference is treated, the present invention provide a kind of intrusion detection warning system based on multiple-camera data fusion and its
Detection method, pass through control system and common camera, monopod video camera, zoom wide-angle multi-target tracking system, multi-level alarm
Processing module is connected, and can realize and monitor area is covered without dead angle, Intelligent Measurement is carried out to the object of intrusion, according to detection
As a result different monitor modes is taken, makes monitoring system more flexibly and intelligent.
2. technical scheme
In order to solve the above problems, the technical solution adopted in the present invention is as follows：
A kind of intrusion detection warning system based on multiple-camera data fusion, including fore-end, hop, control
Partly, display portion, multi-level alarm processing module and power supply module, described fore-end includes common camera, head is taken the photograph
Camera and zoom wide-angle multi-target tracking system, for obtaining shooting information and being transferred to control section；Described control section
Obtained including the identification of PC formulas DVR, control system, cradle head controllor and invader and processing module, PC formula DVRs
Handled after taking the shooting information that fore-end transmits and the camera picture after processing is transferred to control system；It is described
Control system call invader identification and processing module to PC formulas DVR processing after camera picture information sentence
It is disconnected, complete the response of multi-level alarm processing module and the display of display picture；Described hop is used to transmit leading section
Divide, the data between control section, display portion and multi-level alarm processing module；Described power supply module supplies for whole system
Electricity.
Preferably, described common camera and monopod video camera are correspondingly arranged on same position, and its position is entered
Line number.
Preferably, described hop is transmitted using optical fiber, has and the characteristics of low, transmission quality is high is lost.
Preferably, described zoom wide-angle multi-target tracking system includes zoom wide-angle multi-target tracking video camera, superelevation
Fast intelligent sphere and embedded video server.
Preferably, the identification of described invader and processing module include sound state recognizer module, stationary body contrasts
Determining program module, stationary body contour detecting program module, recognition of face contrast program module, target finder module.
A kind of detection method of above-mentioned intrusion detection warning system based on multiple-camera data fusion, detection process
For：The image information of shooting is transferred to PC formula DVRs, PC formula HD recordings by the fore-end in detection warning system
Machine after shooting picture processing to being shown in display portion；When there is invader to occur, control system call invader identification and
Target finder module Real-time Feedback its positional information in processing module, while sound state recognizer module is called to general
Logical shot by camera image information carries out sound state identification；When invader is motionless object, control system is called static
Object contrast determining program module compared with preset threshold value and is judged invader, then passes through display if less than threshold value
Part directly displays and stored the information captured by common camera, and responds multi-level alarm processing module and send 3 grades of alarms
Signal；If greater than threshold value, control system then calls stationary body contour detecting program module to judge the specific profile of invader,
Then the monopod video camera for sending command adapted thereto control reference numeral position is shot to its profile inspection, returns to shooting letter in real time
Breath, while respond multi-level alarm processing module and send 2 grades of alarm signals；When invader is mobile object, control system is adjusted
Recognition of face contrast is carried out to it with recognition of face contrast program module, if comparing result is not staff or can't detect people
Face information, then shooting is tracked to invader by zoom wide-angle multi-target tracking system, and responds multi-level alarm processing mould
Block sends 1 grade of alarm signal.
Preferably, described target finder module using the Camera Calibration Algorithm of radial arrangement restraint (RAC) come
The inside and outside parameter of monocular-camera is obtained, then recovers three-dimensional coordinate of the target object in world coordinate system, specifically
Step is：
(1) camera model that uses of the present invention is the pinhole camera model that is distorted with lens radial direction single order, if
(xw,yw,zw) be object P in three-dimensional world coordinate system coordinate, (x, y, z) is three-dimensional coordinates of the P in camera coordinate system.
XOiY is center in OiPoint is parallel to x, the image coordinate system of y-axis.(xu,yu) be the P points under preferable pinhole camera model figure
As coordinate, (xd,yd) it is the deviation (x as caused by lens distortionu,yu) real image coordinate.The coordinate of image in a computer
Unit is pixel count, so need to be by the way that the three-dimensional coordinate of object point is transformed into plane of delineation coordinate.
(2) RAC calibrating camera parameters.N number of coplanar characteristic point is chosen first, determines the image coordinate and generation of this N number of point
Boundary's coordinate.Because the external parameter and inner parameter that need to demarcate are respectively 6, and proportionality coefficient and picture centre point coordinates etc.
Parameter is fixed, and RAC two-stage calibration methods include obtaining most calibrating parameters with analytic solutions and estimate that processing radial distortion is produced with iterative method
Three parameters such as raw effective focal length, the depth constituent element of translation vector.The spin matrix in rigid body translation can be determined using RAC
R and translation matrix T Tx, TyComponent.RAC is represented by direction
IfBy rigid body staticsIn RACIt can obtain：
[xwYd ywYd zwYd Yd -xwYd -xwYd -zwYd]·[r1/Ty r2/Ty r3/Ty Tx/Ty r4/Ty r5/Ty
r6/Ty]T=Xd
Row vector is, it is known that column vector is parameter to be asked.Determined using conplane space point coordinates, choose world coordinates
System, makes zw=0, then above formula be represented by：
R can then be solved1,r2,r4,r5Four independent variables, orthogonal matrix add a ratio (1/Ty) also there are four independent changes
Amount, therefore can determine that spin matrix R and translational component Tx,Ty。
(3) effective focal length f, T are calculatedzComponent and distortion coefficient k.To each characteristic point PiCan have：
If wi=r7xwi+r8ywi, it is contemplated that zi=wi+Tz, Yu=(Yf-Yc)/NY, above formula is represented by：
[yi -(Yf-Yc)/NY]·[f Tz]T=(Yf-Yc)wi/NY
K=0 is made, solution can try to achieve effective focal length f, translation matrix T T respectivelyzComponent.
(4) position (X according to target in the picturef,Yf) and step (2) (3) in try to achieve camera parameters and can solve target
Position (the x in world coordinate systemw,yw,zw)。
Preferably, described sound state recognizer module uses background differential technique, concretely comprises the following steps：
(1) initialization background：The image for choosing one section of video is averaged, then with the gray value of the average image with it is each
The gray value of two field picture subtracts each other gained absolute value, and the subregion of least absolute value correspondence image forms current background image；
Seek the average value of 3 two field picture gray valuesThe gray value and average value of each two field picture subtract each other exhausted
To value, formula is：
Background image is considered as image section corresponding to minimum value, wherein i≤3, and formula is：
(2) obtain and work as the continuous image of former frames, current background image F is tried to achieve according to step (1)i+1(x,y)
F (x, y)=λ Fi(x,y)+(1-λ)Fi+1(x,y)
In formula：F (x, y) is newest background image, Fi(x, y) is last time background image, Fi+1(x, y) is current new acquirement
Background image, Fj(x, y) and Fj+1(x, y) is continuous several two field pictures, and λ is weights.The shadow changed in view of noise and background
Ring, when | Fj+1(x,y)-Fj(x,y)|<T, current frame image are taken as background image.
(3) image currently gathered and background image are made into calculus of differences, when result exceedes certain threshold value, can determine whether figure
Corresponding pixel belongs to motion target area as in, produces corresponding binary image, and further use inter-frame difference
Method, current frame image and previous frame image subtraction if difference is less than certain threshold value, count to the pixel.
When count value Count exceedes certain threshold value, illustrate the pixel oneself through incorporate background, be judged as static state；It is on the contrary then judge mesh
Mark is motion.
Preferably, the specific implementation step of described stationary body contrast determining program module is：
(1) adjacent two frame in video sequence is subjected to difference first, obtains difference image, be described as follows with formula：
Dk(x, y)=| Ik+1(x,y)-Ik(x,y)|
Dk(x,y)、Ik+1(x,y)、Ik(x, y) corresponds to difference image, the frame of kth+1 and kth frame original image respectively；
(2) threshold division is carried out to difference image and obtains the foreground image of binaryzation, when foreground image is less than segmentation threshold
It is otherwise 1 labeled as 0 during value；
(3) binaryzation foreground image contour area and the ratio M of background image entire area are calculated.
Preferably, described stationary body contour detecting program module uses Canny edge detection algorithms, and it is embodied
Step is：
(1) noise is eliminated：Generally, using Gaussian filter convolution to image noise reduction.
(2) gradient magnitude and direction are calculated：
UseCalculate gradient magnitude and direction.
(3) non-edge pixels is excluded using non-maxima suppression, only retains candidate edge.
(4) hysteresis threshold：If a certain location of pixels amplitude exceedes high threshold, the pixel is left edge pixel；Such as
Fruit is less than Low threshold, then excludes；If between two threshold values, the pixel is only being connected to a picture for being higher than high threshold
It is retained when plain.
Preferably, described monopod video camera and zoom wide-angle multi-target tracking system receive the finger that control system is sent
Order, the static invader that monopod video camera is more than preset threshold value to specific physical features are shot；Zoom wide-angle multiple target chases after
The track system multiple mobile invaders of tracking lock, and can further and track invader to obtain feature, display picture simultaneously
Switch between multiple targets.
Preferably, described monopod video camera has adaptive zoom function, according to the size and distance of camera lens of invader,
Automatic focus adjustable, obtain high-resolution image information；Monopod video camera has two respectively in both horizontally and vertically mobile
Individual different motor driving, when needing to call, issues a command to cradle head controllor, and then control head shooting under control system
Machine is shot.
In the present invention, PC formula DVRs can be carried out remotely after connecing interchanger using mobile devices such as mobile phone, flat boards
Access；Display portion can be the mobile devices such as display or mobile phone plane plate；Fore-end is carried out without dead to wanted monitor area
Angle all standing is laid, and fore-end shooting picture (includes the sound state of invader by reception, the processing of PC formula DVRs
Detection, stationary body contrast judge, mobile object recognition of face and target positioning) and transmission etc. operate after be shown in display
On, and real-time storage is conveniently checked and managed afterwards, multi-level alarm processing module is connected with PC formula DVRs.Control system
Unite and sound state identification is carried out to the abnormal object in common camera, when invader is motionless object, by its specific thing
Compared with managing the feature such as threshold value preset with system such as size, length, then directly display and store if less than threshold value and be general
The information of logical shot by camera, and respond multi-level alarm processing module and send 3 grades of alarm signals；If greater than threshold value, control
The monopod video camera of system control reference numeral position processed carries out inspection shooting to its specific profile, and it is same to return to photographing information in real time
When response multi-level alarm processing module send 2 grades of alarm signals.When invader is mobile object, control system is carried out to it
Recognition of face contrasts, and if not staff or can't detect facial image, then controls zoom wide-angle multi-target tracking system
It is shot, returning to photographing information in real time, responding multi-level alarm processing module sends 1 grade of alarm signal simultaneously.
Monopod video camera is realized by two operating motors both horizontally and vertically rotated, and motor receives to come from
The signal of control system is accurately positioned and rotated.The motorized zoom lenses of monopod video camera can realize the optics such as aperture, focusing
Adjustment, photographed scene is furthered, pushes away the image information that fine definition is far obtained with this.
Zoom wide-angle multi-target tracking system is embedded by third party's video camera, a ultrahigh speed intelligent sphere and one
Video server forms.When there is multiple targets to occur, embedded video server is controlled by running embedded algorithm routine
The feature video and photo of each target are followed the trail of in the timesharing of ultrahigh speed intelligent sphere, and system can automatically control the movement of ultrahigh speed intelligent sphere
And select optimal focal length realize picture optimize display, while monitor picture can between multiple targets free switching.
Whether zoom wide-angle multi-target tracking system starts is determined by control system to the feature recognition result of invader.
Multi-level alarm processing module is connected in PC formula DVRs, control system picture according to captured by common camera
Determine the corresponding warning level of response.
3. beneficial effect
Compared to prior art, beneficial effects of the present invention are：
(1) present invention needs special messenger ceaselessly to check monitor, easily lose important information for current monitor system
Situation, by control system by picture captured by common camera carries out the identification of sound state, stationary body contrast judges and face
The processing such as identification, different monitoring strategies are taken for different situations, are improved intelligence degree；Monopod video camera is not
It can only be shot in horizontal and vertical directions adjustment angle, and camera lens has automatic focusing function, improves resolution ratio
In order to ex-post analysis.
(2) compared with prior art, zoom wide-angle multi-target tracking system can be detected simultaneously by multiple targets to the present invention, and
And to target carry out Intelligent Recognition, display picture can also between multiple targets free switching；Multi-level alarm processing module
It is divided into 3 grades, each grade represents different monitored object respectively, has separated the order of importance and emergency, simple and clear.
(3) a kind of method of three-dimensional coordinate of the present invention using recovery target object in world coordinate system is come to invader
Positioning, its coordinate information of Real-time Feedback, facilitates doubtful situations quick response, ensures the efficiency of emergency processing.
Brief description of the drawings
Fig. 1 is the network topology structure signal of the intrusion detection warning system of the invention based on multiple-camera data fusion
Figure；
Fig. 2 is the system flow chart of the intrusion detection warning system of the invention based on multiple-camera data fusion.
In figure：1st, control system；2nd, display；3rd, PC formulas DVR；4th, cradle head controllor；5th, interchanger；6th, nothing
Gauze；7th, the mobile device such as mobile phone and flat board；A1-An, common camera；B1-Bn, monopod video camera；C, zoom wide-angle multiple target
Tracing system；S, 3 grades of alarm devices；T, 2 grades of alarm devices；U, 1 grade of alarm device.
Embodiment
The present invention is further described below with reference to specific embodiment.
Embodiment 1
As depicted in figs. 1 and 2, the intrusion detection warning system based on multiple-camera data fusion includes：Control system 1,
Display 2, PC formulas DVR 3, cradle head controllor 4, common camera A1-An, monopod video camera B1-Bn, zoom wide-angle it is more
It is target tracking system (including zoom wide-angle multi-target tracking video camera C, ultrahigh speed intelligent sphere and embedded video server), more
Level warning processing module (including 3 grades of alarm device S, 2 grades of alarm device T, 1 grade of alarm device U).Common camera A1-AnTo monitor area
Deployed to ensure effective monitoring and control of illegal activities without dead angle, monopod video camera B1-BnWith common camera A1-AnIt is corresponding to be arranged on same position and its position is compiled
Number, zoom wide-angle multi-target tracking video camera C can overall view monitoring Zone Full, it is hard that the vision signal of three is directly accessed PC formulas
The high-performance video and audio compress card of disk video recorder 3.Video/audio signal is handled after video and audio card Compress softwares by software
Shown on display 2, and can realize the picture of invader 1,4,9,16 is shown and the tracking to mobile target, at amplification
Reason.3 grades of alarm device S, 2 grades of alarm device T, 1 grade of alarm device U are connected to the input of external various alarm control units, pass through RS485
Controlling bus passes to the controlling alarm inquiry mouth of PC formula DVRs and realizes alarm linkage.Control system 1 is to PC formula hard disks
The image information that video recorder 3 transmits judges, and determines monopod video camera B1-Bn, zoom wide-angle multi-target tracking video camera C be
No work, while control PC formulas DVR 3 to respond corresponding multi-level alarm processing module.PC formulas DVR 3 will be through
The video flowing of overcompression is passed in each sub-control terminal of network by network interface card using modes such as multicasts, facilitates mobile phone and flat board etc.
Mobile device 8 is checked whenever and wherever possible.When there is invader to occur, control system 1 calls invader identification and processing module
In target finder module Real-time Feedback its positional information.
As shown in Fig. 2 sound state recognizer module uses background differential technique, its specific implementation step is as follows：
(1) initialization background.The image for choosing one section of video is averaged, then with the gray value of the average image with it is each
The gray value of two field picture subtracts each other gained absolute value, and the image formed using the subregion of the minimum correspondence image of absolute value is exactly
Current background image.
(2) obtain and work as the continuous image of former frames, current background image F is tried to achieve according to step (1)i+1(x,y)
F (x, y)=λ Fi(x,y)+(1-λ)Fi+1(x,y)
In formula：F (x, y) is newest background image, Fi(x, y) is last time background image, Fi+1(x, y) is current new acquirement
Background image, Fj(x, y) and Fj+1(x, y) is continuous several two field pictures, and λ is weights.The shadow changed in view of noise and background
Ring, when | Fj+1(x,y)-Fj(x,y)|<T, current frame image are taken as background image.
(3) image currently gathered and background image are made into calculus of differences, when result exceedes certain threshold value, can determine whether figure
Corresponding pixel belongs to motion target area as in, produces corresponding binary image, and determine whether the pixel
Whether also changed within a period of time, if continuing to change, it is dynamic to judge target, is otherwise static state.
When invader is mobile object, control system 1 calls recognition of face contrast program module, and the module uses feature
Face method, its specific implementation step are as follows：
(1) obtain the set S of staff's facial image and calculate its average image Ψ, be described as follows with formula：
S={ Г1,Г2,......ГM}
(2) the difference Φ of every image and the average image is calculated, i.e. each element in S set subtracts the average value in Ψ
Φi=Гi-Ψ
(3) with M orthogonal unit vector unΦ distributions, u are describednThe middle individual vectorial u of kth (k=1,2,3...M)kUnder
Formula can be calculated,
Wherein ukFor unit orthogonal vectors, u is calculatedkCalculate the characteristic vector of C covariance matrix, wherein A=
{Φ1,......Φn}
(4) recognition of face.When someone occurs, detect face from a frame of video flowing first and separated, so
After extract feature wk
Wherein k=1,2...M, for k-th of eigenface uk, above formula can calculate its corresponding weight, and M weight can be with
Form vectorial ΩT。
ΩT=[w1,w2,......wM]
Wherein Ω is the face to be differentiated, Ω k represent some face in training set, both with the weight of eigenface
Represent.Euclidean distance is asked to both, illustrates that k-th of face in the face and training set to be differentiated is same when distance is less than threshold value
Personal.
If comparison result is not staff or can't detect facial image, the multi-target tracking shooting of control zoom wide-angle
Machine C is tracked shooting to object, returns to photographing information in real time and responds 1 grade of alarm signal simultaneously.When the multiple mobile targets of appearance
When, can be with the picture and feature of the multiple targets of manual switching.
When invader is static state, control system 1 obtains corresponding common camera A1-AnPositional information, and call static thing
Body contrasts determining program module.Its specific implementation step is：
(1) adjacent two frame in video sequence is subjected to difference first, obtains difference image, be described as follows with formula:
Dk(x, y)=| Ik+1(x,y)-Ik(x,y)|
Dk(x,y)、Ik+1(x,y)、Ik(x, y) corresponds to difference image, the frame of kth+1 and kth frame original image respectively；
(2) threshold division is carried out to difference image and obtains the foreground image of binaryzation, when foreground image is less than segmentation threshold
It is otherwise 1 labeled as 0 during value；
(3) the ratio M of binaryzation foreground image contour area and background image is calculated.
Common camera A is directly displayed and stores if M is less than threshold value1-AnCaptured information, and respond 3 grades of reports
Alert signal；If M is more than threshold value, control system 1 first determines whether the specific profile of invader, then controls reference numeral position
Monopod video camera B1-BnInspection shooting is carried out to its specific profile, photographing information is returned in real time and responds 2 grades of alarm signals simultaneously.System
Common camera A in system1-AnNeed shooting free of discontinuities, zoom wide-angle multi-target tracking video camera C and monopod video camera B1-BnConnect
Started working after receiving instruction.Each shot by camera information can recorded in PC formulas DVR 3 in real time, facilitate thing
After check and manage.PC formulas DVR 3 is by common camera A1-An, monopod video camera B1-Bn, zoom wide-angle multiple target chases after
Track video camera C shooting pictures information is shown over the display in real time.
It is comprehensive to refer to shown in Fig. 1 and Fig. 2, the multistage intelligent monitoring and alarming system with multiple-camera is arranged on one
1000 square metres of factory, after startup, common camera A1-AnStart working.At a time, common camera A2Detect
Occurs a unknown object beside automatic producing device, control system 1 feeds back its more specific location information and by the article size
Compared with system intialization threshold value.If M is less than (1/30), common camera A is shown on display2Image information, and respond 3
Level alarm signal；If M is more than (1/30), control system first determines whether the specific profile of the object, then controls reference numeral
The monopod video camera B of position2Its profile inspection is shot.Automatic zoom camera lens can be with far and near according to target and lens location, automatically
Suitable focal length is selected to obtain clearly picture.Common camera A is shown on display2Image information, and respond 2 grades of reports
Alert signal.If now someone passes through, control system 1 intercepts one section of video flowing, detects face in a frame therefrom and carries out
Separation, then the feature extracted is compared with face characteristic in existing face database.It is not work that comparison result, which is shown,
Personnel, control system 1 send instruction control zoom wide-angle multi-target tracking video camera C and shooting are tracked to target, return in real time
Return photographing information and respond 1 grade of alarm signal, while Real-time Feedback target position information and can be over the display to it simultaneously
Carry out feature enhanced processing.When there are more people to occur, the picture of multiple targets can be realized using existing more picture monitoring softwares
With the free switching of feature.
Claims (7)
- A kind of 1. detection method of the intrusion detection warning system based on multiple-camera data fusion, it is characterised in that：Described Intrusion detection warning system based on multiple-camera data fusion includes fore-end, hop, control section, display part Point, multi-level alarm processing module and power supply module, it is wide that described fore-end includes common camera, monopod video camera and zoom Angle multi-target tracking system, for obtaining shooting information and being transferred to control section；Described control section includes PC formula hard disks Video recorder, control system, cradle head controllor and invader identification and processing module, the identification of described invader and processing module bag Sound state recognizer module, stationary body contrast determining program module, stationary body contour detecting program module, face is included to know Dui Bi not program module, target finder module；PC formulas DVR obtains the shooting information that fore-end transmits Handled afterwards and the camera picture after processing is transferred to control system；Described control system calls invader identification and place Reason module is judged the camera picture information after the processing of PC formulas DVR, completes the response of multi-level alarm processing module With the display of display picture；Described cradle head controllor is used to control monopod video camera；Described hop is used to transmit Data between fore-end, control section, display portion and multi-level alarm processing module；Described power supply module is whole system System power supply；Described common camera and monopod video camera are correspondingly arranged on same position, and its position is numbered；Inspection Survey process is：The image information of shooting is transferred to PC formula DVRs by the fore-end in detection warning system, and PC formulas are hard Disk video recorder after shooting picture processing to being shown in display portion；When there is invader to occur, control system calls invader Target finder module Real-time Feedback its positional information in identification and processing module, while call sound state recognizer mould Block carries out sound state identification to image information captured by common camera, and when invader is motionless object, control system is adjusted Invader compared with preset threshold value and is judged with stationary body contrast determining program module, then led to if less than threshold value Cross display portion and directly display and store information captured by common camera, and respond multi-level alarm processing module and send 3 Level alarm signal；If greater than threshold value, control system then calls stationary body contour detecting program module to judge the tool of invader Body profile, the monopod video camera inspection shooting of command adapted thereto control reference numeral position is then sent, returns to photographing information in real time, Responding multi-level alarm processing module sends 2 grades of alarm signals simultaneously；When invader is mobile object, control system calls people Face identification contrast program module carries out recognition of face contrast to it, if comparing result is not staff or can't detect face letter Breath, then be tracked shooting to invader by zoom wide-angle multi-target tracking system, and responds multi-level alarm processing module hair Go out 1 grade of alarm signal.
- A kind of 2. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：Described hop is transmitted using optical fiber.
- A kind of 3. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：Described zoom wide-angle multi-target tracking system includes zoom wide-angle multi-target tracking video camera, ultrahigh speed Intelligent sphere and embedded video server.
- A kind of 4. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：Described sound state recognizer module uses background differential technique, and its specific implementation step is：(1) initialization background：The image for choosing one section of video is averaged, then the gray value with the average image and each frame figure The gray value of picture subtracts each other gained absolute value, and the image of the subregion composition of least absolute value correspondence image is exactly current background Image；(2) obtain and work as the continuous image of former frames, current background image F is tried to achieve according to step (1)i+1(x,y)F (x, y)=λ Fi(x,y)+(1-λ)Fi+1(x,y)In formula：F (x, y) is new background image, Fi(x, y) is last time background image, Fi+1(x, y) is current background image, Fj(x, And F y)j+1(x, y) is continuous several two field pictures, and λ is weights；The influence changed in view of noise and background, when | Fj+1(x,y)- Fj(x,y)|<T, current frame image are taken as background image；(3) image currently gathered and background image are made into calculus of differences, when result exceedes certain threshold value, can determine whether in image Corresponding pixel belongs to motion target area, produces corresponding binary image, and determines whether the pixel one Whether also changed in the section time, if continuing to change, it is dynamic to judge target, is otherwise static state.
- A kind of 5. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：The specific implementation step of described stationary body contrast determining program module is：(1) adjacent two frame in video sequence is subjected to difference first, obtains difference image, be described as follows with formula：Dk(x, y)=| Ik+1(x,y)-Ik(x,y)|Dk(x,y)、Ik+1(x,y)、Ik(x, y) corresponds to difference image, the frame of kth+1 and kth frame original image respectively；(2) threshold division is carried out to difference image and obtains the foreground image of binaryzation, when foreground image is less than segmentation threshold, It is otherwise 1 labeled as 0；(3) binaryzation static object image outline area and the ratio M of image entire area are calculated.
- A kind of 6. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：Described monopod video camera and zoom wide-angle multi-target tracking system receives the finger that control system is sent Order, the static invader that monopod video camera is more than preset threshold value to specific physical features are shot；Zoom wide-angle multiple target chases after The track system multiple mobile invaders of tracking lock, and can further and track invader to obtain feature, display picture simultaneously Switch between multiple targets.
- A kind of 7. detection side of intrusion detection warning system based on multiple-camera data fusion according to claim 1 Method, it is characterised in that：Described monopod video camera has adaptive zoom function, according to the size and distance of camera lens of invader, Automatic focus adjustable, obtain high-resolution image information；Monopod video camera has two respectively in both horizontally and vertically mobile Individual different motor driving, when needing to call, issues a command to cradle head controllor, and then control head shooting under control system Machine is shot.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610449982.1A CN105894702B (en) | 2016-06-21 | 2016-06-21 | Intrusion detection alarm system based on multi-camera data fusion and detection method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610449982.1A CN105894702B (en) | 2016-06-21 | 2016-06-21 | Intrusion detection alarm system based on multi-camera data fusion and detection method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105894702A CN105894702A (en) | 2016-08-24 |
| CN105894702B true CN105894702B (en) | 2018-01-16 |
Family
ID=56731152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610449982.1A Expired - Fee Related CN105894702B (en) | 2016-06-21 | 2016-06-21 | Intrusion detection alarm system based on multi-camera data fusion and detection method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105894702B (en) |
Families Citing this family (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018081945A1 (en) * | 2016-11-02 | 2018-05-11 | 深圳市汇顶科技股份有限公司 | Method for detecting background noise of sensor, and device thereof |
| CN108073859A (en) * | 2016-11-16 | 2018-05-25 | 天津市远卓自动化设备制造有限公司 | The monitoring device and method of a kind of specific region |
| CN107333107A (en) * | 2017-07-21 | 2017-11-07 | 广东美的制冷设备有限公司 | Monitor image pickup method, device and its equipment |
| CN107545687A (en) * | 2017-07-21 | 2018-01-05 | 合肥未来计算机技术开发有限公司 | A kind of security protection video warning system based on sports rule |
| CN107483864B (en) * | 2017-09-15 | 2020-09-18 | 威海元程信息科技有限公司 | Monitoring video recording device |
| CN107590936A (en) * | 2017-09-20 | 2018-01-16 | 深圳市鑫汇达机械设计有限公司 | A kind of warehouse warning system based on video monitoring |
| CN107832701A (en) * | 2017-11-06 | 2018-03-23 | 佛山市章扬科技有限公司 | The character recognition method and device of a kind of monitor video |
| CN109982029B (en) * | 2017-12-27 | 2021-11-16 | 浙江宇视科技有限公司 | Automatic adjusting method and device for camera monitoring scene |
| CN108345247A (en) * | 2018-02-26 | 2018-07-31 | 杭州智仁建筑工程有限公司 | A kind of autocontrol method |
| CN108730771A (en) * | 2018-04-02 | 2018-11-02 | 上海新奥新能源技术有限公司 | A kind of gas pressure regulating station |
| CN108765466A (en) * | 2018-05-31 | 2018-11-06 | 重庆科技学院 | A kind of intelligent video monitoring system and method based on IP Camera |
| CN110716803A (en) * | 2018-07-13 | 2020-01-21 | 中强光电股份有限公司 | Computer system, resource allocation method and image identification method thereof |
| CN109241896B (en) * | 2018-08-28 | 2022-08-23 | 腾讯数码（天津）有限公司 | Channel safety detection method and device and electronic equipment |
| CN110909579A (en) * | 2018-09-18 | 2020-03-24 | 杭州海康威视数字技术股份有限公司 | Video image processing method and device, electronic equipment and storage medium |
| CN109120849B (en) * | 2018-09-19 | 2020-09-18 | 安徽文香信息技术有限公司 | Intelligent shooting method for micro-class tracking |
| CN109495723A (en) * | 2018-11-30 | 2019-03-19 | 济南浪潮高新科技投资发展有限公司 | A kind of domestic intelligent monitoring system calculated based on mist |
| CN109620177A (en) * | 2018-12-14 | 2019-04-16 | 昆明天博科技有限公司 | A kind of contactless Biont information detection alarm device and method |
| CN109639931B (en) * | 2018-12-25 | 2022-08-19 | 努比亚技术有限公司 | Photographing noise reduction method, mobile terminal and computer readable storage medium |
| CN112998475B (en) * | 2018-12-27 | 2022-10-11 | 艾感科技(广东)有限公司 | Signal induction system for triggering early warning |
| CN110087036A (en) * | 2019-04-29 | 2019-08-02 | 国家电网有限公司 | A kind of method and system for electric power tunnel safety monitoring |
| CN110110245B (en) * | 2019-05-06 | 2021-03-16 | 山东大学 | Dynamic article searching method and device in home environment |
| CN110536074B (en) * | 2019-07-17 | 2021-11-09 | 理光软件研究所（北京）有限公司 | Intelligent inspection system and inspection method |
| CN110555964B (en) * | 2019-07-25 | 2021-11-09 | 武汉烽火众智数字技术有限责任公司 | Multi-data fusion key area early warning system and method |
| CN110633648B (en) * | 2019-08-21 | 2020-09-11 | 重庆特斯联智慧科技股份有限公司 | Face recognition method and system in natural walking state |
| CN110532951B (en) * | 2019-08-30 | 2020-05-26 | 江苏航天大为科技股份有限公司 | Subway passenger abnormal behavior analysis method based on interval displacement |
| CN111063146A (en) * | 2019-12-18 | 2020-04-24 | 浙江大华技术股份有限公司 | Defense method and equipment for monitoring picture and storage device |
| CN110726407A (en) * | 2019-12-18 | 2020-01-24 | 深圳光启空间技术有限公司 | Positioning monitoring method and device |
| CN111726584B (en) * | 2020-06-24 | 2022-04-29 | 陈侠 | Video monitoring system |
| CN112040196A (en) * | 2020-09-15 | 2020-12-04 | 南京智能仿真技术研究院有限公司 | Indoor intelligent security system based on machine vision |
| CN112486231B (en) * | 2020-12-04 | 2022-03-22 | 山东杨湖酒业有限公司 | Wine cellar monitoring and early warning system |
| CN112804485A (en) * | 2020-12-30 | 2021-05-14 | 济南和普威视光电技术有限公司 | Method and system for monitoring moving target in video and security monitoring method and system |
| CN113538976A (en) * | 2021-07-13 | 2021-10-22 | 中国民用航空飞行学院 | Track invasion detection method based on Mask R-CNN target detection technology |
| CN113743214B (en) * | 2021-08-02 | 2023-12-12 | 国网安徽省电力有限公司超高压分公司 | Intelligent cradle head camera |
| CN115052109B (en) * | 2022-08-16 | 2022-11-29 | 北京理工大学 | Target positioning method and system based on multiple types of cameras |
| CN115223022B (en) * | 2022-09-15 | 2022-12-09 | 平安银行股份有限公司 | Image processing method, device, storage medium and equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201248107Y (en) * | 2008-04-30 | 2009-05-27 | 深圳市飞瑞斯科技有限公司 | Master-slave camera intelligent video monitoring system |
| CN101465033A (en) * | 2008-05-28 | 2009-06-24 | 丁国锋 | Automatic tracking recognition system and method |
| CN101635834A (en) * | 2008-07-21 | 2010-01-27 | 侯荣琴 | Automatic tracing identification system for artificial neural control |
| CN104125433A (en) * | 2014-07-30 | 2014-10-29 | 西安冉科信息技术有限公司 | Moving object video surveillance method based on multi-PTZ (pan-tilt-zoom)-camera linkage structure |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060227237A1 (en) * | 2005-03-31 | 2006-10-12 | International Business Machines Corporation | Video surveillance system and method with combined video and audio recognition |
-
2016
- 2016-06-21 CN CN201610449982.1A patent/CN105894702B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201248107Y (en) * | 2008-04-30 | 2009-05-27 | 深圳市飞瑞斯科技有限公司 | Master-slave camera intelligent video monitoring system |
| CN101465033A (en) * | 2008-05-28 | 2009-06-24 | 丁国锋 | Automatic tracking recognition system and method |
| CN101635834A (en) * | 2008-07-21 | 2010-01-27 | 侯荣琴 | Automatic tracing identification system for artificial neural control |
| CN104125433A (en) * | 2014-07-30 | 2014-10-29 | 西安冉科信息技术有限公司 | Moving object video surveillance method based on multi-PTZ (pan-tilt-zoom)-camera linkage structure |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105894702A (en) | 2016-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105894702B (en) | Intrusion detection alarm system based on multi-camera data fusion and detection method thereof | |
| US11594031B2 (en) | Automatic extraction of secondary video streams | |
| US9805566B2 (en) | Scanning camera-based video surveillance system | |
| EP3648448B1 (en) | Target feature extraction method and device, and application system | |
| CN104754302B (en) | A kind of target detection tracking method based on rifle ball linked system | |
| CN103824070B (en) | A kind of rapid pedestrian detection method based on computer vision | |
| CN111242025B (en) | Real-time action monitoring method based on YOLO | |
| CN109872483B (en) | Intrusion alert photoelectric monitoring system and method | |
| US8848053B2 (en) | Automatic extraction of secondary video streams | |
| CN104883548B (en) | Monitor video face captures processing method and its system | |
| CN101106700A (en) | Intelligent target detail capturing device and method in video monitoring system | |
| CN109887040A (en) | The moving target actively perceive method and system of facing video monitoring | |
| CN113052876B (en) | Video relay tracking method and system based on deep learning | |
| KR100326203B1 (en) | Method and apparatus for face photographing and recognizing by automatic trading a skin color and motion | |
| CN103929592A (en) | All-dimensional intelligent monitoring equipment and method | |
| Doulamis et al. | A real-time single-camera approach for automatic fall detection | |
| CN105931429A (en) | Intelligent nighttime approach recognition and alarming method and device | |
| WO2021248564A1 (en) | Panoramic big data application monitoring and control system | |
| CN113194249A (en) | Moving object real-time tracking system and method based on camera | |
| KR102108391B1 (en) | Moving Object Linkage Tracking System and Method Using Multiple Cameras | |
| Chowdhury et al. | Human surveillance system for security application | |
| CN111985331A (en) | Detection method and device for preventing secret of business from being stolen | |
| Mecocci et al. | Real-time automatic detection of violent-acts by low-level colour visual cues | |
| Tang et al. | Active tracking using color silhouettes for indoor surveillance | |
| Molina-Giraldo et al. | Video segmentation framework by dynamic background modelling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20170614 Address after: 210168 Jiangsu, Nanjing, Pu Pu Road, Pukou District No. 30 Applicant after: Nanjing Tech University Applicant after: Taizhou Yuandong hi tech Automation Engineering Co.,Ltd. Address before: 210168 Jiangsu, Nanjing, Pu Pu Road, Pukou District No. 30 Applicant before: Nanjing Tech University |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180116 Termination date: 20190621 |