KR102008672B1 - System for Performing Linkage Operation of Augmented Reality and Event in Association with Camera and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a system for linking an augmented reality primitive and an event in association with a camera, and a method of driving the system. The system for linking an augmented reality primitive and an event according to an embodiment of the present invention includes an image captured by a camera. The storage unit may store a mark for representing the region of interest of the user as an augmented reality, and the controller may control the display to activate and display the stored mark on the screen when an event occurs in the region of interest.

Description

System for Performing Linkage Operation of Augmented Reality and Event in Association with Camera and Driving Method Thereof}

The present invention relates to a system for linking an augmented reality primitive and an event in conjunction with a camera, and a method of driving the system. More specifically, the monitoring agent is specified in a captured image of a security PTZ (Pan-Tilt-Zoom) camera. The present invention relates to a system capable of quickly recognizing an event of a region and / or a specific region through an augmented reality primitive, and a method of driving the system.

Television systems are classified into closed circuit systems and open circuit systems. The latter aims to convey image information from an unspecified majority, and generally speaking TV broadcasts belong to this. An electronic closed circuit system refers to a system for delivering image information to a specific user for a specific purpose, which is called a closed circuit TV, or CCTV. CCTV is a TV that connects the transmission and reception of images by wire or wirelessly, and cannot be arbitrarily received except for a reception target. CCTV is sometimes called ITV, the abbreviation for industrial TV, but its use is diverse for industrial, educational, medical, traffic control surveillance, disaster prevention, and in-house image information transfer. Recently, the installation of CCTV for crime prevention, especially in the hot spots such as alleys, has been increasing the issue of privacy infringement. Normally such CCTVs for surveillance are monitored by monitoring personnel at control centers.

However, as a conventional camera used for CCTV, a PTZ camera that monitors a 360-degree wide area or a zoom camera that enlarges a specific area at a high magnification is a location and a feature that a user, that is, a surveillance agent is currently looking at. There is a problem that can not be recognized quickly.

In addition, when a specific event, such as an incident, an accident occurs, the monitoring personnel should be able to quickly check the occurrence of the event to perform a quick response, but there is a problem that can not quickly identify such event.

An embodiment of the present invention provides a system and a system in which a monitoring agent can quickly recognize events of a specific region and / or a specific region through an augmented reality primitive in a captured image of a security pan-tilt-zoom (PTZ) camera. The purpose is to provide a driving method.

According to an embodiment of the present invention, a system for linking an augmented reality primitive and an event may include: a storage unit configured to store a primitive for representing an area of interest of a user as an augmented reality in an image captured by a camera, and a region of interest; If an event occurs, the controller includes a control unit for controlling the display to activate and display the stored mark on the screen.

The storage unit may store at least one attribute value among a type, a color, a line thickness, and a name associated with a feature of the region of interest.

The controller may acquire at least one unit image of the captured image of the camera, set the mark on the acquired unit image, and store the set mark on the storage unit.

The controller may be configured to match the coordinate values of the set mark with the coordinate values based on the type of the camera, store the stored values in the storage unit, and display the stored marks retrieved based on the coordinate values provided by the camera. Can be controlled.

The controller may further store the marker having a different size according to the magnification of the camera in the storage unit and control the display to display the stored marker retrieved based on the coordinate value and the magnification provided by the camera. Can be.

The controller may control the display to selectively display the mark based on the magnification of the camera.

The controller may control the display to display the mark when one side length of the mark whose size is adjusted according to the magnification is greater than or equal to a preset length.

The controller may control the display to be activated and displayed on the screen in a manner of blinking the mark, by outputting the content of the event at the position of the mark, or by changing the screen to the stored mark. .

In addition, the driving method of the system for interlocking the event with the augmented reality primitive according to an embodiment of the present invention, the step of storing a marker for representing the user's region of interest in augmented reality in the image captured by the camera, and the region of interest If the event occurs, and activating and displaying the stored mark on the screen.

The storing of the marker may store at least one attribute value among a kind of the marker, a color and a thickness of a line, and a name associated with a feature of the region of interest.

The storing of the mark may include acquiring at least one unit image of the captured image of the camera, setting the mark on the acquired unit image, and storing the set mark. .

The storing of the set mark includes: matching and storing coordinate values of the set mark with coordinate values based on the type of the camera, and activating and displaying the stored mark on the screen. The stored marker may be searched and displayed on the screen based on the coordinate values provided by the camera.

The storing of the set mark may further include storing the mark having a different size according to the magnification of the camera, and activating and displaying the stored mark on the screen may include coordinates provided by the camera. The stored marker may be searched and displayed on the screen based on the value and the magnification.

The activating and displaying the stored mark on the screen may include selectively displaying the mark on the screen based on the magnification of the camera.

In the selectively displaying on the screen, the marker may be displayed on the screen when one side length of the marker whose size is adjusted according to the magnification is greater than or equal to a predetermined length.

The step of activating and displaying the stored marker on the screen is activated on the screen by blinking the marker, outputting the content of the event at the location of the marker, or changing the screen to the stored marker. Can be displayed.

The step of activating and displaying the stored mark on the screen may further include calling up a mark to be displayed at a predetermined position and displaying it on the screen when at least one event of movement and enlargement of the camera occurs. have.

On the other hand, in a computer-readable recording medium including a program for executing a method of driving a system for interlocking an event with an augmented reality primitive according to an embodiment of the present invention, the method of driving a system for interlocking the event with the augmented reality primitive The method may include setting a marker for representing the region of interest of the user as an augmented reality in the image captured by the camera, and activating and displaying the set marker on the screen when an event occurs in the region of interest.

According to an embodiment of the present invention, for example, a monitoring agent may quickly recognize a location and a feature he is looking at in a captured image photographed by a PTZ camera or a zoom camera.

In addition, when an event occurs in a specific area, the event location can be quickly confirmed, so that a faster response process related to an event or an accident can be made.

1 is a view showing a monitoring system according to an embodiment of the present invention,
2A and 2B are views for explaining a monitoring process according to an embodiment of the present invention;
3 is a block diagram showing a detailed structure of the monitoring apparatus of FIG. 1 according to the first embodiment of the present invention;
4 is a block diagram showing a detailed structure of the monitoring apparatus of FIG. 1 according to a second embodiment of the present invention;
5 illustrates a type of primitive according to an embodiment of the present invention;
6 is a view for explaining a method for outputting a primitive on a screen according to an embodiment of the present invention;
7 is a view for explaining the LOD technology according to an embodiment of the present invention,
8 is a flowchart illustrating a monitoring method according to an embodiment of the present invention; and
9 is a flowchart illustrating a process for setting augmented reality primitive according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a monitoring system according to an embodiment of the present invention, Figures 2a and 2b is a view for explaining a monitoring process according to an embodiment of the present invention.

As shown in FIG. 1, the monitoring system 90 according to an embodiment of the present invention includes the camera 100, the communication network 110, and some or all of the monitoring device 120.

Here, the term “comprising some or all” means that some components such as the communication network 110 are omitted and the system 90 is configured so that the camera 100 and the monitoring device 120 perform direct communication or the monitoring device. Some components, such as 120, may be integrated into other components, such as communication network 110, to be a device in the communication network 110, etc., which are all included to help a full understanding of the invention. It explains.

First, the camera 100 may include, for example, a PTZ camera or a zoom camera as a security camera. Here, the PTZ camera is a camera capable of monitoring a wide area by rotating the lens 360 degrees, and a zoom camera corresponds to a camera in which a captured image of a specific area is enlarged at a high magnification. Of course, in the case of a PTZ camera, a zoom function is possible like a zoom camera, and may have a higher performance or a lower performance than a zoom camera. The camera 100 photographs the surrounding area in real time and transmits the captured image to the communication network 110 to provide the captured image to the monitoring device 120. The camera 100 may operate according to the control of the monitoring device 120 to change the shooting area to a specific area (or area), and in the process, for example, when there is a zoom-in command from a monitoring agent, increase the magnification to capture the area. The captured image may be provided to the monitoring device 120.

In addition, if an event occurs while capturing a specific area with the camera 100 and is set to zoom in on the corresponding area from the monitoring device 120, the camera 100 automatically detects the area according to a command of the monitoring device 120. You will be able to zoom in. For example, when an incident, an accident, or a lot of people are analyzed in the area, the camera 100 automatically zooms in and captures the area according to the request of the monitoring device 120. Of course, the operation of the monitoring device 120 detects the occurrence of an event through image analysis and receives an instruction from the monitoring device 120 according to the detection result, but the camera 100 operates itself. Operation may be sufficiently possible and will not be particularly limited to the above. For example, when the traffic of the communication network 110 is increased, since the service may not be easy, the quality of the service may be further improved by allowing the camera 100 to perform the above operation by itself. Alternatively, the camera 100 may determine whether to perform the zoom-in operation according to the traffic state, and according to the determination result, the monitoring apparatus 120 may analyze the image and automatically perform the zoom-in operation according to the analysis result. There will be.

For example, suppose the event situation is a fire, a crowd of people, or a car crash. If the situation is determined as a result of the image analysis, the monitoring apparatus 120 may request the camera 100 to perform zoom-in photographing for a predetermined specific region. In this process, it may be possible to change the zoom-in magnification for each event situation. In addition, such a specific event is associated with (eg, matching) a marker (hereinafter, referred to as a primitive) for implementing augmented reality, so that the corresponding primitive is displayed on the screen. In other words, since the primitives displayed on the screen may not be the same for the particular region of interest or observation, the primitives displayed on the screen may be different depending on the type of such event (ex. Size, resolution, etc.) Here, the "primitive" refers to a mark made up of points, lines, shapes, and combinations thereof in order to inform a user of a specific region in the image reflected on the screen as augmented reality. The dictionary meaning of primitives is "an element necessary to create graphic designs such as lines, circles, curves and polygons that can be drawn, stored and manipulated as individual entities by computer programs in computer graphics, also called geometric primitives." Is defined.

Furthermore, the camera 100 may provide a coordinate value together with the captured image in the case of the PTZ camera so that the monitoring apparatus 120 may determine whether the captured image is an image captured in the posture of the camera 100. . Then, the monitoring device 120 may implement augmented reality by searching for previously stored primitives based on the received coordinate values. More details on this will be discussed later.

The communication network 110 includes both wired and wireless communication networks. For example, a wired or wireless Internet network may be used or interworked as the communication network 110. The wired network includes an internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), Wibro network, and the like. It is meant to include. Of course, the communication network 110 according to an embodiment of the present invention is not limited thereto, and may be used as a connection network of a next generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment. For example, when the communication network 110 is a wired communication network, the access point in the communication network 110 may be connected to an exchange office of a telephone company, but in the case of a wireless communication network, data may be connected to SGSN or Gateway GPRS Support Node (GGSN) operated by a communication company. Or, it can be connected to various repeaters such as BTS (Base Station Transmission), NodeB, e-NodeB and process data.

The communication network 110 may include an access point. Access points include small base stations, such as femto or pico base stations, which are frequently installed in buildings. Here, the femto or pico base station is classified according to how many cameras 100 can be connected according to the classification of the small base station. Of course, the access point includes a near field communication module for performing near field communication such as the camera 100 and Zigbee and Wi-Fi. The access point may use TCP / IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, the short-range communication may be performed in various standards such as Bluetooth, Zigbee, Infrared (IrDA), Ultra High Frequency (UHF) and Very High Frequency (VHF) and Ultra Wideband Communication (UWB). Can be. This allows the access point to extract the location of the data packet, specify the best communication path for the extracted location, and forward the data packet to the next device, such as the monitoring device 120, along the designated communication path. An access point may share several circuits in a general network environment, and may include, for example, a router, a repeater, and a repeater.

A monitoring device (or a system that links events with augmented reality primitives in conjunction with a camera) 120 may be provided, for example, in a control center. According to an embodiment of the present invention, the monitoring device 120 is preferably used for security. For example, the monitoring personnel performing administrative tasks may be replaced periodically, and since the monitoring personnel may not be able to know the geography of a specific region in detail, the apparatus and method according to the embodiment of the present invention may be such. This can be very useful in situations. The monitoring device 120 may further include a server and a computer connected to the server. Of course, since the computer can also be used as a server, it will not be limited to this. The monitoring device 120 may pre-store map data for a specific area within the photographing range of the camera 100. The map data may be provided through a separate path, but only the image photographed and provided by the camera 100 may be used as the map data. Here, the "separate path" means provided through a server of an administrative agency that provides map data or through a recording medium such as a CD.

As such, when the map data of the specific region is secured, the monitoring device 120 performs a process of setting an augmented reality primitive for a specific region of interest. This may be regarded as a construction process of the DB 120a. For example, the monitoring device 120 sets a primitive for displaying together with augmented reality on a map of a specific region as shown in FIGS. 2A and 2B and stores the primitives in the DB 120a. In addition, when a specific event occurs in the area A as shown in FIG. 2A, the monitoring device 120 enlarges the screen (at a preset magnification) and shows it to the user as shown in FIG. 2B. That is, the screen is changed or enlarged in one area of the screen. Here, the augmented reality primitive setting process will be described in detail later, so that the user can easily recognize the region and the feature by assigning attribute values such as name, color, and line thickness to the primitives such as points, lines, and shapes. It is a process. Primitives set by this process will be displayed on the screen in augmented reality.

For example, the monitoring device 120 compares the captured image received through the camera 100 with map data stored in the DB 120a or coordinate values (eg, PTZ cameras) of a specific area received through the camera 100. The primitive is searched in the DB 120a based on the coordinate value of and is synthesized on the received captured image and displayed on the screen. At this time, the monitoring device 120 determines what state or magnification of the magnification of the camera 100 is in an event state, finds a corresponding primitive, synthesizes the image, and displays it on the screen. To this end, the DB 120a stores the primitives according to the event situation or the magnification.

For example, the monitoring device 120 may determine that an event has occurred through image analysis in the image photographed by the camera 100, or, if there is a separate request from a monitoring agent, enlarge a specific region, for example, to display a screen as illustrated in FIG. 2B. In FIG. 2A, a specific primitive may be activated and displayed on the screen. Here, "activation" means that the augmented reality primitive blinks in the image, outputs the event contents as text at the augmented reality primitive position, or controls (screen) movement (or change) to the augmented reality primitive position. it means. If the image shown in FIG. 2A is a PTZ camera image at 1 magnification, the image shown in FIG. 2B moves the pan and tilt positions of the PTZ camera such that the region A shown in FIG. 2A is at the center of the screen. 10 times after zooming in. The augmented reality primitive may be moved and scaled to match the features of the image even if the image is moved, enlarged or reduced by controlling the PTZ camera by the operator.

As such, the monitoring device 120 after the construction process of the DB (120a) setting the augmented reality primitive according to an embodiment of the present invention, the image (or image and coordinate values) input from the camera 100, and its Augmented reality can be implemented in various forms depending on whether an event occurs in the process, and also what event occurs.

As a result of the above configuration, users such as monitoring personnel can quickly recognize the location and feature that they are currently looking at, and if an event occurs in a specific area, the user can quickly check the event occurrence location and take a quick response. Will be.

3 is a block diagram showing a detailed structure of the monitoring apparatus of FIG. 1 according to the first embodiment of the present invention.

As shown in FIG. 3, the monitoring device 120 according to the first embodiment of the present invention includes some or all of the communication interface unit 300 and the control unit 310, and includes “some or all of them”. That is the same as the previous meaning.

The communication interface unit 300 communicates with the communication network 110 to control the camera 100 of FIG. 1. The communication interface unit 300 may transmit an image captured by the camera 100 to the controller 310, and transmit control data for controlling the camera 100 to the camera 100 under the control of the controller 310. . Here, the control data may include, for example, coordinate values (θp, θ _T , θz) for controlling the PTZ camera.

In addition, the communication interface 300 may be linked to the display unit or the user interface unit. For example, when the display unit is interlocked with the communication interface unit 300, various operations for implementing augmented reality of the primitive may be performed according to a command instructed by a user such as a monitoring agent. In the process, for example, if voice recognition can be used, the user interface may include a voice collector, for example a microphone. Since the event by the monitoring personnel, that is, the command to enlarge the specific area can be made by voice any number, the embodiment of the present invention will not be specifically limited to any one way.

The controller 310 controls the overall operation of the monitoring device 120. To this end, the controller 310 may include, for example, a CPU 311 and an augmented reality primitive execution unit 313. Here, the CPU 311 may include a control circuit, an operation unit (ALU), a register, and the like, and the augmented reality primitive execution unit 313 may include a program for implementing augmented reality of the primitive. In this regard, the augmented reality primitive execution unit 313 may include a memory, and the memory may include a ROM (ex. EEPROM), a RAM, and the like. Here, EEPROM may be very useful because it is convenient to write or delete programs.

The controller 310 may operate the augmented reality primitive execution unit 313 to perform various operations for implementing augmented reality of the primitive. For example, an operation for setting (or specifying) a primitive may be performed, and the set primitive may be shown to the user in various ways according to an event situation. For example, even if the primitive is set to 1x, if the user wants 5x, the primitive set to 5x can be synthesized into the 5x image, but the primitive set to 1x can be simultaneously converted to bit to fit the 5x image. Synthesis is possible as much as possible, and embodiments of the present invention will not be specifically limited to any one manner. Other details will be described with reference to FIG. 4.

4 is a block diagram showing a detailed structure of the monitoring apparatus of FIG. 1 according to the second embodiment of the present invention.

As shown in FIG. 4, the monitoring device 120 ′ according to the second embodiment of the present invention includes a part or all of the communication interface 400, the controller 410, and the augmented reality primitive execution unit 420. do.

Here, "comprises some or all" means that some components such as the augmented reality primitive execution unit 420 are integrated with the control unit 410 or various components constituting the augmented reality primitive execution unit 420. These means that at least one can be omitted or integrated with each other, etc., will be described as including all to aid a sufficient understanding of the invention.

The communication interface 400 may include an image receiver 401 and a coordinate receiver 403. The image receiver 401 may demodulate the received image, separate the demodulated image, and decode the separated image (eg, a video image) under the control of the controller. In addition, when the camera 100 of FIG. 1 is a PTZ camera, the coordinate receiver 403 may receive the coordinate values θp and θ _T and provide them to the image synthesizer 424. The coordinate value may be usefully used to determine which region the currently received captured image corresponds to. Of course, such coordinate values are received together with the image to the image receiving unit 401 and separated by the demux, and the separated coordinate values may be provided to the image synthesizing unit 424 by the control unit. It will not be specifically limited to the structure of. In addition, the coordinate receiver 403 may further perform a coordinate transformation operation. For example, since the coordinate value at which the primitive is set is a planar coordinate (x, y), the pan and tilt coordinate values (θp, θ _T ) are converted into the planar coordinates (x, y), and the image synthesizer 424 is used. May be provided.

 The controller (not shown) may be referred to as a first controller according to an exemplary embodiment of the present invention as a screen controller, and performs an overall control operation of the monitoring device 120 ′. In other words, when a user wants to perform a primitive setting operation, the augmented reality primitive execution unit 420 may be operated. When an image is received from the outside based on the set primitive, the set primitive may be synthesized and displayed to the user. have. In this case, the primitive may perform various interlocking operations such that the matched primitive is searched in the DB 120a and the retrieved primitive is synthesized in the image, or the synthesized primitive blinks according to the event situation.

The controller 410 may be referred to as a second controller, for example, a PTZ camera controller, and may control the PTZ camera in one of various ways. For example, the camera controller 410 may perform a method of controlling the camera by transferring the direction and speed values to which the PTZ camera should be moved, that is, the PTZ camera. Alternatively, the camera controller 410 may transmit a absolute coordinate value (P, T, Z) or a relative position (Dp, Dt, Dz) from the current position to the PTZ camera to control the camera. ) Can also store the P, T, and Z positions in the PTZ camera itself in a predetermined integer index and move them to the index value.

The augmented reality primitive execution unit 420 is not significantly different from the augmented reality primitive execution unit 313 of FIG. 3. However, when compared with the augmented reality primitive executor 313 of FIG. 3, the augmented reality primitive executor 420 of FIG. 4 is not one software but at least one software module may be implemented in hardware. it means.

The image analyzer 421 receives the image received by the image receiver 401 and analyzes the received image. Such image analysis is performed to determine whether an event occurs in the region photographed by the camera 100 of FIG. 1 based on the received image. For example, there may be a fire, a car crash, illegal parking, or a lot of people. The analysis result may be provided to the event interworking processor 423.

The event receiver 422 receives a user command by a user interface. For example, if a user generates an event through a button or a microphone, for example, if the user requests to enlarge and display a specific area, an event signal for this is provided to the event interworking processor 423.

Then, the event interworking processor 423 determines whether there is an event signal received from the image analyzer 421 and the event receiver 422, and processes the received event signal to provide the image synthesizer 424. . Here, "processing" means to provide both if there is a signal and to deliver only one if there is only one signal.

The image synthesizing unit 424 searches for the pre-stored primitives by matching the currently received image, synthesizes the retrieved primitives with the received image, and provides them to the image display unit 425. For example, suppose that the event received by the event receiver 422 is associated with a specific primitive, and the primitive is an indicator expressed at an appropriate size at a zoom 10x. Then, the image synthesizing unit 424 displays the primitive storage unit 427 for displaying a specific area at a 10x magnification based on the coordinate values received from the coordinate receiving unit 403 and the event information received from the event interworking processor 423. ) Can be output by combining the searched primitives with a 10x image.

The image display unit 425 may include a display unit or operate in conjunction with the display unit. For example, when outputting the 10x magnification image received by the image synthesizing unit 424 to the image display unit 425, it may serve to convert the resolution according to the resolution of the display unit and output the same. In this process, the bit conversion of the primitive may be performed according to the magnification.

The primitive setting unit 426 may perform an operation such as setting, changing, or deleting a primitive based on map data stored in the primitive storage unit 427. Of course, since the primitive setting unit 426 can set the primitive by using the image received in real time, it will not be limited to any one method in the embodiment of the present invention. For example, the user may magnify a specific area by 10 times in the image received through the camera 100 and may capture an image of the corresponding area. The captured still image is stored together with the coordinate values of the camera 100. Thereafter, the stored still image may be called to set a primitive, and the set primitive may be matched with a coordinate value and stored. The above map data may be at least one unit image selected from among real-time unit images photographed and provided by the camera 100. The controller 410 may include, for example, a primitive storage unit 427 or a separate unit whenever there is a scene change. If the unit image is stored in the memory, the primitive setting unit 426 may call up the stored unit image to perform the primitive setting operation.

The primitive storage unit 427 matches and stores the primitive set by the user, coordinate values of the region or place where the primitive is set, and various information such as attributes of the primitive. For example, even in the same region, stored primitives may be stored in various forms, such as storing primitives with different sizes by magnification. In this case, the coordinate values of the stored primitives are preferably stored as coordinate values on a spherical coordinate system. Accordingly, the primitive storage unit 427 can output a primitive matching the received coordinate value.

5 is a diagram illustrating the types of primitives according to an embodiment of the present invention.

As shown in (a) to (c) of Figure 5, the primitive according to an embodiment of the present invention, for example, to display a specific point, boundary line, specific area, etc. in the PTZ camera image Include geometric primitives.

The 'point' shown in (a) of FIG. 5 is added or set in such a manner as to specify the (x, y) position on the PTZ screen, and in actual storage, the 'point' is a (P, T) value on the spherical coordinate system corresponding to the point position. Is converted and stored.

A line, such as a polyline, shown in FIG. 5B is added in a manner of designating n point (xi, yi) positions on a PTZ screen, and in actual storage, (Pi) on a spherical coordinate system corresponding to a point position. , Ti) value is converted and stored.

In addition, the 'shape' shown in FIG. 5 (c), for example, a polygon is added in a manner of designating n points (xi, yi) positions on a PTZ screen, and when actually stored, (Pi, Ti) value is converted and stored.

6 is a diagram for describing a method of outputting a primitive on a screen according to an exemplary embodiment of the present invention.

For convenience of description, referring to FIG. 6 together with FIG. 4, for example, the coordinate values of Pan, Tilt and Zoom obtained in real time by the (PTZ) coordinate receiver 403 of FIG. CT, CZ). This coordinate value is always updated in real time. Each vertex of the augmented reality primitive is stored as a coordinate value (Pi, Ti) on the spherical coordinate system.

Therefore, in FIG. 6, the f value is a focal length and must be measured in advance. The unit of the f value is the number of pixels. The process of projecting the vertex values represented in the spherical coordinate system onto the image plane at the current pan, tilt and zoom positions is well illustrated in FIG. 6. 6, reference numeral 600 denotes a projection plane when the PTZ coordinate values are pan: CP, tilt: CT, and zoom: CZ magnification, and reference numeral 610 denotes a PTZ coordinate value of pan: 0, tilt: 0, Zoom: The projection plane in the case of 1 magnification is shown.

Meanwhile, the equation for converting the vertices Pi and Ti constituting the augmented reality primitive into a direction vector in the camera coordinate system is shown in Equation 1.

The matrix for converting the coordinate system to the pan and tilt positions corresponding to the current coordinates is shown in Equation 2.

The formula for converting the direction vector Vi of the vertices constituting the augmented reality primitive into the direction vector Vi 'on the coordinate system to the current pan and tilt positions is shown in Equation 3 below.

Since the projection position should be reflected according to the zoom value of the current camera, if the focal length measured at 1 magnification is f, the focal length of the plane to be actually projected should reflect the current zoom value, so fc = f × CZ.

The formula for projecting the vertex Vi 'constituting the augmented reality primitive on the PTZ screen by reflecting this focal length on the screen is shown in Equation 4.

Finally, the Xi points projected on the image plane on the current pan and tilt positions are drawn and drawn in the form of polylines or polygons. If there is only one vertex, it is drawn as a point.

7 is a view for explaining the LOD technology according to an embodiment of the present invention.

As shown in FIG. 7, augmented reality primitives according to embodiments of the present invention are not always displayed in the range of all magnifications. At one magnification, features that are too small to be identified are often visible at higher magnifications. Therefore, the augmented reality primitive object is also determined whether the screen output according to the magnification.

As can be seen in Figure 7 (a), if the magnification 1 and 4 magnification hardly appear on the screen in the size that the operator can not recognize, as shown in Figure 7 (b) 8 and 16 times magnification Since the size is recognizable by the operator, it may be displayed on the screen. In FIG. 7, reference numeral 700 denotes an augmented reality primitive (ex. Polygon type), and reference numeral 710 denotes an augmented reality primitive bounding box.

An augmented reality primitive is projected onto an image and is only displayed if the longest length of the bounding box of the projected primitive is greater than or equal to a certain length. Of course, the bounding box may be displayed on the screen and may not be displayed (recognized only by the system and on the screen), but at least one length of the object displayed on the screen may be determined by applying bounding boxes according to different magnifications. You can then decide whether to display the object based on the result.

In the embodiment of the present invention, by applying the LOD (Level of Detail) technology, more information can be represented on the monitoring screen.

8 is a flowchart illustrating a monitoring method according to an embodiment of the present invention.

For convenience of description, referring to FIG. 8 together with FIG. 1, the monitoring apparatus 120 according to an embodiment of the present invention stores (or augmented reality primitives for indicating to a region of interest of a user in an image captured by the camera 100). Setting) (S800). In order to set an augmented reality primitive, for example, the monitoring apparatus 120 acquires an image of an operator's region of interest by changing a photographing region of the camera 100. For example, even if a unit image of 60 frames is acquired in one second, since the same region will be continuously photographed, the primitive for augmented reality may be set with the first frame of image.

Thereafter, when an event occurs in the ROI of the user, the monitoring apparatus 120 activates and displays the stored augmented reality primitive on the screen (S810). Here, the event may mean a case in which a specific event or an accident occurs that is requested by an operator of the monitoring device 120 or preset by a system designer in a corresponding region of interest. This can be seen through image analysis, as mentioned above, and if an event occurs, magnify the area of interest to allow the shot to be taken, flickering augmented reality primitives within the image, or texting the augmented reality primitive position. You can activate primitives on the screen in the same way that you output the event content.

4, the events generated by the image analyzer 421 or the event receiver 422 are interlocked with the augmented reality primitive in the event interworking processor 423. It should be set in conjunction. Accordingly, when the event occurs, the monitoring device 120 displays the augmented reality primitive associated with the generated event in the memory or the DB 120a to indicate that the event has occurred in the augmented reality primitive.

9 is a flowchart illustrating a process for setting augmented reality primitive according to an embodiment of the present invention.

For convenience of description, referring to FIG. 9 together with FIG. 1, the monitoring apparatus 120 according to an exemplary embodiment of the present invention acquires an image by moving a PTZ camera to a region of interest as the camera 100 (S900). For example, the lens unit of the PTZ camera may be moved left or right or moved to acquire an image of a region of interest. During PTZ control, the current PTZ coordinates are updated in real time.

Next, the monitoring device 120 adds an augmented reality primitive on the current image coordinate system (S910). For example, the image of the region of interest may be one unit image acquired from images received in real time through the camera 100, and an augmented reality primitive added to the unit image is set as a coordinate value on a plane coordinate system. Can be.

Thereafter, the monitoring device 120 edits or sets an attribute value together with the augmented reality primitive (S920). Attribute values can include ① type of primitive, ② name, ③ color, thickness of dot or line. Here, the type of primitive includes points, lines, and figures, and the name includes a name of a region of interest or an object, a distance, or an altitude indication.

When the editing of the attribute value is completed, the monitoring apparatus 120 converts and stores the augmented reality primitive vertices (x, y) into pan, tilt (Pi, Ti) coordinates (S930 and S940).

After the storage is completed as described above, the monitoring device 120 may search for and display the augmented reality primitives stored in advance on the screen based on the pan and tilt coordinate values provided from the camera 100.

On the other hand, even if all the components constituting the embodiment of the present invention is described as being combined or operated in combination, the present invention is not necessarily limited to these embodiments. In other words, within the scope of the present invention, all of the components may be selectively operated in combination with one or more. In addition, although all of the components may be implemented in one independent hardware, each or some of the components of the components are selectively combined to perform some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art. Such a computer program may be stored in a computer-readable non-transitory computer readable media and read and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium storing data for a short time such as a register, a cache, or a memory. . Specifically, the above-described programs may be stored and provided in a non-transitory readable recording medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.

While the above has been shown and described with respect to preferred embodiments of the invention, the invention is not limited to the specific embodiments described above, it is usually in the art to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

100: camera 110: communication network
120: monitoring device 120a: DB
300, 400: communication interface 310, 410: control unit
311: CPU 313, 420: augmented reality primitive execution unit
401: image receiving unit 403: coordinate receiving unit
421: Image analyzer 422: Event receiver
423: event interworking unit 424: video synthesizing unit
425: Image display unit 426: Primitive setting unit
427: primitive storage

Claims (17)

Storing a primitive for representing an area of interest of the user as augmented reality in an image captured by a camera; And
When an event occurs in the region of interest, activating and displaying the stored mark on the screen; including,
The step of storing the marker,
Acquiring at least one unit image of the captured image of the camera;
Setting the marker on the acquired unit image; And
And storing the set mark.
The storing of the set mark,
And matching and storing the set mark with coordinate values related to the type of the camera.
Activating and displaying the stored mark on the screen,
The stored marker is searched and displayed on the screen based on the coordinate values provided by the camera,
The storing of the set mark,
Storing the marker of a different size according to the magnification of the camera;
Activating and displaying the stored mark on the screen,
The stored marker is searched and displayed on the screen based on the coordinate value and the magnification provided by the camera,
Activating and displaying the stored mark on the screen,
Selectively displaying the mark on the screen based on the magnification of the camera;
The selectively displaying on the screen,
The length of one side of the mark is adjusted according to the magnification, by applying an augmented reality primitive bounding box according to different magnification And a method of driving an event with an augmented reality primitive that displays the mark on the screen when the length is greater than or equal to a predetermined length. The method of claim 1,
The step of storing the marker,
And a method of linking an event with an augmented reality primitive, which stores at least one attribute value among a kind, a color, a line thickness, and a name associated with a feature of the region of interest. delete delete delete delete delete The method of claim 1,
Activating and displaying the stored mark on the screen,
A system for interlocking an event with an augmented reality primitive that activates and displays the screen in a manner of blinking the marker, outputting the content of the event at the position of the marker, or changing the screen to the stored marker. Way. The method of claim 1,
Activating and displaying the stored mark on the screen,
When the at least one event of the camera movement and enlargement occurs, calling up the marker to be displayed at a predetermined position and displaying on the screen drive of the system integrating the event with the augmented reality primitive, characterized in that Way. A storage unit storing a primitive for representing an area of interest of the user as an augmented reality in an image captured by a camera; And
And a controller configured to control a display to activate and display the stored mark on a screen when an event occurs in the region of interest.
The control unit,
Acquiring at least one unit image of the photographed image of the camera to set the mark on the acquired unit image, and storing the set mark on the storage unit;
The control unit,
Matching the coordinate value of the set mark with a coordinate value associated with the type of the camera to store in the storage, and controlling the display to display the stored mark retrieved based on the coordinate value provided by the camera,
The control unit,
Further storing the mark of different size according to the magnification of the camera in the storage unit, controlling the display to display the stored mark retrieved based on the coordinate value provided by the camera and the magnification,
The control unit,
Control the display to selectively display the marker based on the magnification of the camera,
The control unit,
The length of one side of the mark is adjusted according to the magnification, by applying an augmented reality primitive bounding box according to different magnification A system for linking an event with an augmented reality primitive that controls the display to display the marker when more than a predetermined length. The method of claim 10,
The storage unit, the augmented reality primitive system for storing an attribute value of at least one of the type of the marker, the color and the thickness of the line, the name associated with the feature of the region of interest. delete delete delete delete delete The method of claim 10,
The control unit, the augmented reality to control the display to display on the screen in a way of blinking the marker, the method of outputting the content of the event at the location of the marker or the method of changing the screen to the stored marker. A system that links primitives and events.

Images