KR102502300B1 - Fusion method and system using radar and camera - Google Patents

Abstract

A radar and camera fusion method is disclosed. The radar and camera fusion method includes a radar sensor, a camera module, and a controller. The radar and camera fusion method includes the controller receiving image frames captured at different times from the camera module, the controller comparing the image frames with each other to identify a plurality of objects in the image frames, Deriving, by the controller, a relationship between the coordinates of the radar sensor and the coordinates of the camera module by applying the coordinates of the radar sensor to a neural network; Matching moving objects in radar data generated from the radar sensor with objects of the controller, calculating the energy of any one object among the identified plurality of objects, the controller determines that the calculated energy is Determining whether the calculated energy is greater than the arbitrary threshold, and when the calculated energy is greater than the arbitrary threshold, the controller sending a message to the server to pay attention to the manager and look at the image frames displayed on the display; , The step of matching, by the controller, moving objects in the radar data generated from the radar sensor with the plurality of objects identified in the image frames using the derived relationship, wherein the controller determines the plurality of motions in the image frames. Generating vectors, identifying, by the controller, the plurality of objects in the image frames using the generated plurality of motion vectors, wherein the controller identifies the plurality of motion vectors using the identified plurality of objects. selecting representative motion vectors from among the plurality of motion vectors, the controller calculating magnitudes and directions of the selected representative motion vectors, the controller calculating Doppler velocities of the plurality of objects recognized from radar data generated from the radar sensor. and extracting directions, wherein the controller compares magnitudes of the calculated representative motion vectors with the Doppler velocities, and compares the calculated directions of the representative motion vectors with the plurality of Comparing directions of objects, and matching, by the controller, the identified plurality of objects in image frames with the plurality of objects recognized in the radar data according to the comparison.

Description

Radar and camera fusion method and system {Fusion method and system using radar and camera}

Embodiments according to the concept of the present invention relate to a radar and camera fusion method and system, and in particular, to a radar and camera fusion method and system capable of monitoring the surroundings using a radar together with a camera.

A method of monitoring the surroundings using a camera such as a CCTV has been used. In order to find out an abnormal situation or an emergency situation, security personnel must continuously monitor the video captured by the camera. However, it is not easy for security personnel to continuously monitor the video captured by the camera. In addition, when there are many objects to be monitored in the images captured by the camera, it is difficult for security personnel to monitor the many objects to be monitored one by one. In addition, when the resolution of the camera is not good or the weather is bad, it is not easy to identify a monitoring target from an image captured by the camera.

Korean Registered Patent Publication No. 10-1854461 (2018.04.26.)

A technical problem to be achieved by the present invention is to provide a radar and camera convergence method and system capable of monitoring the surroundings using a camera and a radar together in order to compensate for the disadvantages of monitoring the surroundings using only a camera.

A radar and camera fusion method of a video surveillance system including a radar sensor, a camera module, and a controller according to an embodiment of the present invention includes the steps of receiving, by the controller, image frames captured at different times from the camera module; A controller compares the image frames with each other and identifies a plurality of objects in the image frames, wherein the controller applies the coordinates of the radar sensor to a neural network to determine a relationship between the coordinates of the radar sensor and the coordinates of the camera module. deriving, and matching, by the controller, moving objects from radar data generated from the radar sensor with a plurality of objects identified from the image frames using the derived relationship.

An image monitoring method using a radar and a camera of an image monitoring system including a radar sensor, a camera module, and a controller according to an embodiment of the present invention includes receiving image frames captured at different times from the camera module by the controller. The controller generates a plurality of motion vectors in the image frames, the controller identifies a plurality of objects in the image frames using the generated plurality of motion vectors, the controller identifies a plurality of objects selecting representative motion vectors from among the plurality of motion vectors by using a plurality of objects, the controller calculating magnitudes of the selected representative motion vectors, the controller recognizing from radar data generated from the radar sensor extracting Doppler velocities and directions of the plurality of objects, the controller comparing magnitudes of the calculated representative motion vectors with the Doppler velocities, and comparing the calculated directions of the representative motion vectors with the radar data. Comparing directions of the plurality of recognized objects, and matching, by the controller, the plurality of objects identified from the image frames and the plurality of objects recognized from the radar data according to the comparison. .

The video monitoring method using the radar and the camera includes the steps of: calculating, by the controller, the energy of one of the identified plurality of objects; determining, by the controller, whether the calculated energy is greater than a certain threshold value; and When the calculated energy is greater than the certain threshold, the controller sending a message to a server.

The step of calculating, by the controller, the energy of any one object among the identified plurality of objects, the controller identifying the Doppler velocity of the object recognized from the radar data corresponding to the one object, and the controller and calculating a motion change rate of the one object by dividing the identified Doppler speed by a preset Doppler speed.

The step of calculating, by the controller, the energy of any one object among the identified plurality of objects, wherein the controller counts pixel values of any one object in any one of the image frames to determine the number of the one object. Calculating the size, the controller calculating a difference between the size of any one object and the average object size, the controller converting the difference between the size of any one object and the average object size into the average object size and calculating, by the controller, the motion change rate of the one object and the size suitability of the one object as the energy of the one object.

An image surveillance system using a radar and a camera according to an embodiment of the present invention includes a camera.

The camera includes a radar sensor for generating radar data by transmitting a radar signal, a camera module for generating image frames by capturing images at different times, receiving the image frames, and generating a plurality of motion vectors from the image frames; A plurality of objects are identified in the image frames using the generated plurality of motion vectors, representative motion vectors are selected from among the plurality of motion vectors using the identified plurality of objects, and the selected representative motion vector is selected. Calculate magnitudes and directions of , extract Doppler velocities and directions of the plurality of objects recognized from radar data generated from the radar sensor, compare magnitudes of the calculated representative motion vectors with the Doppler velocities, , Comparing directions of the calculated representative motion vectors with directions of the plurality of objects recognized from the radar data, and according to the comparison, the plurality of objects identified from the image frames and the directions recognized from the radar data It includes a controller that matches multiple objects.

A radar and camera fusion system according to an embodiment of the present invention includes a camera. The camera receives a radar sensor that generates radar data by transmitting a radar signal, a camera module that generates image frames by capturing images at different times, and receives the image frames, compares the image frames with each other, It identifies a plurality of objects, derives a relationship between the coordinates of the radar sensor and the coordinates of the camera module by applying the coordinates of the radar sensor to a neural network, and derives a relationship between the coordinates of the radar sensor and the coordinates of the camera module using the derived relationship. and a controller matching moving objects in the objects of and radar data generated from the radar sensor.

The radar and camera convergence method and system according to an embodiment of the present invention has an effect of efficiently monitoring images by identifying a monitoring target using a radar and notifying a manager when an abnormal situation of the monitoring target is predicted. there is.

A detailed description of each drawing is provided in order to more fully understand the drawings cited in the detailed description of the present invention.
1 shows a block diagram of an image surveillance system using a radar and a camera according to an embodiment of the present invention.
FIG. 2 shows images of video frames for explaining a selection operation of representative motion vectors performed by the controller shown in FIG. 1 .
FIG. 3 shows radar data for explaining an operation of extracting Doppler velocities and directions performed by the controller shown in FIG. 1 .
Figure 4 shows a flow chart for explaining the operation of the controller shown in Figure 1;
FIG. 5 is a diagram of a neural network for deriving a relationship between coordinates of a radar sensor and coordinates of a camera performed by the controller shown in FIG. 1 .
FIG. 6 is a flowchart for explaining an operation of the controller shown in FIG. 1 according to another embodiment.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are merely exemplified for the purpose of explaining the embodiment according to the concept of the present invention, and the embodiments according to the concept of the present invention may take various forms. It can be implemented as and is not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can be applied with various changes and can have various forms, so the embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, e.g., without departing from the scope of rights according to the concept of the present invention, a first component may be termed a second component and similarly a second component may be referred to as a second component. A component may also be referred to as a first component.

It should be understood that when a component is referred to as being "connected" or "connected" to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. will be. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle. Other expressions describing the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in this specification, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined herein, they are not interpreted in an ideal or excessively formal meaning. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a block diagram of an image surveillance system using a radar and a camera according to an embodiment of the present invention.

Referring to FIG. 1 , an image surveillance system 100 using a radar and a camera means a system used to detect objects such as people in various places such as parks, factories, hospitals, and ports. According to an embodiment, the video monitoring system 100 may be referred to as a radar and camera convergence system. The radar and the camera mean a radar sensor 11 and a camera module 13. The video monitoring system 100 includes a camera 10 .

The camera 10 may be installed outside the building 9 . Depending on the embodiment, the camera 10 may be installed on a road or other external environment. The camera 10 includes a radar sensor 11 , a camera module 13 , and a controller 15 . The radar sensor 11, the camera module 13, and the controller 15 are implemented as one device. According to embodiments, the radar sensor 11, the camera module 13, and the controller 15 may be implemented in different devices. Also, the radar sensor 11 and the camera module 13 are implemented as one device, but the controller 15 may be implemented as another device. The controller 15 is used to control the radar sensor 11 and the camera module 13 . The controller 15 may include a processor (not shown). The operation of the controller 15 is driven by instructions, which are executed by the processor.

The camera module 13 photographs the surroundings. That is, the camera module 13 generates image frames VFS captured at different times. The controller 15 receives the image frames VFS generated by the camera module 13 . The controller 15 receives radar data RD generated by the radar sensor 11 . The controller 15 receives the image frames VFS received from the camera module 13 and the radar radar RD received from the radar sensor 11, and converts the image frames VFS and radar data RD. Perform operations to match each other.

FIG. 2 shows images of video frames for explaining a selection operation of representative motion vectors performed by the controller shown in FIG. 1 .

Referring to FIGS. 1 and 2 , the controller 15 generates a plurality of motion vectors MVS1 to MVS3 in image frames FR1 and FR2. When the first image frame FR1 is an image frame captured at the first time t0, the second image frame FR2 is an image captured at the second time t1, which is later than the first time t0. It is a frame. The image frames FR1 and FR2 are divided into microblocks MB.

The controller 15 compares two image frames FR1 and FR2 generated at different times to generate a plurality of motion vectors MVS1 to MVS3. The controller 15 identifies a plurality of objects 21, 23, and 25 in the image frames FR1 and FR2 by using the plurality of motion vectors MVS1 to MVS3. The plurality of objects 21 , 23 , and 25 mean the three people shown in FIG. 2 . Identifying the plurality of objects 21, 23, and 25 in the image frames FR1 and FR2 using the plurality of motion vectors MVS1 to MVS3 means that the directions of the motion vectors MVS1 to MVS3 This means that a plurality of objects 21, 23, and 25 are detected by analysis. For example, in the first image frame FR1, the first object 21 is located in the upper left corner in the first image frame FR1, but is located in the lower left corner in the second image frame FR2. That is, the first object 21 moved from the top of the first image frame FR1 to the bottom of the second image frame FR2. The controller 15 analyzes the directions of the motion vectors MVS1 and identifies the motion vectors MVS1 as motion vectors MVS1 of the first object 21 . That is, the controller 15 identifies the first object 21 by analyzing the directions of the motion vectors MVS1. In the first image frame FR1, the second object 23 is located at the bottom of the center in the first image frame FR1, but is located at a slightly raised portion from the bottom of the center in the second image frame FR2. That is, the second object 23 moved from the bottom of the center of the first image frame FR1 to a slightly raised portion from the center of the second image frame FR2. The controller 15 analyzes the directions of the motion vectors MVS2 and identifies the motion vectors MVS2 as motion vectors MVS2 of the second object 23 . That is, the controller 15 identifies the second object 23 by analyzing the directions of the motion vectors MVS2. In the first image frame FR1, the third object 25 is located at the rightmost bottom in the first image frame FR1, but is located at the rightmost top in the second image frame FR2. That is, the third object 25 moves from the bottom right of the first image frame FR1 to the top right of the second image frame FR2. The controller 15 analyzes the directions of the motion vectors MVS3 and identifies the motion vectors MVS3 as motion vectors MVS3 of the third object 25 . That is, the controller 15 identifies the third object 25 by analyzing the directions of the motion vectors MVS3.

According to an embodiment, the controller 15 may identify the plurality of objects 21 , 23 , and 25 by analyzing not only the plurality of motion vectors MVS1 to MVS3 but also the image frames FR1 and FR2 . The controller 15 compares the image frames FR1 and FR2 and identifies the objects 21 , 23 , and 25 using pixel portions where differences occur. The controller 15 identifies a plurality of objects 21 , 23 , and 25 in the image frames FR1 and FR2 by comparing the image frames FR1 and FR2 with each other.

The controller 15 analyzes the directions of the motion vectors MVS1 to MVS3 and classifies the motion vectors MVS1 to MVS3 into motion vectors MVS1, motion vectors MVS2, and motion vectors MVS3. do. Distinguishing the motion vectors MVS1 from the motion vectors MVS2 is easy because the directions of the motion vectors MSV1 and MVS2 are opposite to each other. However, it is difficult to distinguish the motion vectors MVS2 and MVS3 because the directions of the motion vectors MVS2 and MVS3 are the same. At this time, the controller 15 may use a pixel difference between the first image frame FR1 and the second image frame FR2. This is because the objects 21, 23, and 25 in the first image frame FR1 and the second image frame FR2 have different pixel values due to the movement of the objects 21, 23, and 25.

The controller 15 selects representative motion vectors SMV1, SMV2, or SMV3 from among a plurality of motion vectors MVS1, MVS2, or MVS3 using the identified plurality of objects 21, 23, and 25. do. Using the identified plurality of objects 21, 23, and 25 means that the plurality of objects 21, 23, and 25 are identified so that the plurality of motion vectors MVS1, MVS2, or MVS3 are clearly This means that it is divided into three parts. According to an exemplary embodiment, the controller 15 may select representative motion vectors SMV1 , SMV2 , or SMV3 from among a plurality of motion vectors MVS1 , MVS2 , or MVS3 . The controller 15 may select any one motion vector among the plurality of motion vectors MVS1 , MVS2 , or MVS3 as the representative motion vectors SMV1 , SMV2 , or SMV3 . For example, the controller 15 may select an arbitrary motion vector from among a plurality of motion vectors MVS1 and set it as the representative motion vector SMV1.

According to an exemplary embodiment, the controller 15 may select each of the largest motion vectors among the plurality of motion vectors MVS1 , MVS2 , or MVS3 as the representative motion vectors SMV1 , SMV2 , or SMV3 . For example, the controller 15 may select a motion vector having the largest magnitude among a plurality of motion vectors MVS1 and set it as the representative motion vector SMV1. The controller 15 may first calculate the magnitudes of the plurality of motion vectors MVS1. In FIG. 2 , each of the plurality of motion vectors MVS1 , MVS2 , or MVS3 is illustrated as having the same size, but the size of the plurality of motion vectors MVS1 , MVS2 , or MVS3 may vary according to embodiments.

The controller 15 calculates the magnitudes and directions of the selected representative motion vectors SMV1, SMV2, and SMV3. The controller 15 calculates the magnitude of the selected representative motion vector SMV1 and identifies the downward direction. The controller 15 calculates the magnitude of the selected representative motion vector SMV2 and identifies the upward direction. The controller 15 calculates the magnitude of the selected representative motion vector SMV3 and identifies the upward direction.

FIG. 3 shows radar data for explaining an operation of extracting Doppler velocities and directions performed by the controller shown in FIG. 1 . In FIG. 3 , radar data is represented by dots in a graph.

Referring to FIGS. 1 and 3 , the radar sensor 11 generates radar data RD by transmitting a radar signal. Radar data RD generated by the radar sensor 11 is transmitted to the controller 15 .

Points indicated by arrows in the radar data RD shown in FIG. 3 mean moving objects 31, 33, and 35. Dots in the radar data RD mean objects reflected by a radar signal transmitted from the radar sensor 11 . The points may be moving objects or non-moving objects. The moving objects 31, 33, and 35 are marked with arrows at points. For example, in FIG. 3 there are points indicated by three arrows. Points marked with three arrows mean moving objects 31 , 33 , and 35 . The moving objects 31 , 33 , and 35 in FIG. 3 correspond to the objects 21 , 23 , and 25 in the image frames FR1 and FR2 shown in FIG. 2 . In FIG. 3 , the arrows are all shown in the same size, but the arrows may have different sizes. The Doppler velocities DV1, DV2, and DV3 and the size of the arrows are proportional to each other. That is, the larger the Doppler velocities DV1, DV2, and DV3, the larger the size of the arrows.

The controller 15 may extract Doppler velocities DV1 , DV2 , and DV3 and directions of the moving objects 31 , 33 , and 35 through three arrows. For example, the controller 15 may extract the Doppler velocity DV1 and direction of the moving object 31 indicated by the arrow. The direction is 90 degrees relative to the y-axis. Similarly, the controller 15 may extract the Doppler velocity (DV2) and direction of the moving object 33 indicated by the arrow. The direction is between 40 degrees and 60 degrees relative to the y-axis.

Referring to FIGS. 1 to 3 , the controller 15 compares calculated amplitudes of representative motion vectors SMV1 , SMV2 , and SMV3 with Doppler velocities DV1 , DV2 , and DV3 . The controller 15 sorts the amplitudes of the representative motion vectors SMV1, SMV2, and SMV3 in ascending order, and also sorts the Doppler velocities DV1, DV2, and DV3 in ascending order. The controller 15 matches the aligned representative motion vectors SMV1 , SMV2 , and SMV3 and the aligned Doppler velocities DV1 , DV2 , and DV3 to each other. For example, the representative motion vector SMV1 may be matched with the Doppler velocity DV1. At this time, the representative motion vector SMV1 is the largest among the representative motion vectors SMV1, SMV2, and SMV3, and the Doppler velocity DV1 is the largest among the Doppler velocities DV1, DV2, and DV3. The representative motion vector SMV2 may be matched with the Doppler velocity DV2, and the representative motion vector SMV3 may be matched with the Doppler velocity DV3. Among the representative motion vectors SMV1 , SMV2 , and SMV3 , the representative motion vector SMV2 is the next largest, and among the Doppler velocities DV1 , DV2 , and DV3 , the Doppler velocity DV2 is the next largest.

The controller 15 overlaps the frame DFR shown in FIG. 2 with the plurality of motion vectors MVS1 , MVS2 , or MVS3 and the radar data RD shown in FIG. 3 .

The controller 15 controls the directions of the representative motion vectors SMV1, SMV2, and SMV3 of the objects 21, 23, and 25 and the plurality of objects 31, 33, and 35) are compared. For example, the representative motion vector SMV1 of the object 21 shown in FIG. 2 is directed downward. In the radar data RD shown in FIG. 3, the direction of the object 31 is also directed downward. When displaying the object 31 with (x, y) coordinates, the object 31 may be displayed as (39, 45). Since the representative motion vector SMV1 of the object 21 shown in FIG. 2 and the direction of the object 31 located at (39, 45) in the radar data RD are in the same direction, the controller 15 is shown in FIG. The representative motion vector SMV1 of the object 21 and the direction of the object 31 located at (39, 45) in the radar data RD are matched. A representative motion vector SMV2 of the object 23 shown in FIG. 2 is directed in an upper right direction. The direction of the object 33 located at (15, 21) in the radar data RD shown in FIG. 3 is also toward the upper right direction. Since the representative motion vector SMV2 of the object 23 shown in FIG. 2 and the direction of the object 33 located at (15, 21) in the radar data RD are in the same direction, the controller 15 is shown in FIG. The representative motion vector SMV2 of the object 23 and the direction of the object 33 located at (15, 21) in the radar data RD are matched. Since the representative motion vector SMV3 of the object 25 shown in FIG. 2 and the direction of the object 35 located at (8, -42) in the radar data RD are in the same direction, the controller 15 is The representative motion vector SMV3 of the illustrated object 25 and the direction of the object 35 located at (8, -42) in the radar data RD are matched.

The controller 15 determines the plurality of objects 21, 23, and 25 identified in the image frames (FR1 or FR2 in FIG. 2) and the plurality of objects recognized in the radar data RD according to the comparison. 31, 33, and 35).

Specifically, the representative motion vector SMV1 and the Doppler velocity DV1 are matched, and the representative motion vector SMV1 of the object 21 and the direction of the object 31 located at (39, 45) in the radar data RD When this is matched, the controller 15 matches the object 21 shown in FIG. 2 and the object 31 shown in FIG. That is, the controller 15 recognizes the object 21 shown in FIG. 2 and the object 31 shown in FIG. 3 as the same object. The representative motion vector SMV2 and the Doppler velocity DV2 are matched, and the representative motion vector SMV2 of the object 23 and the direction of the object 33 located at (15, 21) in the radar data RD are matched. At this time, the controller 15 matches the object 23 shown in FIG. 2 and the object 33 shown in FIG. That is, the controller 15 recognizes the object 23 shown in FIG. 2 and the object 33 shown in FIG. 3 as the same object. The representative motion vector (SMV3) and the Doppler velocity (DV3) are matched, and the representative motion vector (SMV3) of the object 25 and the direction of the object 35 located at (8, -42) in the radar data (RD) are matched. When done, the controller 15 matches the object 25 shown in FIG. 2 and the object 35 shown in FIG. That is, the controller 15 recognizes the object 25 shown in FIG. 2 and the object 35 shown in FIG. 3 as the same object.

The controller 15 uses the generated plurality of motion vectors MVS1 to MVS3 to select any one of the plurality of objects (eg, 21, 23, and 25) identified in the image frames FR1 and FR2. Calculate the energy of (e.g. 21, 23, or 25). The energy is calculated as follows.

The controller 15 identifies the Doppler speed of the recognized object (eg, 31) in the radar data RD corresponding to any one object (eg, 21).

The controller 15 divides the identified Doppler speed by a preset Doppler speed to calculate a motion change rate of any one object 21 . The preset Doppler speed may be an average Doppler speed according to a general movement of an arbitrary person. The motion change rate of any one of the objects 21 has a value between 0 and 1.

A motion change rate of any one of the objects 21 may be calculated as the energy. That is, the motion change rate of any one object 21 is the energy.

According to an embodiment, the controller 15 may calculate the energy in consideration of the size fit as well as the motion change rate. The size fit is calculated as follows.

The controller 15 counts pixel values of any one object (eg, 21) in any one of the image frames FR1 and FR2, and determines the size of the one object 21 Calculate

The controller 15 calculates a difference between the size of any one object 21 and the average object size. The difference value is an absolute value. The average object size means the size of a typical adult. If any one of the objects 21 is a human, the difference between the size of the one object 21 and the average object size will have a small value. On the other hand, if the one object 21 is not a human but an animal such as a pigeon or cat, the difference between the size of the one object 21 and the average object size will have a large value.

The controller 15 divides the difference between the size of the one object 21 and the average object size by the average object size to calculate the size conformity of the one object 21 . The size suitability of any one of the objects 21 has a value between 0 and 1.

The controller 15 calculates the sum of the motion change rate of the one object 21 and the size conformity of the one object 21 as the energy of the one object 21 . At this time, the energy may be expressed by the following equation.

[Equation 1]

EN = α * MCR + β * IOSP

The EN represents energy, the α and β represent coefficients, the MCR represents a motion change rate, and the IOSP represents a size fit. The energy (EN) has a value between 0 and 1, and the coefficients (α, β), motion rate of change (MCR), or size fit (IOSP) have a value between 0 and 1. The coefficients α and β may be arbitrarily determined according to the importance of motion change rate (MCR) and size fit (IOSP).

Depending on the embodiment, the controller 15 may calculate energy by considering the size ratio of any one of the objects 21 . A method for calculating the size ratio is as follows.

The controller 15 divides the calculated size of the object 21 by the resolution of the camera module 13 to calculate the size ratio of any one of the objects 21 . The reason why the size ratio of any one of the objects 21 is calculated is that the size of the real object 21 may be distorted according to the resolution of the camera module 13 . For example, when the resolution of the camera module 13 is low, the size of the object may be calculated to be larger than the size of the actual object 21 . The value of the calculated size ratio of the object 21 has a value between 0 and 1.

The controller 15 calculates the sum of the motion change rate of the one object 21, the size conformity of the one object 21, and the size ratio of the one object 21 as the energy of the one object 21. Calculate with At this time, the energy may be expressed by the following equation.

[Equation 2]

EN = α * MCR + β * IOSP+γ*OSR

The EN represents energy, the α, β, and γ represent coefficients, the MCR represents a motion change rate, the IOSP represents a size fit, and the OSR represents a size ratio. The energy (EN) has a value between 0 and 1, and the coefficients (α, β, γ), motion rate of change (MCR), size fit (IOSP), or size ratio (OSR) have a value between 0 and 1. .

When the calculated energy is greater than the arbitrary threshold, the controller 15 determines that any one of the objects 21 is an object of interest, such as a person. When the calculated energy is greater than the arbitrary threshold, the controller 15 sends a message to the server 20. This message is a message to the manager 23 to pay attention to the image frames FR1 and FR2 displayed on the display 41 . The message is transmitted to the server 20 and displayed on the display 41 . Therefore, the manager 43 can more efficiently monitor the object of interest (eg, 21) included in the image frames FR1 and FR2 displayed on the display 41 with attention.

When the calculated energy is less than the arbitrary threshold, the controller 15 determines that the object is not of interest, such as a person. When not the object of interest, such as the person, the object may be a bird or an animal such as a cat. When the calculated energy is less than the arbitrary threshold, the controller 15 does not send a message to the server 20.

Figure 4 shows a flow chart for explaining the operation of the controller shown in Figure 1;

1 to 4 , the controller 15 receives image frames VFS captured at different times from the camera module 13 (S10).

The controller 15 generates a plurality of motion vectors MVS1, MVS2, and MVS3 from the image frames FR1 and FR2 (S20).

The controller 15 identifies a plurality of objects 21, 23, and 25 in the image frames FR1 and FR2 using the generated plurality of motion vectors MVS1, MVS2, and MVS3 ( S30).

The controller 15 uses the identified plurality of objects 21, 23, and 25 to generate representative motion vectors (eg, SMV1, SMV2, and SMV3) among a plurality of motion vectors MVS1, MVS2, and MVS3. ) is selected (S40).

The controller 15 calculates magnitudes and directions of the selected representative motion vectors SMV1, SMV2, and SMV3 (S50).

The controller 15 extracts Doppler velocities DV1, DV2, and DV3 and directions of the plurality of objects 31, 33, and 35 recognized from the radar data RD generated from the radar sensor 11. Do (S60).

The controller 15 compares the magnitudes of the calculated representative motion vectors SMV1, SMV2, and SMV3 with the Doppler velocities DV1, DV2, and DV3, and compares the calculated representative motion vectors SMV1, Directions of SMV2 and SMV3) and directions of the plurality of objects 31, 33, and 35 recognized from the radar data RD are compared (S70).

The controller 15 controls the plurality of objects 21, 23, and 25 identified from the image frames FR1 and FR2 according to the comparison and the plurality of objects 31, 33, and 25 recognized from the radar data RD. and 35) are matched (S80).

FIG. 5 is a diagram of a neural network for deriving a relationship between coordinates of a radar sensor and coordinates of a camera performed by the controller shown in FIG. 1 .

Referring to FIGS. 1 to 3 and 5 , coordinates of the camera module 13 may be expressed as (u, v). The coordinates of the radar sensor 11 may be expressed as (x, y). A coordinate system of the camera module 13 and a coordinate system of the radar sensor 11 are different from each other. A neural network is used to match the coordinate system of the different camera modules 13 and the coordinate system of the radar sensor 11 . Coordinates of the camera module 13 and coordinates of the radar sensor 11 are used as training data to learn the neural network. The neural network is implemented with an input layer, a hidden layer, and an output layer. The neural network outputs u and v of the camera module 13, respectively. A relationship between the coordinates of the camera module 13 and the coordinates of the radar sensor 11 may be derived through learning of the neural network. Each node of the hidden layer may be implemented as a Gaussian function.

FIG. 6 is a flowchart for explaining an operation of the controller shown in FIG. 1 according to another embodiment.

Referring to FIGS. 1 to 3, 5, and 6 , the controller 15 receives image frames VFS captured at different times from the camera module 13 (S100).

The controller 15 identifies a plurality of objects 21, 23, and 25 in the image frames FR1 and FR2 by comparing the image frames FR1 and FR2 with each other (S110).

The controller 15 derives a relationship between the coordinates of the radar sensor 11 and the coordinates of the camera module 13 by applying the coordinates of the radar sensor 11 to the neural network (S120).

The controller 15 uses the derived relationship to move the plurality of objects 21, 23, and 25 identified in the image frames FR1 and FR2 and the radar data generated from the radar sensor 11. (31, 33, and 35) are matched (S130).

As such, the present invention is not limited to the described embodiments, and it is obvious to those skilled in the art that various modifications and variations can be made without departing from the spirit and scope of the present invention. Accordingly, it should be said that such modifications or variations fall within the scope of the claims of the present invention.

100: video surveillance system;
10: camera;
11: radar sensor;
13: camera module;
15: controller;
20: server;
41: display

Claims (5)

In the radar and camera fusion method of a video surveillance system including a radar sensor, a camera module, and a controller,
receiving, by the controller, image frames captured at different times from the camera module;
identifying, by the controller, a plurality of objects in the image frames by comparing the image frames with each other;
deriving, by the controller, a relationship between the coordinates of the radar sensor and the coordinates of the camera module by applying the coordinates of the radar sensor to a neural network;
matching, by the controller, moving objects from radar data generated from the radar sensor with a plurality of objects identified from the image frames using the derived relationship;
calculating, by the controller, energy of one object among the identified plurality of objects;
determining, by the controller, if the calculated energy is greater than a certain threshold; and
When the calculated energy is greater than the arbitrary threshold value, the controller sends a message to the server to pay attention to the manager and view the image frames displayed on the display,
The controller matching the moving objects in the radar data generated from the radar sensor with the plurality of objects identified in the image frames using the derived relationship,
generating, by the controller, a plurality of motion vectors from the image frames;
identifying, by the controller, the plurality of objects in the image frames using the generated plurality of motion vectors;
selecting, by the controller, motion vectors having the largest magnitude among the plurality of motion vectors as representative motion vectors;
calculating, by the controller, magnitudes and directions of the selected representative motion vectors;
extracting, by the controller, Doppler velocities and directions of the plurality of objects recognized from radar data generated from the radar sensor;
arranging, by the controller, magnitudes of the calculated representative motion vectors in ascending order, and arranging the Doppler velocities in ascending order;
matching, by the controller, the sorted representative motion vectors and the sorted Doppler velocities to each other according to the sorted order;
comparing, by the controller, magnitudes of the calculated representative motion vectors with the Doppler velocities, and comparing directions of the calculated representative motion vectors with directions of the plurality of objects recognized from the radar data; and
and matching, by the controller, the plurality of objects identified from the image frames and the plurality of objects recognized from the radar data according to the comparison. delete The method of claim 1, wherein the step of calculating, by the controller, the energy of any one of the identified plurality of objects,
identifying, by the controller, a Doppler velocity of an object recognized from the radar data corresponding to the one object; and
The controller calculates a motion change rate of the one object by dividing the identified Doppler speed by a preset Doppler speed. The method of claim 3, wherein the step of calculating, by the controller, the energy of any one object among the identified plurality of objects,
calculating, by the controller, a size of the one object by counting pixel values of the one object in one of the image frames;
calculating, by the controller, a difference between the size of any one object and the average object size;
calculating, by the controller, size conformity of the one object by dividing a difference between the size of the one object and the average object size by the average object size; and
Wherein the controller calculates, by the controller, a sum of a motion change rate of the one object and a size conformity of the one object as energy of the one object. includes a camera;
the camera,
a radar sensor that generates radar data by transmitting a radar signal;
a camera module generating image frames by capturing images at different times; and
The image frames are received, the image frames are compared with each other to identify a plurality of objects in the image frames, and the coordinates of the radar sensor are applied to a neural network to determine the relationship between the coordinates of the radar sensor and the coordinates of the camera module. Derives, and uses the derived relationship to match a plurality of objects identified in the image frames with moving objects in radar data generated from the radar sensor, and of any one of the identified plurality of objects. Calculate the energy, determine whether the calculated energy is greater than a certain threshold value, and when the calculated energy is greater than the certain threshold value, send a message to the server to pay attention to the manager and look at the image frames displayed on the display. Including a controller that transmits,
The controller generates a plurality of motion vectors in the image frames to match moving objects in radar data generated from the radar sensor with a plurality of objects identified in the image frames, and the generated plurality of motion vectors The plurality of objects are identified in the image frames using , and each of the motion vectors having the largest magnitude among the plurality of motion vectors is selected as representative motion vectors using the identified plurality of objects, Magnitudes and directions of the selected representative motion vectors are calculated, Doppler velocities and directions of the plurality of objects recognized from the radar data generated from the radar sensor are extracted, and magnitudes of the calculated representative motion vectors are ranked in order of increasing order. sort, sort the Doppler velocities in increasing order, match the sorted representative motion vectors and the sorted Doppler velocities with each other according to the sorted order, and match the magnitudes of the calculated representative motion vectors and the Doppler velocity and compares directions of the calculated representative motion vectors with directions of the plurality of objects recognized from the radar data, and the controller compares the identified plurality of objects in image frames according to the comparison with the directions of the plurality of objects. A radar and camera fusion system that matches the plurality of objects recognized in radar data.

Images