WO2023098362A1

WO2023098362A1 - Target area security and monitoring system based on hundred-million-level pixel camera

Info

Publication number: WO2023098362A1
Application number: PCT/CN2022/128144
Authority: WO
Inventors: 袁潮; 温建伟; 邓迪旻
Original assignee: 北京拙河科技有限公司
Priority date: 2021-12-01
Filing date: 2022-10-28
Publication date: 2023-06-08
Also published as: CN114187724A; CN114187724B

Abstract

A target area security and monitoring system based on a hundred-million-level pixel camera, belonging to the technical field of high-resolution image recognition and processing. The system comprises: a light field camera array, which comprises a plurality of light field cameras; a light field information storage array, which is used for storing light field information that is acquired by the light field camera array with respect to a target area; an overlapping light field recognition sub-system, which is used for recognizing overlapping light field information in two different groups of light field information that are obtained by at least two light field cameras; a target light field voxelization sub-system, which is used for executing voxelization processing on a target light field corresponding to the overlapping light field information, so as to obtain a plurality of voxelization results; and a security and monitoring processing sub-system, which sends a security or monitoring signal on the basis of the plurality of voxelization results, with at least one of the plurality of light field cameras having more than a hundred million pixels. The system can realize intelligent and smart monitoring and security recognition of a hundred-million-level pixel monitoring area on the basis of a light field camera array.

Description

A target area security and monitoring system based on a megapixel camera

technical field

The invention belongs to the technical field of high-resolution image recognition and processing, and in particular relates to a target area security and monitoring system based on a billion-level pixel camera.

Background technique

The emergence of light field cameras based on microlens arrays makes it possible to apply light field technology to ordinary cameras. This type of light field camera records the light field information in space, and then calculates and processes the light field, which can realize functions such as digital focus, depth of field extension, and even aberration correction. Compared with traditional imaging methods, light field cameras have great advantages. The images or videos captured are rich in information, and the resolution reaches 100 million levels.

High resolution (high pixel) represents high quality and high definition, and high-definition video images have a wide range of applications in military, medical, monitoring, astronomy, etc. Especially in area monitoring, high-definition video images not only greatly enhance the direct observation ability of the naked eye, but also make the subsequent image or video recognition processing results easier to visualize.

The Chinese invention patent application with the application number CN201410637958 proposes a light field acquisition control method, which claims that the refocusing point adjustment range of the light field image captured by the light field camera can be adjusted to a certain extent, thereby better satisfying users or Diversified practical application requirements of equipment and other subjects.

However, when the imaging resolution exceeds 100 million levels, the amount of data generated also increases dramatically; and when using a light field camera to acquire image data, the light field camera can not only collect image information (planar, two-dimensional image information), but also Similar to lidar, the depth map information corresponding to this picture and the point cloud information corresponding to this depth map are generated by collecting a frame of image; the light field camera can also obtain the spatial information and angle information of the light during imaging at the same time, This increases the dimension of the data accordingly. In the case of simultaneous increase in data dimension and data volume, how to quickly and timely process the monitoring scene data with 100-million-level pixels has become a technical problem that technicians in the field need to deal with urgently.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a target area security and monitoring system based on a megapixel camera.

Structurally, the target area security and monitoring system includes a light field camera array, a light field information storage array, an overlapping light field identification subsystem, a target light field voxelization subsystem, and a security and monitoring processing subsystem.

Wherein, the light field camera array includes a plurality of light field cameras; the light field information storage array is used to store the light field information obtained by the light field camera array for the target area; the overlapping light field identification subsystem is used to identifying overlapping light field information in two sets of different light field information obtained by at least two light field cameras;

The target light field voxelization subsystem is configured to perform voxelization processing on the target light field corresponding to the overlapping light field information to obtain multiple voxelization results.

The security and monitoring processing subsystem sends a security signal or a monitoring signal based on the plurality of voxelization results;

The monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array;

The security signal prompts managers that there is an abnormality in the current monitoring area.

As the most prominent feature of the application scene of the present invention, at least one of the plurality of light field cameras has more than 100 million pixels.

As one of the further improvements of the present invention, the target light field voxelization subsystem is used to perform voxelization processing under multiple viewing angles on the target light field corresponding to the overlapping light field information, and obtain multiple volumes under different viewing angles. primed result.

The security and monitoring processing subsystem obtains the voxelization results under the multiple different viewing angles, and judges whether the target objects contained in the voxelization results under the multiple different viewing angles are abnormal;

If there is an abnormality, a security signal is sent, and the security signal prompts the manager that there is an abnormality in the current monitoring area.

If there is an abnormal target object in the target area, the monitoring signal is generated at the same time, and the monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array, and the shooting angle is determined by the first One of the angle of view, the second angle of view or the third angle of view.

Specifically, the first viewing angle is a first viewing angle parallel to the first coordinate axis of the Cartesian coordinate system, the second viewing angle is a second viewing angle parallel to the second coordinate axis of the Cartesian coordinate system, and the first viewing angle is parallel to the second coordinate axis of the Cartesian coordinate system. The triangular viewing angle is a third viewing angle parallel to the third coordinate axis of the Cartesian coordinate system.

As the light field camera array used in the technical solution of the present invention, the light field camera array includes a first group of light field cameras, a second group of light field cameras and a third group of light field cameras;

The first set of light field cameras captures a first angle range, the second set of light field cameras captures a second angle range, and the third light field camera captures a third angle range;

The first angular range overlaps with the second angular range, the second angular range overlaps with the third angular range, and the first angular range does not overlap with the third angular range.

Moreover, the shooting angles of the first group of light field cameras and the third group of light field cameras are adjustable; the shooting angles of the second group of light field cameras are not adjustable.

The technical solution of the present invention can realize multi-angle target area monitoring through only one light field camera array; and, when the light field camera in the target area obtains billion-level pixel data, multi-view voxelization processing is performed, Comprehensive multi-view voxelization processing results to determine whether there is any abnormality, thereby ensuring the accuracy of security and monitoring of the target area with billions of pixels, and avoiding the one-sidedness of single-angle monitoring processing.

Further advantages of the present invention will be further embodied in detail in the specific embodiments in conjunction with the accompanying drawings.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a schematic structural diagram of a target area security and monitoring system based on a megapixel camera according to an embodiment of the present invention;

Fig. 2 is a structural schematic diagram of the light field camera array used in the target area security and monitoring system based on the megapixel camera described in Fig. 1;

Fig. 3 is a schematic diagram of the principle of performing voxelization processing on light field information;

Fig. 4 is a schematic diagram of voxelization results of multiple different viewing angles used in various embodiments of the present invention;

Fig. 5 is a schematic diagram of the effect of the security and monitoring of the target area realized by the present invention

Detailed ways

In the following, the invention will be further described in conjunction with the accompanying drawings and specific embodiments.

Before introducing various embodiments of the present invention, technical terms related to the technical solutions of the present application are introduced first, so as to better understand the technical solutions and improved ideas of the present application.

Array: It is an array of components formed by arranging multiple components together in a certain shape or rule.

In the present invention, the light field camera array is an element array formed by arranging multiple light field cameras or multiple light field sensors (lenses) together in a certain shape or rule.

Light field, similar to the concept of electric field and magnetic field, is used to describe some characteristics of light, which contains information such as light intensity, position, and direction;

A light-field camera, also known as a plenoptic camera, specifically uses a large number of tiny lenses to capture light from different sources and angles, and each lens is responsible for processing a certain number of pixels. Theoretically, if the number of lenses is large enough, the light captured in the entire light field area can be clearly distinguished.

The light field camera can not only collect image information (planar, two-dimensional image information), but also can generate the depth map information corresponding to this picture through a frame of image collected similar to lidar, and the corresponding depth map information of this depth map. Point cloud information; the light field camera can also obtain the spatial information and angle information of the light during imaging at the same time, and map the pixels in the two-dimensional image into a multi-dimensional (more than 2-dimensional, such as three-dimensional or four-dimensional) light field according to certain rules for reprojection. Obtain in-focus images of different viewing angles and different phase planes.

A point cloud refers to a massive point collection of surface properties of a target.

The point cloud obtained according to the principle of laser measurement, including three-dimensional coordinates (XYZ) and laser reflection intensity (Intensity).

The point cloud obtained according to the principle of photogrammetry, including three-dimensional coordinates (XYZ) and color information (RGB).

The point cloud is obtained by combining the principles of laser measurement and photogrammetry, including three-dimensional coordinates (XYZ), laser reflection intensity (Intensity) and color information (RGB).

After obtaining the spatial coordinates and depth information of each sampling point on the surface of the object, a collection of points is obtained, which is called "Point Cloud".

For further introduction about the light field camera, please refer to the following prior art:

Adelson E H, Wang J Y A.Single Lens Stereo with a Plenoptic Camera[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1992,14(2):99-106.

US7965936B2: 4D light field cameras;

Su H, Maji S, Kalogerakis E, et al.Multi-view Convolutional Neural Networks for 3D Shape Recognition[J].IEEE,2015.

Veeraraghavan A, Raskar R, Agrawal A, et al. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture[J].Acm Trans Graph,2007,26(3):69.

The above prior art is introduced as part of the technical solution disclosure of the present invention.

In Figure 1, the target area security and monitoring system based on megapixel cameras includes:

A light field camera array, the light field camera array comprising a plurality of light field cameras;

A light field information storage array, the light field information storage array is used to store the light field information acquired by the light field camera array for the target area;

An overlapping light field identification subsystem, the overlapping light field identification subsystem is used to identify overlapping light field information in two sets of different light field information obtained by at least two light field cameras;

A target light field voxelization subsystem, the target light field voxelization subsystem is configured to perform voxelization processing on the target light field corresponding to the overlapping light field information to obtain multiple voxelization results;

A security and monitoring processing subsystem, where the security and monitoring processing subsystem sends a security or monitoring signal based on the plurality of voxelization results;

Wherein, at least one of the plurality of light field cameras has more than 100 million pixels.

The shooting angle of the light field camera included in the light field camera array is adjustable;

The monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array.

Specifically, on the basis of FIG. 1 , refer to FIG. 2 .

In FIG. 2, the light field camera array includes a first group of light field cameras 1, a second group of light field cameras 2 and a third group of light field cameras 3;

The first group of light field cameras 1 captures a first angle range, the second group of light field cameras 2 captures a second angle range, and the third light field camera captures a third angle range;

And, referring to the schematic diagram on the right side of FIG. 2 , the shooting angles of the first group of light field cameras 1 and the third group of light field cameras 3 can be adjusted;

The shooting angles of the second group of light field cameras 2 are not adjustable.

According to this group configuration, there is always a group of light field cameras with a fixed shooting angle, while the other two groups of light field cameras can adjust their respective shooting ranges (angles) in opposite directions.

In terms of specific structure, each light field camera is composed of multiple lenses. The video pictures taken by each lens are consistent in the vertical direction, and different angles are taken in the horizontal direction. The pictures taken by each camera have a certain overlapping area to ensure that adjacent lenses There are enough features between video frames for splicing and fusion.

Therefore, the present invention can realize multi-angle target area monitoring through only one light field camera array.

Preferably, in an embodiment of the present invention, the target area security and monitoring system based on megapixel cameras only includes one light field camera array, and the one light field camera array includes three sets of light field cameras.

After the light field camera in Fig. 1-Fig. 2 sorts and obtains the light field information obtained for the target area, it sends it to the light field information storage array;

Since the light field camera array is composed of multiple light field cameras, and each light field camera captures light from different sources and at different angles by a very large number of tiny lenses, each lens is responsible for processing a certain number of pixels. In practical applications, When each light field camera includes enough lenses, the number of pixels of the camera can reach more than 100 million;

In addition, the light field camera can not only collect image information (planar, two-dimensional image information), but also can generate the depth map information corresponding to this picture through a frame of image collected similar to lidar, and this depth map The corresponding point cloud information; the light field camera can also obtain the spatial information and angle information of the light during imaging at the same time, so its imaging pixels can easily reach 100 million levels, and the amount of data is huge.

As a more preferred example, the imaging resolution of the light field camera array exceeds 100 million.

Preferably, the light field information storage array is a circular storage stack.

Using the storage method of the ring stack instead of the queue can avoid the "overflow" phenomenon of the storage space in the case of a large number.

When the ring stack is full, the stored data of the full stack can be sent to the overlapping light field recognition subsystem at one time instead of sending it at any time, so as to reduce the pressure of data processing.

Then, the overlapping light field identifying subsystem is used to identify overlapping light field information in two sets of different light field information obtained by at least two light field cameras.

Next, enter the voxelization phase.

Specifically, perform voxelization processing, and obtain multiple voxelization results including:

identifying multiple key frame images containing the target object in the overlapping light field information;

Acquiring spatial angle information of light rays corresponding to the multiple key frame images generated by the light field camera;

determining a plurality of voxelized viewing angles based on the spatial angle information;

Perform voxelization on the target object contained in the overlapping light field information respectively under the plurality of voxelization viewing angles.

Voxelization refers to dividing the three-dimensional space into rasterized units according to the spatial position, and the point cloud is encoded and calculated in a unified manner according to the grid to which it belongs. Generally speaking, voxelization can improve the regularity of point cloud data and improve computing efficiency.

Fig. 3 shows a schematic schematic diagram for performing voxelization on point cloud data.

However, Fig. 3 shows that the point cloud scene is described by segmenting voxels in the three directions of x, y, and z, and the segmentation method is finely divided in all three dimensions.

The inventors found that when performing voxelization in the three directions of x, y, and z, the choice of voxel size has a great influence on the accuracy and time-consuming. If the division is too dense, the calculation amount will be greatly increased and the time-consuming will be increased; while the division will be sparse. It will lead to a significant reduction in accuracy and is highly subjective.

Therefore, in the technical solution of the present invention, the target light field voxelization subsystem is used to perform voxelization processing under multiple viewing angles on the target light field corresponding to the overlapping light field information, and obtain multiple different viewing angles. Voxelization result.

The multiple perspectives here are no longer voxelization in the three directions of x, y, and z.

Specifically, the plurality of viewing angles include one of the following: viewing angles parallel to coordinate axes of a Cartesian coordinate system, viewing angles of a cylindrical coordinate system, and viewing angles of a spherical coordinate system.

As described in Figure 4.

Taking the Cartesian coordinate system as an example, the viewing angle parallel to the coordinate axis of the Cartesian coordinate system includes: the viewing angle parallel to the first coordinate axis of the Cartesian coordinate system, and the viewing angle parallel to the second coordinate axis of the Cartesian coordinate system , the viewing angle parallel to the third axis of the Cartesian coordinate system.

The security and monitoring processing subsystem sends a security signal based on the multiple voxelization results, specifically including:

The security and monitoring processing subsystem obtains the voxelization results under the multiple different viewing angles, and judges whether the target objects included in the voxelization results under the multiple different viewing angles are abnormal;

The security and monitoring processing subsystem obtains a first voxelization result at a first viewing angle parallel to the first coordinate axis of the Cartesian coordinate system, and a voxelization result at a second viewing angle parallel to the second coordinate axis of the Cartesian coordinate system. A second voxelization result and a third voxelization result under a third viewing angle parallel to the third coordinate axis of the Cartesian coordinate system;

If the depth information of the first voxelization result, the second voxelization result, and the third voxelization result do not match, it is determined that there is an abnormal target object in the target area.

As a specific implementation, the action mode analysis is performed on the planar imaging information at the angle of view (under the third angle of view) where the third voxelization result in the light field imaging information is parallel to the Z-axis of the Cartesian coordinate system, and it is obtained Analysis results of the first action mode;

Perform action pattern analysis on the planar imaging information at a viewing angle (under the first viewing angle) where the first voxelization result in the light field imaging information is parallel to the Y-axis of the Cartesian coordinate system, and obtain a second action pattern analysis result ;

Perform action pattern analysis on the planar imaging information at a viewing angle (under the second viewing angle) where the second voxelization result in the light field imaging information is parallel to the Z-axis of the Cartesian coordinate system, and obtain a third action pattern analysis result ;

Based on the multiple action pattern analysis results and the depth of field information corresponding to each imaging angle of view, it is judged whether there is an abnormality in the local target monitoring area, specifically including:

If the first action pattern analysis result, the second action pattern analysis result, and the third action pattern analysis result all determine that there is an abnormal movement, and the depth of field data of the z-axis, the depth of field data of the x-axis, and the depth of field data of the y-axis match each other , it is judged that there is an abnormality in the local target monitoring area.

The mutual matching here means that the target object in the monitoring area can be restored after the combination of the field depth data of the z axis, the depth of field data of the x axis and the depth of field data of the y axis.

Of course, the above only takes the Cartesian coordinate system as an example. In practical applications, the perspective of the cylindrical coordinate system and the perspective of the spherical coordinate system can also be used; or divided into bird's-eye perspective and projective perspective. As long as at least two or more plane imaging information are obtained from multiple viewing angles to perform motion pattern analysis, and multiple motion pattern analysis results are obtained, combined with the depth of field data, the above-mentioned steps of performing monitoring pattern analysis on the target monitoring area can be performed.

If there is an abnormal target object in the target area, the monitoring signal is generated at the same time, and the monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array, and the shooting angle is determined by the One of the first perspective, second perspective or third perspective is determined.

Fig. 5 shows a schematic diagram of the effect of the security and monitoring of the target area realized by the present invention.

In Fig. 5, the left side is the action mode analysis performed on the planar imaging information under the angle of view (under the first angle of view) where the first voxelization result in the light field imaging information is parallel to the Y-axis of the Cartesian coordinate system, and it is obtained The analysis result of the second action mode; the right side is the planar imaging information execution for the second voxelization result in the light field imaging information parallel to the Z-axis of the Cartesian coordinate system (under the second viewing angle). The action mode analysis is the third action mode analysis result obtained.

It can be seen from the figure that the depth information of the plane imaging information under the angle of view parallel to the Y axis (under the first angle of view) cannot match the depth information of the plane imaging information under the angle of view parallel to the Z axis (under the second angle of view), so the z axis The target object in the monitoring area cannot be restored after combining the depth of field data of the X axis, the depth of field data of the X axis, and the depth of field data of the Y axis, so there is an abnormality.

At this time, it is necessary to schedule the shooting angle of at least one light field camera in the light field information array to be an angle off the Y axis or the Z axis, that is, the shooting angle is clearer in the first viewing angle or the second viewing angle.

Through the local multi-view multi-modal analysis results, it is possible to accurately judge whether there is an abnormality in the local target monitoring area, avoiding the limitations of single-modal analysis. In terms of global recognition, through global overlapping light field recognition and multi-view voxelization As a result, it can be recognized from a global perspective whether the current shooting angle meets the monitoring needs of the current target area or target person.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall fall within the protection scope of the claims of the present invention.

For some module structures not particularly defined in the present invention, the content recorded in the prior art shall prevail. The prior art mentioned in the background of the present invention can be used as a part of the present invention to understand the meaning of some technical features or parameters. The protection scope of the present invention shall be determined by the contents actually recorded in the claims.

Claims

A target area security and monitoring system based on a megapixel camera, characterized in that:

The system includes:

A light field camera array, the light field camera array comprising a plurality of light field cameras;

A light field information storage array, the light field information storage array is used to store the light field information acquired by the light field camera array for the target area;

An overlapping light field identification subsystem, the overlapping light field identification subsystem is used to identify overlapping light field information in two sets of different light field information obtained by at least two light field cameras;

A target light field voxelization subsystem, the target light field voxelization subsystem is used to perform voxelization processing under multiple viewing angles on the target light field corresponding to the overlapping light field information, and obtain multiple volumes under different viewing angles. prime result;

A security and monitoring processing subsystem, the security and monitoring processing subsystem obtains a first voxelization result at a first viewing angle parallel to the first coordinate axis of the Cartesian coordinate system, and a second coordinate axis of the Cartesian coordinate system A second voxelization result under a parallel second viewing angle and a third voxelization result under a third viewing angle parallel to the third coordinate axis of the Cartesian coordinate system;

If the depth information of the first voxelization result, the second voxelization result, and the third voxelization result do not match, it is judged that there is an abnormal target object in the target area, and a security or monitoring signal is sent.
A target area security and monitoring system based on a megapixel camera as claimed in claim 1, characterized in that:

The shooting angle of the light field camera included in the light field camera array is adjustable;

The monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array.
A target area security and monitoring system based on a megapixel camera as claimed in claim 1, characterized in that:

At least one of the plurality of light field cameras has more than one hundred million pixels.
A target area security and monitoring system based on a megapixel camera as claimed in claim 1, characterized in that:

The security signal prompts managers that there is an abnormality in the current monitoring area.
A target area security and monitoring system based on a megapixel camera according to any one of claims 1-4, characterized in that:

The light field camera array includes a first group of light field cameras, a second group of light field cameras and a third group of light field cameras;

The first group of light field cameras captures a first angle range, the second group of light field cameras captures a second angle range, and the third group of light field cameras captures a third angle range;

The first angular range overlaps with the second angular range, the second angular range overlaps with the third angular range, and the first angular range does not overlap with the third angular range.
A target area security and monitoring system based on a megapixel camera as claimed in claim 5, characterized in that:

The shooting angles of the first group of light field cameras and the third group of light field cameras are adjustable;

The shooting angle of the second group of light field cameras is not adjustable.
A target area security and monitoring system based on a megapixel camera as claimed in claim 3 or 4, characterized in that:

The multiple viewing angles further include one of the following: a viewing angle of a cylindrical coordinate system, and a viewing angle of a spherical coordinate system.
A target area security and monitoring system based on a megapixel camera as claimed in claim 1, characterized in that:

If there is an abnormal target object in the target area, the monitoring signal is generated at the same time, and the monitoring signal is used to schedule the shooting angle of at least one light field camera in the light field information array, and the shooting angle is determined by the One of the first perspective, second perspective or third perspective is determined.