WO2020005114A1

WO2020005114A1 - Television camera and method for forming a panoramic video image and recognizing objects therein

Info

Publication number: WO2020005114A1
Application number: PCT/RU2019/050070
Authority: WO
Inventors: Ярослав Ярославович ПЕТРИЧКОВИЧ; Станислав Юрьевич МИЛЛЕР; Владимир Владимирович САМОЙЛОВ
Original assignee: Акционерное общество Научно-производственный центр "Электронные вычислительно-информационные системы"
Priority date: 2018-05-29
Filing date: 2019-05-29
Publication date: 2020-01-02
Also published as: RU2686154C1

Abstract

The invention relates to the field of video surveillance and objects recognition. The technical result of the claimed invention is to create a TV camera and a method to form panoramic video image and recognize objects in this video image with simpler and cost-effective design through the use of a built-in camera computer, that controls the designated route movements of a tilt-turn platform on which the camera is installed; at the same time the camera computer forms an integrated panoramic video image, and also identifies and recognizes objects in each frame of the video image.

Description

CAMERA AND METHOD FOR FORMING A PANORAMIC VIDEO IMAGE AND RECOGNITION OF OBJECTS ON IT

Technical field

The invention relates to the field of video surveillance and object recognition, namely to television cameras and methods for forming a panoramic video image and object recognition on it, and can be used in video surveillance systems.

State of the art

Currently, in television cameras they use as far as the main methods of forming a panoramic video image.

In multi-objective cameras (US2018095275), the formation of a panoramic video image is carried out from several video streams both inside the camera, by using the computing resources of the computer integrated in the camera, and using an external computer.

In a set of stationary wide-angle cameras with an overlapping field of view (CN205750270 (U)), the panoramic video image is formed using an external computer that forms a single panoramic video stream from several video streams from stationary television cameras.

Closest to the claimed invention is a rotary camera described in patent CN106331653, in which the formation of a panoramic video image is carried out when the camera moves along a predetermined route with stops at predetermined points. At these points on the route, frames of the video stream are captured. In an external computer, a panoramic image is formed from the captured frames. On the next pass along a predetermined route, the image is updated. Thus, a panoramic video stream with a low frame rate is formed in an external computer. The frame rate of the final video stream is determined by the speed capabilities of the PTZ camera and the computing capabilities of an external computer. For a wide range of tasks for the organization of video surveillance at stationary objects, such a frame rate is enough. This camera is selected as a prototype of the claimed invention. The disadvantages of analog cameras and the prototype of the invention are the required high power of external computers, as well as the high quality and bandwidth of the data channel from the camera to an external computer, which complicates the design and, accordingly, increases the cost of the entire system.

Disclosure of invention

The technical result of the claimed invention is the creation of a television camera and a method for forming a panoramic video image and object recognition on it with a simpler and cheaper design, through the use of a calculator integrated in the camera, which controls the movement of the pan-tilt platform, with a television camera installed on it, along a given route , while forming a single panoramic video image, and also identifies and recognizes objects on each frame of the video image.

The technical result achieved is achieved by creating a television camera for forming a panoramic video image and object recognition on it, containing an inclined-swivel platform on which a video sensor 1 is installed, the output of which is connected to the input of the image recognition unit 2 and to the input of the service information overlay unit 5, the output of which is connected with a video codec 9, the first and second outputs of which are connected respectively with the fourth input of the panoramic image forming unit 10 and with the first input of the pho block 12 the formation of video streams, the second input of which is connected to the first output of the panoramic image forming unit 10, and the output of the image recognition unit 2 is connected to the first input of the decision and metadata generating unit 6, the output of which is connected to the first input of the panoramic image forming unit 10, the second output of which is connected with the input of the block for generating alarms 13, the first, second, third, fourth and fifth outputs of the control unit 7 of the pan-tilt platform are connected respectively to the power transformer 4, in steps motors 3 of the pan-tilt platform, with the second input of the decision making and metadata generating unit 6, with the second input of the panoramic image forming unit 10 and with the first input of the communication unit 15, the second, third, fourth, fifth and sixth inputs of which are connected respectively to the third the output of the panoramic image forming unit 10, with the output the alarm generation unit 13, with the output of the video stream generation unit 12, with the output of the ONVIF support unit 14 and with the output of the WEB interface 16, the input of which is connected to the first output of the communication unit 15, the second input of which is connected to the output of the ONVIF support unit 14, and the third and fourth the outputs of the panoramic image forming unit 10 are connected respectively to the first input of the rotary support control unit 7 and to the input of the frame accumulating unit 11, the output of which is connected to the fourth input of the panoramic image forming unit 10 Ia, wherein

- the video sensor 1 is configured to form the main video stream and transmit it to block 5 overlay service information and

- block 2 recognition of images, which is configured to frame-by-frame detection of objects in the image, determining the type and coordinates relative to the frame of the detected objects and transmitting them to

- block 6 decision making and the formation of metadata, which is configured to analyze data received from block 2 image recognition, while deciding on the allocation or ignoring of the object, and in the case of a decision on the selection of the object, with the possibility of forming metadata in the coordinates of the panoramic image based on information about the current position (coordinates) of the pan-tilt platform received from the pan-tilt platform control unit 7, and with the possibility of transmitting the metadan stream s in block 10 forming the panoramic image, and

- block 5 overlay service information, which is configured to overlay frames on detected objects and other visual information about the object on the video frames from the input stream, while forming the output video stream and transmitting it to

- a video codec 9, which is configured to compress a video stream, while generating a plurality of compressed video streams with different resolutions, in accordance with a given mode of operation of the camera, and transmitting the compressed video streams to the video stream forming unit 12 and

- a panoramic image forming unit 10, which is configured to receive a compressed video stream, a metadata stream synchronized with them, information about the current position (coordinates) of the pan-tilt platform from the pan-tilt-platform control unit 7, and of the previous frame from the frame accumulation unit 11, and based on the obtained data with the possibility of gluing a panoramic image, eliminating duplication of metadata from neighboring frames and generating a command to control the pan-tilt platform to move along a given path, and transmit the panoramic image to block 12 the formation of video streams and in block 11 accumulation of frames, and thinned stream of metadata in block 13 of the formation of alarms and block 15 communications,

- block 11 accumulation of frames made with the possibility of buffering frames,

- block 12 the formation of video streams is made with the possibility of preparing compressed video streams for broadcast to the consumer, while converting them into video streams of the RTSP format and transmitting to block 15 communications,

- the block 13 generating alarms is configured to generate alarm messages based on the metadata received in it and predefined alarm lines and zones, and transmitting alarm messages to the communication unit 15,

- the control unit 7 of the pan-tilt platform is configured to receive information from the motion and stop sensor 8 and the command to control the pan-tilt platform and with the possibility of generating control signals of the stepper motors 3 of the pan-tilt platform, as well as zoom control signals 4, while transmitting information about the current position (coordinates) of the pan-tilt platform and zoom in block 6 decision and metadata generation, in block 10 of the panoramic image and in block 15 communications,

- ONVIF support unit 14 is configured to allow the camera to communicate through the communication unit 15 with external objects in accordance with the ONVIF standard,

- WEB interface 16 is configured to transmit external camera control commands,

- communication unit 15 is configured to connect and interact with external objects.

In a preferred embodiment of the camera, the alarm generation unit 13 is configured to generate alarm messages selected from a set of alarm messages: the object has crossed the line; the object appeared in the zone; the object disappeared in the zone; the object entered the zone; the object has left the zone; The object has been in the zone for too long.

The technical result achieved is also achieved by creating a method for forming a panoramic video image and object recognition on it, in which

- using the video sensor 1 form the main video stream and transmit it to the block 5 overlay service information and

- recognition unit 2, with which frame-by-frame detect objects in the image, determine the type and coordinates relative to the frame of the detected objects and transmit them to

- decision making and metadata generation unit 6, by which the data received from the image recognition unit 2 is analyzed, and a decision is made whether to allocate or ignore an object, and if a decision is made to select an object, metadata is generated in the coordinates of the panoramic image based on the information about the current position (coordinates) of the pan-tilt platform received from the pan-tilt platform control unit 7, and the metadata stream is transmitted to the panoramic image generating unit 10 and in

- block 5 overlay service information with which to impose frames on the detected objects and other visual information about the object on the video frames from the input stream, while forming the output video stream and transmit it to

- video codec 9, with the help of which the video stream is compressed, while compressed video streams with different resolutions are formed, in accordance with the specified camera operating mode, and the compressed video streams are transmitted to the video stream generation unit 12 and to

- a panoramic image forming unit 10, with the help of which compressed video streams, a metadata stream synchronized with them, information about the current position (coordinates) of the pan-and-tilt platform from the pan-and-tilt platform control unit 7, and the previous frame from the frame accumulating unit 11 are received, and Based on the received data, a panoramic image is “glued together”, metadata duplication from neighboring frames is eliminated, and the pan-tilt platform control commands are formed for its movement along a given th path, and transmit the panorama image in block 12 of the formation of video streams and in block 11 of the accumulation of frames, and the thinned stream of metadata in block 13 of the formation of alarms and block 15 communications,

- using block 11 accumulation frames buffer frames,

- using block 12 of the formation of video streams prepare compressed video streams for broadcast to the consumer, while converting them into video streams of the RTSP format and transmit to block 15 communications,

- using the block 13 for generating alarms, alarm messages are generated based on the metadata received in it and predefined alarm lines and zones, and the alarm messages are transmitted to the communication unit 15,

- using the control unit 7 of the pan-tilt platform receive information from the motion sensor 8 and stop and control commands of the pan-tilt platform and generate control signals of stepper motors 3 of the pan-tilt platform, as well as control signals of zoom 4, while transmitting information about the current the position (coordinates) of the pan-tilt platform and the zoom to decision making and metadata generating unit 6, to panoramic image forming unit 10 and to communication unit 15,

- using block ONVIF support 14 provide the interaction of the camera through block 15 communications with external objects in accordance with the ONVIF standard,

- using the WEB interface 16 transmit external camera control commands,

- using the block 15 communications carry out the connection and interaction with external objects.

In a preferred embodiment of the method, using the alarm generation unit 13, alarm messages are generated that are selected from the set of alarm messages: the object has crossed the line; the object appeared in the zone; the object disappeared in the zone; the object entered the zone; the object has left the zone; The object has been in the zone for too long.

Brief Description of the Drawings

For a better understanding of the claimed invention the following is a detailed description with the corresponding graphic materials. FIG. 1. General functional diagram of a television camera for forming a panoramic video image and object recognition on it, made according to the invention.

Items:

1 - video sensor;

2 - image recognition unit;

3 - stepper motors of an inclined rotary platform;

4 - zoom;

5 - block overlay service information;

6 - block decision making and the formation of metadata;

7 - control unit pan-tilt platform;

8 - motion and stop sensor;

9 - video codec;

10 - block forming a panoramic image;

11 - block accumulation of frames;

12 - block for the formation of video streams;

13 - block forming alarms;

14 - ONVIF support unit;

15 - communication unit;

16 - WEB interface;

17 - external objects;

18 - the main video stream;

19 - coordinates, relative to the frame, and the type of detected objects;

20 - control signals of stepper motors;

21 - zoom control signals;

22 - metadata;

23 - information about the current position (coordinates) of the pan-tilt platform;

24 - information from the motion sensor and stop;

25 - output video stream;

26 - metadata of objects;

27 - command control pan-tilt platform;

28 - the current position (coordinates) of the pan-tilt platform; 29 - compressed video streams;

30 - panoramic frame;

31 - previous frame;

32 - the current position (coordinates) of the pan-tilt platform;

33 - compressed video streams;

34 is a panoramic image;

35, 36 - metadata;

37 - video streams format RTPS;

38 - alarm messages;

39 - 43 - data block communication.

Consider briefly the functioning of the claimed camera for the formation of a panoramic video image (Fig. 1). The panoramic image forming unit 10 controls the movement of the pan-tilt platform to ensure its non-stop movement along a predetermined route. The remaining elements of the camera, located on an inclined-rotary platform, continuously form a video stream. Each frame of the generated video stream is transmitted to the image recognition unit 2, in which the detection and recognition of all objects falling into the frame are carried out. The thus obtained stream of metadata about the detected objects (type of object, its coordinates within the frame, detection time) is transmitted to the panoramic image forming unit 10.

In the block 10 for forming a panoramic image based on the data, the location from the pan-tilt platform embeds the received video frame at the appropriate place in the panoramic image, the coordinates of the recognized objects are converted from the coordinates within the frame to the coordinates within the general panorama.

The panoramic image thus formed, together with the flow of metadata about the recognized objects, is transmitted as one of the output video streams generated by the camera.

The implementation of the invention

Consider in more detail the functioning of the claimed camera for the formation of a panoramic video image (Fig. 1).

Using the video sensor 1 (Fig. 1) form the main video stream 18, which goes directly to block 2 image recognition and block 5 overlay service information.

The image recognition unit 2 operates on a single frame, which allows the detection of objects in the image obtained from a moving camera. The type and coordinates 19 relative to the frame of the detected objects are transmitted to the block 6 of decision making and the formation of metadata.

In block 6, the decision and the formation of metadata, based on the assessments of conformity of the object received from block 2 recognition of the image, a decision is made on the selection or ignoring the object. If a decision is made to allocate an object, based on information about the current position (coordinates) of the pan-tilt platform received from the pan-tilt platform control unit 7, metadata is generated in the coordinates of the panoramic image. The thus formed metadata stream 26 is supplied to the panoramic image forming unit 10 for embedding the metadata in the panoramic image. A similar stream of metadata 22 is passed to the block 5 overlay overhead information.

In the block 5 overlay service information, based on the received metadata 22, overlay frames on the detected objects and other visual information about the object on the video frames from the input main video stream 18, while forming the output video stream 25.

Using the video codec 9, the input video stream 25 is compressed and one or more video streams 33 are formed with different resolutions, in accordance with the specified operating mode of the camera. The compressed video streams 33 thus obtained are transmitted via the video codec 9 to the input of the video stream generating unit 12 (RTSP streaming) and to the input of the panoramic image forming unit 10.

The compressed video streams 29, the metadata stream 26 synchronized with it, information 28 about the current position (coordinates) of the pan-and-tilt platform and the previous frame 31 from the frame accumulating unit 11 are input to the panoramic image forming unit 10. Based on these data, in the panoramic image forming unit 11, the panoramic image is “glued”, metadata duplication from neighboring frames is eliminated, and the pan-tilt platform control commands 27 are formed for it movement along a given trajectory. The resulting panoramic image 34 is transmitted to the block 12 of the formation of video streams (RTSP streaming), the same frame of the panoramic image 30 is transmitted to the block 11 frame accumulation. The thinned stream of metadata 35, 36 is transmitted to the alarm generation unit 13 and the communication unit 15.

In block 11 accumulation of frames buffer the frames, for the subsequent formation of a single panoramic image frame.

In block 12 of the formation of video streams (RTSP streaming), compressed video streams 33 and 34 are prepared for broadcast to the consumer in the form of video streams 37 in the RTSP format (real time streaming protocol). Video streams 37 may be several, with different resolutions. Ready-made video streams of the RTPS 37 format are transmitted to the communication unit 15.

In the block 13 for generating alarms, the following types of alarm messages are generated on the basis of the metadata 35 and predefined alarm lines / zones received in it:

- the object crossed the line;

- the object appeared in the zone;

- the object disappeared in the zone;

- the object entered the zone;

- the object has left the zone;

- The object has been in the zone for too long.

The thus generated alarm message flow 38 is transmitted to the communication unit 15.

The input control unit 7 of the pan-tilt platform receives information 24 from the motion and stop sensor 8 and the command 27 to move the pan-tilt platform. Based on this information, in the control unit 7 of the pan-tilt platform, the control signals 20 for stepper motors 3 of the pan-tilt platform and signals 21 for controlling the zoom 4 are generated, and the information 23, 28 and 32 of the current the position of the pan-tilt platform and zoom.

With the help of block 14 support ONVIF (Open Network Video Interface Forum - an international industry organization that is engaged in the development of standardized protocols for the interaction of various equipment and software included in the security systems) provide interaction 39, 41 through block 15 communications 15 with external objects 17 in accordance with the ONVIF standard.

Using the WEB interface 16 transmit external commands 40, 42 control the camera.

Using block 15 communications provide connection and interaction 43 with external, relative to the camera, objects 17.

The claimed invention is a fully autonomous system implemented in the form of a camera mounted on a controlled pan-tilt platform, without using external computers to control the movement of the pan-tilt platform along a predetermined route, to identify and recognize objects on each frame of the video stream and to form a single panoramic frame in the process of movement of the inclined-rotary platform along the route. Such a solution can drastically reduce the requirements for computing power of computing devices and systems that control video surveillance systems or increase the number of simultaneously involved television cameras. In addition, the use of the claimed invention can reduce the requirements for quality and bandwidth of the data channel from the camera to the control computing device.

Although the embodiment described above has been set forth to illustrate the claimed invention, it is clear to those skilled in the art that various modifications, additions and substitutions are possible without departing from the scope and meaning of the claimed invention disclosed in the attached claims.

Claims

CLAIM

1. A television camera for forming a panoramic video image and object recognition on it, comprising an inclined-swivel platform on which a video sensor 1 is installed, the output of which is connected to the input of the image recognition unit 2 and to the input of the service information overlay unit 5, the output of which is connected to the video codec 9, the first and second outputs of which are connected respectively to the fourth input of the panoramic image forming unit 10 and to the first input of the video stream forming unit 12, the second input of which is connected to the output of the panoramic image forming unit 10, and the output of the image recognition unit 2 is connected to the first input of the decision and metadata generating unit 6, the output of which is connected to the first input of the panoramic image forming unit 10, the second output of which is connected to the input of the alarm generating unit 13, the first , the second, third, fourth and fifth outputs of the inclined-rotary platform control unit 7 are connected respectively to the power transformer 4, to the stepper motors 3 of the inclined-rotary platform, with the second the ode of decision making and metadata generation unit 6, with the second input of the panoramic image forming unit 10 and with the first input of the communication unit 15, the second, third, fourth, fifth and sixth inputs of which are connected respectively to the third output of the panoramic image forming unit 10, with the output of the unit 13 for generating alarms, with the output of block 12 for generating video streams, with the output of block 14 supporting ONVIF and with the output of WEB interface 16, the input of which is connected to the first output of block 15 of communications, the second input of which is connected to Odom support block 14 ONVIF, and the third and fourth forming unit 10 outputs the panoramic image are respectively connected to the first input of the control unit 7, supporting-turntable and accumulation input of the frame 11, whose output is connected to a fourth input of block 10 forming the panoramic image, and

- block 2 image recognition, which is made with the possibility frame-by-frame detection of objects in the image, determining the type and coordinates relative to the frame of detected objects and transferring them to

- block 6 decision making and the formation of metadata, which is configured to analyze data received from block 2 image recognition, while deciding on the allocation or ignoring of the object, and in the case of a decision on the selection of the object, with the possibility of forming metadata in the coordinates of the panoramic image based on the information about the current position (coordinates) of the pan-tilt platform received from the pan-tilt platform control unit 7, and with the possibility of transmitting the metadata stream ny in the block

10 forming a panoramic image and in

- a video codec 9, which is configured to compress a video stream, while generating a plurality of compressed video streams with different resolutions, in accordance with a predetermined camera operating mode, and transmitting the compressed video streams to a video stream generating unit 12 and to - a panoramic image generating unit 10, which is configured to the possibility of obtaining a compressed video stream, a metadata stream synchronized with them, information about the current position (coordinates) of the pan-tilt platform from the pan-tilt-turn control unit 7 and the previous frame from the frame accumulation unit 11, and based on the data obtained with the possibility of gluing a panoramic image, eliminating duplication of metadata from neighboring frames and generating a command to control the pan-tilt platform to move along a given path, and transmit a panoramic image to the block 12 for generating video streams and to the block 11 for accumulating frames, and the thinned stream of metadata to the block 13 for generating alarms and the block 15 for communications,

- block 12 the formation of video streams made with the possibility of preparation compressed video streams to broadcast to the consumer, while converting them into video streams of the RTSP format and transmitting them to the communication unit 15,

the control unit 7 of the pan-tilt platform is configured to receive information from the motion and stop sensor 8 and the command to control the pan-tilt platform and with the possibility of generating control signals of the stepper motors 3 of the pan-tilt platform, as well as control signals of the zoom 4, while transmitting information about the current position (coordinates) of the pan-tilt platform and zoom into decision making and metadata generating unit 6, into panoramic image generating unit 10 and in block 15 communications,

- WEB interface 16 is configured to transmit external camera control commands,

communication unit 15 is configured to connect and interact with external objects.

2. The camera according to claim 1, with the exception that the alarm generation unit 13 is configured to generate alarm messages selected from a set of alarm messages: the object has crossed the line; the object appeared in the zone; the object disappeared in the zone; the object entered the zone; the object has left the zone; The object has been in the zone for too long.

3. A method for forming a panoramic video image and recognition of objects on it, in which

- recognition unit 2, with which frame-by-frame detect objects in the image, determine the type and coordinates relative to the frame detected objects and pass them to

- a panoramic image forming unit 10, with the help of which compressed video streams, a metadata stream synchronized with them, information about the current position (coordinates) of the pan-and-tilt platform from the pan-and-tilt platform control unit 7, and a previous frame from the frame accumulation unit 11 are received, and Based on the data obtained, a panoramic image is “glued together”, metadata duplication from neighboring frames is eliminated, and the pan-tilt platform control commands are formed for its movement along a given th path, and transmitting the panorama image generating portion 12 streams and accumulating frame portion 11 and a thinned stream metadata generation unit 13 and alarm unit 15 communications,

- using block 11 accumulation frames buffer frames,

- using the block 13 of the formation of alarms generate alarm messages, on based on the metadata received in it and predefined alarm lines and zones, and transmit alarm messages to the communication unit 15,

- using the control unit 7 of the pan-tilt platform receive information from the motion sensor 8 and stop and control commands of the pan-tilt platform and generate control signals of stepper motors 3 of the pan-tilt platform, as well as control signals of zoom 4, while transmitting information about the current the position (coordinates) of the pan-tilt platform and the zoom in block 6 decision making and the formation of metadata, in block 10 of the formation of panoramic images and in block 15 of communications,

- using the WEB interface 16 transmit external camera control commands,

using block 15 communications carry out the connection and interaction with external objects.

4. The method according to p. 3, with the fact that using the block 13 generating alarms generate alarm messages selected from a set of alarm messages: the object crossed the line; the object appeared in the zone; the object disappeared in the zone; the object entered the zone; the object has left the zone; The object has been in the zone for too long.