WO2022052409A1 - 用于多机位摄像的自动控制方法和系统 - Google Patents

用于多机位摄像的自动控制方法和系统 Download PDF

Info

Publication number
WO2022052409A1
WO2022052409A1 PCT/CN2021/073537 CN2021073537W WO2022052409A1 WO 2022052409 A1 WO2022052409 A1 WO 2022052409A1 CN 2021073537 W CN2021073537 W CN 2021073537W WO 2022052409 A1 WO2022052409 A1 WO 2022052409A1
Authority
WO
WIPO (PCT)
Prior art keywords
wide
angle
camera
cameras
target object
Prior art date
Application number
PCT/CN2021/073537
Other languages
English (en)
French (fr)
Inventor
余飞
周昌印
Original Assignee
视见科技(杭州)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 视见科技(杭州)有限公司 filed Critical 视见科技(杭州)有限公司
Publication of WO2022052409A1 publication Critical patent/WO2022052409A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • the present application relates to the field of human-computer interaction, and more particularly, to the field of human-computer interaction for multi-camera photography.
  • the multi-camera system is generally composed of multiple cameras (usually operated by the cameraman).
  • the cameras are connected to a director station (usually operated by the director) through cables or wireless signals, and finally the director station outputs one or more channels. road signal.
  • a live broadcast or filming event is generally completed by a filming team.
  • Existing camera systems generally rely on 1) tacit understanding and rapid communication between the director and the cameraman; 2) the cameraman's quick operations (steering, aligning, zooming, focusing).
  • the multi-camera camera control system of the present invention aims to allow the director to quickly control multiple cameras independently without the cooperation of the cameraman.
  • PTZ Pan (horizontal movement)/Tilt (vertical movement)/Zoom (zoom).
  • a PTZ camera represents a camera with a motorized pan/tilt that can move in all directions (left/right/up and down), as well as lens zoom and zoom controls. Such cameras have been deployed in large numbers in security or filming.
  • the advantage of the PTZ camera compared to the traditional camera is that after obtaining the target with its own wide-angle advantage, it can obtain a magnified and clear target image by zooming and zooming, and control and track the target under certain conditions.
  • the traditional PTZ camera is controlled remotely by a multi-use remote control or a dedicated console.
  • Patent JP2012520650A It is about a very common system of wide-angle camera and PTZ camera. The position of the PTZ camera that needs close-up is obtained through the wide-angle camera, and then the close-up zoom of the PTZ camera is used.
  • Patent US20040263476A1 It is about using a joystick like a joystick to control the PTZ camera.
  • a quick viewfinder system for multi-camera photography comprising: a user interface configured to display two or more wide-angle cameras for multi-camera photography and a picture of the one or more PTZ cameras; and a central processing unit configured to receive information about each of the two or more wide-angle cameras displayed in the user interface (1) the position information of other wide-angle cameras on the wide-angle camera screen marked respectively on the screen of a wide-angle camera; and (2) the position information of all PTZ cameras on the wide-angle camera screen; The displayed position information of the target object or area marked on the screen of any two wide-angle cameras among the two or more wide-angle cameras on the screen of the wide-angle camera; determining the two The positional relationship between each of the multiple wide-angle cameras and the wide-angle camera, other PTZ cameras, or the target object or area, respectively; the level at which the PTZ camera needs to be rotated to aim at the target object or area is determined based on at least part of the determined positional relationship The moving angle and the
  • a fast view framing method for multi-camera photography comprising: receiving information on the following content: a camera in two or more wide-angle cameras displayed on a user interface (1) the positions of other wide-angle cameras on the wide-angle camera screen and (2) the position information of all PTZ cameras on the wide-angle camera screen respectively marked on the screen of each wide-angle camera; and (3) displayed on the user interface
  • the moving angle and the vertical moving angle are used to control the PTZ camera to roughly aim at the target
  • FIG. 1 is a schematic diagram of a scenario according to an embodiment of the present disclosure.
  • FIG. 2 is a schematic imaging screen of the wide-angle camera A in FIG. 1 according to an embodiment of the present disclosure.
  • FIG. 3 is a schematic imaging screen of the wide-angle camera B in FIG. 1 according to an embodiment of the present disclosure.
  • FIG. 4 shows a detailed method flow diagram according to an embodiment of the present disclosure.
  • Figure 5 is the correspondence between ⁇ and ⁇ in the equidistant cylindrical projection.
  • FIG. 6 is a schematic diagram of a scenario according to an embodiment of the present disclosure.
  • FIG. 1 is a schematic diagram of a scenario according to an embodiment of the present disclosure.
  • a system including a wide-angle camera A, a wide-angle camera B, and a PTZ camera C is shown.
  • three schematic objects are shown, namely cylindrical object 1 , cube object 2 and sphere object 3 .
  • cameras A and B see the scene from different angles, the imaging images of each object in the image in different cameras are very different.
  • FIG. 2 is a schematic imaging screen of the wide-angle camera A in FIG. 1 according to an embodiment of the present disclosure.
  • 3 is a schematic imaging picture of the wide-angle camera B in FIG. 1 according to an embodiment of the present disclosure.
  • the images of the objects in the pictures change greatly, and feature point matching is difficult.
  • the imaging pictures of PTZ cameras at different locations on the site and far apart will also be quite different.
  • the present invention is aimed at the above-mentioned scene. More specifically, the present invention needs to solve the problem of how to capture more images of the entire field when one or more PTZ cameras are required to shoot at different locations of the field when the area of the shooting site is large. The problem of framing the target object on the screen of a wide-angle camera, so that any PTZ camera in the field can automatically rotate and aim at the target object.
  • the present invention proposes a system and method for solving this problem in view of the above scenarios.
  • a quick viewfinder system for multi-camera photography comprising: a user interface configured to display two or more wide-angle cameras for multi-camera photography and all a picture of the one or more PTZ cameras; and a central processing unit configured to: receive information about each of the two or more wide-angle cameras displayed in the user interface (1) position information of other wide-angle cameras on the wide-angle camera screen marked on the screen of the wide-angle camera; and (2) position information of all PTZ cameras on the wide-angle camera screen; and (3) displayed on the user interface The position information of the target object or area marked on the screen of any two wide-angle cameras among the two or more wide-angle cameras on the screen of the wide-angle camera; determining the two or more based on at least part of the marked position information The positional relationship between each of the multiple wide-angle cameras and the wide-angle camera, other PTZ cameras, or the target object or area; the horizontal movement that needs to be rotated by the PTZ camera to aim at the target object or area is determined
  • the image processing algorithm includes a SIFT algorithm or an image recognition and matching algorithm.
  • the image processing algorithm includes an image processing algorithm that recognizes the target object or region and automatically adjusts the angle of the PTZ camera according to the characteristics of the target object or region so that it is precisely aligned with the target object.
  • the picture of the one or more wide-angle cameras displayed on the user interface is obtained by performing optical de-distortion and equidistant cylindrical projection on the picture captured by each of the one or more wide-angle cameras.
  • the marking process includes: marking each wide-angle projection picture.
  • the markers for other wide-angle cameras and all PTZ cameras are dot-like markers that indicate the position of the lens center of the marked camera in the wide-angle projection picture.
  • the marker includes a pattern drawn by the user on the user interface, and the central processing unit is further configured to form a minimum rectangular frame covering the pattern according to the pattern drawn by the user.
  • the positional relationship between each of the two or more wide-angle cameras and the remaining wide-angle cameras or PTZ cameras and the target object or area includes starting from the center of the lens surface of the wide-angle camera, perpendicular to the The angle between the normal of the lens surface and the ray projected from the center of the lens surface of each wide-angle camera to the center of the lens surface of each other wide-angle camera on the horizontal and vertical planes, respectively, from the center of the lens surface of the wide-angle camera. Starting point, the angle between the normal line perpendicular to the lens surface and the ray from the center of the lens surface of each wide-angle camera pointing to the center of the lens surface of each PTZ camera on the horizontal plane and the vertical plane respectively.
  • the marking process is done by the user on the user interface.
  • the central processing unit generates a guide pattern about the to-be-marked object in the next camera about to be marked based on the position information obtained by the user after the first marking of the to-be-marked object in a wide-angle camera
  • the guide pattern is sent to the user interface, and the user interface displays the guide pattern in relation to the object to be marked to assist the user in completing the marking process, wherein the object to be marked is any one, the one or more of the remaining wide-angle cameras Either of the PTZ cameras or the target object or area.
  • the central processing unit identifies and tracks the target object so as to update the guide pattern associated with the target object in real time.
  • the labeling process for other wide-angle cameras and all PTZ cameras is done automatically by the central processing unit.
  • the marking process for the other wide-angle cameras and all PTZ cameras is performed by the central processing unit based on automatic identification of objects placed in the vicinity of the two or more wide-angle cameras and the one or more PTZ cameras Recognition complete.
  • the central processing unit is configured to track the movement of the PTZ camera or the target object in real time and update the relative positional relationship between the PTZ camera or the target object and the wide-angle camera in real time when the PTZ camera or the target object moves.
  • a fast view framing method for multi-camera photography comprising: receiving information about: each of two or more wide-angle cameras displayed on a user interface (1) position information of other wide-angle cameras on the wide-angle camera screen and (2) position information of all PTZ cameras on the wide-angle camera screen respectively marked on the screen of a wide-angle camera; and (3) displayed on the user interface
  • the moving angle and the vertical moving angle are used to control the PTZ camera to roughly aim at the target object or area; and
  • the picture of the one or more wide-angle cameras displayed on the user interface is obtained by performing optical de-distortion and equidistant cylindrical projection on the picture captured by each of the one or more wide-angle cameras.
  • the markers for other wide-angle cameras and all PTZ cameras are dot-like markers that indicate the position of the lens center of the marked camera in the wide-angle projection picture.
  • the marking for the target object or area includes a pattern drawn by the user on the user interface, and wherein the method further includes forming a minimum rectangular frame covering the pattern according to the pattern drawn by the user.
  • the positional relationship between each of the two or more wide-angle cameras and the remaining wide-angle cameras or PTZ cameras and the target object or area includes starting from the center of the lens surface of the wide-angle camera, perpendicular to the The angle between the normal of the lens surface and the ray projected from the center of the lens surface of each wide-angle camera to the center of the lens surface of each other wide-angle camera on the horizontal plane and the vertical plane respectively. Starting from the center, the angle between the normal line perpendicular to the lens surface and the ray from the center of the lens surface of each wide-angle camera to the center of the lens surface of each PTZ camera on the horizontal plane and the vertical plane respectively.
  • the normal line that starts from the center of the camera lens surface and is perpendicular to the lens surface and the ray from the center of the lens surface of each wide-angle camera to the center of the rectangular frame corresponding to the target object or area is between the horizontal and vertical plane projections, respectively. Angle.
  • the marking process is done by the user on the user interface.
  • the method further comprises: generating a guide pattern about the to-be-marked object in the next camera about to be marked based on the position information obtained by the user after completing the first marking of the to-be-marked object in a wide-angle camera; and displaying the guide pattern in relation to the object to be marked, wherein the object to be marked is any of the remaining wide-angle cameras, any of the one or more PTZ cameras, or a target to assist the user in completing the marking process object or area.
  • the method further includes: when the object to be marked is a target object and the target object moves, recognizing and tracking the target object to update the guide pattern associated with the target object in real time.
  • the labeling process for other wide-angle cameras and all PTZ cameras does not require manual labeling.
  • the tagging process for other wide-angle cameras and all PTZ cameras is done based on automatic identification of identifiers placed on the two or more wide-angle cameras and the one or more PTZ camera attachments.
  • the PTZ camera or target object movement is tracked in real time and the relative positional relationship between the PTZ camera or target object and the wide-angle camera is updated in real time.
  • the system provided in this paper allows a director to quickly complete the camera position control and signal output at different locations of the venue.
  • the present invention realizes the fast positioning of a single person in a large scene with multiple PTZ cameras through UI/UX design and image algorithm.
  • a wide-angle camera and a PTZ camera are integrated in the structure, and the captured images of the wide-angle camera and the PTZ camera are displayed on the screen.
  • the control signal of the PTZ camera and the image processing algorithm are used to finally achieve fast and accurate positioning of the target object or area.
  • the present disclosure can calculate the angle that the PTZ camera needs to rotate to aim at the target object in the field without measuring the distance between the wide-angle cameras or between the wide-angle cameras and the PTZ camera.
  • the schematic architecture includes two or more wide-angle cameras, one or more PTZ cameras, and a fast viewfinder system for multi-camera photography.
  • the quick viewfinder system includes a central processing unit and a user interface. Wherein the user interface can display all wide-angle cameras and all PTZ cameras captured footage, and in one embodiment, the user interface can receive user input.
  • the overall operation process is generally divided into two parts:
  • Control link The user specifies the target object or area in the images captured by any two of the wide-angle cameras, and the central processing unit automatically calculates the control signals for all PTZ cameras and rotates the PTZ cameras and roughly aligns the target object, and then at least according to the image algorithm Adjust the precise alignment and automatically adjust the scale value size.
  • the indicia for the target object or area includes a pattern drawn by the user on the user interface.
  • FIG. 4 shows a detailed method flow diagram according to an embodiment of the present disclosure. This method is aimed at situations where each PTZ camera can be seen within the field of view of all wide-angle cameras in the architecture, and a target object or area can be seen within the field of view of at least two wide-angle cameras.
  • steps 101 and 102 are preparation links
  • steps 103 and 104 are control links.
  • step 101 the picture captured by the wide-angle camera is optically de-distorted, and then the de-distorted picture is converted into an equidistant cylindrical projection picture (hereinafter referred to as a wide-angle projection picture) through a standard computational vision method, and is projected at a wide angle.
  • a wide-angle projection picture an equidistant cylindrical projection picture
  • an XY coordinate system is established, and the center point of the wide-angle projection screen (corresponding to the center point of the imaging surface of the wide-angle camera lens) is the origin (0,0) of the coordinate system.
  • the line segment on it represents the angle formed by the actual object with respect to a fixed reference (for example, on a flat world map obtained by equirectangular projection, the horizontal or vertical line segment represents the clip of longitude or latitude Horn).
  • the horizontal field of view of the wide-angle camera is represented by w fov
  • the vertical field of view is represented by h fov
  • the width of the projection screen is represented by x max pixels
  • the height is represented by y max pixels.
  • ⁇ has nothing to do with the value of y
  • ⁇ has nothing to do with the value of x.
  • ⁇ and ⁇ can be calculated independently according to the values of x and y. This special property is determined by the characteristics of the equidistant cylindrical projection.
  • step 102 mark the position of the center of the lens surface of each of the remaining wide-angle cameras t in the wide-angle projection picture of each wide-angle camera k among all the wide-angle cameras, denoted as (x kt , y kt ), and then by (x kt , y kt ) kt , y kt ) calculates the ray starting from the lens surface center point (0, 0) in the wide-angle camera k, perpendicular to the imaging surface of the wide-angle camera k, and the ray starting from the camera center point and pointing to the center of the lens imaging surface of the wide-angle camera t The angle ( ⁇ kt , ⁇ kt ) formed by the projected rays on the horizontal plane and the projected rays on the vertical plane.
  • the pixel position of the center of the lens imaging surface of each PTZ camera p in all the PTZ cameras is marked in the wide-angle projection pictures of all the wide-angle cameras k, denoted as (x kp , y kp ). Then, by (x kp , y kp ), the ray starting from the center point (0, 0) of the imaging surface of the lens in the wide-angle camera k and perpendicular to the imaging surface of the wide-angle camera k and the ray from the center point of the camera pointing to the PTZ camera p are calculated.
  • the user For each target object or area, the user needs to mark at least two wide-angle cameras respectively. After the target object or area is marked in the first wide-angle projection image, it is often difficult for the user to find the corresponding target object or area in the second wide-angle projection image. So far, the following guided labeling methods are proposed to help users label target objects or regions in the remaining wide-angle camera projection images more quickly and accurately.
  • the user interface can mark the guide pattern (usually a line segment) relative to the possible position of the corresponding target object in all the remaining wide-angle projection pictures or an arc) to help users make reasonable annotations faster with reference to the possible location.
  • the guide pattern usually a line segment
  • the user interface marks a possible position corresponding to the position of the target object or area
  • the central processing unit may determine, based on the first wide-angle projection screen, that the center of the wide-angle camera points to a direction indicating the target object or area.
  • the ray from the center of the rectangular box is calculated.
  • the central processing unit may determine the guide pattern based on the line or arc ray projection corresponding to the center of the above-mentioned rectangular frame corresponding to the target object or area in the remaining wide-angle projection images, and send the guide image to the user interface so as to be displayed by the user interface.
  • the user interface is used to mark the location of the corresponding target object or area. In one embodiment, if the possible position marked in the second wide-angle projection picture is far away from the target object or area in the first wide-angle projection picture, it means that the labeling of the previous steps is inaccurate and needs to be relabeled.
  • the central processing unit will identify and track the target object in the picture, and update the above-mentioned rectangle corresponding to the target object or area in real time.
  • the center of the frame is at the pixel position in the wide-angle projection image, and the guide patterns in the other wide-angle projection images are updated at the same time, thereby assisting the entire marking process to be completed accurately.
  • the above guided labeling method can also be used for other labeling processes described above, such as the process of labeling the remaining wide-angle cameras on each wide-angle camera and the process of labeling all PTZ cameras on each wide-angle camera.
  • the above-described labeling process eg, labeling the remaining wide-angle cameras on each wide-angle camera and labeling all PTZ cameras on each wide-angle camera can be done automatically without user input.
  • the automatic labeling process can be implemented by placing a three-dimensional marker directly above the space of a wide-angle camera, a PTZ camera, etc., where the marker is used to identify each camera.
  • the marker may be a three-dimensional two-dimensional code, etc., and the marker for each camera is different.
  • the central processing unit will automatically find and identify markers in each wide-angle projection screen, automatically mark the positions of other cameras, and complete the entire preparation process without the need for users to make signs.
  • This method has another advantage, that is, the absolute distance before each camera can be estimated by judging the image size of the recognized object in the wide-angle projection screen (because the size of the recognized object is known), so that the PTZ camera to the final target can be completely calculated. The distance of the object or area, assists the PTZ camera to focus quickly.
  • step 103 the position of the target object or region is marked in the wide-angle projection images of any two wide-angle cameras (assuming these two cameras are k 1 , k 2 ) that can photograph the target object or region.
  • the location of the target object or area may be marked by user input regarding the target object or area.
  • the user's input regarding the target object or area may be that the user draws a pattern for the target object or area on the picture captured by the wide-angle camera or the PTZ camera of the touch screen console.
  • the user can draw with a finger, and the pattern can be an arc.
  • the user may use a stylus or other tool to draw.
  • the pattern may also be a circle, a regular rectangle or a line segment.
  • the central processing unit is further configured to form a minimum rectangular frame covering the pattern according to the pattern drawn by the user.
  • the rectangular boxes on the two obtained pictures have coordinates of The center point of , the width and height are and and Similarly, through the coordinates of the center point of the rectangular box and It can be obtained that the ray starting from the center point (0, 0) of the imaging surface of the lens in the wide-angle camera k and perpendicular to the imaging surface of the wide-angle camera k 1 , k 2 and the ray starting from the center point of the camera and pointing to the center point of the rectangular frame are in the horizontal plane.
  • step 104 a PTZ control signal is generated according to each positional relationship determined by the above-mentioned marking process, and the rotation of the PTZ camera is controlled so as to be roughly aligned with the target object or area, and then the PTZ is accurately aligned with the target object or area at least according to the image algorithm And automatically adjust the zoom value size.
  • the PTZ control signal includes a horizontal movement angle and a vertical movement angle of the PTZ camera rotation. Because the horizontal movement angle and vertical movement angle of the PTZ camera need to be calculated separately, but the calculation principle is exactly the same, so only the process of determining the horizontal movement angle is listed below.
  • FIG. 6 is a schematic diagram of a scene according to an embodiment of the present disclosure, which shows the angular relationship formed between a wide-angle camera projected on a horizontal plane and a PTZ camera, a target object or an area, wherein the dotted extension line is emitted from the center point of the camera lens surface A normal guide line that is perpendicular to the camera lens.
  • ⁇ 3 is the angle between the ray from the center of the lens surface of the wide-angle camera B to the center of the lens surface of the wide-angle camera A and the ray from the center of the lens surface of the wide-angle camera B to the center of the rectangular frame corresponding to the target object or area.
  • ⁇ 4 is the angle between the ray pointing from the center of the lens surface of the wide-angle camera B to the center of the PTZ camera screen and the ray pointing from the center of the lens surface of the wide-angle camera B to the center of the rectangular frame corresponding to the target object or area.
  • ⁇ 3 is the angle between the ray from the center of the lens surface of the wide-angle camera A to the center of the lens surface of the wide-angle camera B and the ray from the center of the lens surface of the wide-angle camera A to the center of the rectangular frame corresponding to the target object or area.
  • ⁇ 4 is the angle between the ray pointing from the center of the lens surface of the wide-angle camera A to the center of the PTZ camera screen and the ray pointing from the center of the lens surface of the wide-angle camera A to the center of the rectangular frame corresponding to the target object or area.
  • ⁇ 1 which is the ray from the center of the lens surface of the wide-angle camera B pointing to the center of the PTZ camera screen and the projection from the wide-angle camera.
  • the angle from the center of the picture is perpendicular to the normal of the camera lens surface, which is recorded as ⁇ 2 , which is the angle between the ray from the center of the lens surface of the wide-angle camera B pointing to the rectangular frame corresponding to the target object or area and the normal line starting from the center of the lens surface of the wide-angle camera and perpendicular to the lens surface of the camera.
  • ⁇ 1 is the angle between the ray from the center of the lens surface of the wide-angle camera A that points to the center of the PTZ camera screen and the normal line from the center of the lens surface of the wide-angle camera that is perpendicular to the lens surface of the camera.
  • ⁇ 2 is the angle between the ray from the center of the lens surface of the wide-angle camera A pointing to the center of the rectangular frame corresponding to the target object or area and the normal line perpendicular to the surface of the camera lens from the center of the lens surface of the wide-angle camera.
  • ⁇ 3 , ⁇ 4 , ⁇ 3 , ⁇ 4 can be calculated using known angles respectively:
  • ⁇ 1 can be further determined:
  • the initial orientation position of the PTZ camera In the actual operation process, in order to calculate the final rotation angle of the PTZ camera, it is also necessary to know the initial orientation position of the PTZ camera. In one embodiment, it is necessary to know the center point of the imaging surface of the PTZ camera lens, which is perpendicular to the imaging surface. The angle ⁇ 0 formed between the normal of , and the ray pointing from the center of its lens surface to the center of the imaging surface of one of the wide-angle camera lenses. In one embodiment, the angle can be obtained by measuring the rotation angle reading of the horizontal movement of the PTZ camera when the center of the PTZ camera image is aligned with one of the wide-angle cameras during installation. Therefore, in the scene shown in FIG. 6 , the rotation angle of the horizontal movement that needs to be performed by the PTZ camera to finally align the target object or region is ⁇ 0 + ⁇ 1 .
  • the PTZ camera can be aimed at another target object or area by re-operating the above control steps.
  • the horizontal movement angle of the PTZ camera rotation can be determined in the above manner and for the same reason, the vertical movement angle can also be determined to generate a PTZ control signal to control the PTZ camera to align the target object or area through the horizontal movement angle and the vertical movement angle.
  • the central processing unit estimates the proportion of the target object in the picture based on the proportion of the rectangular frame captured by the user in the wide-angle projection to the picture. After the PTZ camera receives the aspect ratio information, it will automatically adjust the Z value to zoom the image to the same ratio as the user's wide-angle projection.
  • the following describes how to control the movement of the PTZ camera when the target object moves.
  • the target object will move.
  • the central processing unit recognizes the target object in the wide-angle projection screen, it can continuously track the target object on the wide-angle projection screen, continuously output the relative position and angle of the target object in the above control link, and update the PTZ control signal in real time to control it to always Quasi-object.
  • the PTZ camera can be a movable PTZ camera, such as it can be mounted on a large robotic arm to photograph objects from different angles, so its spatial position will change.
  • the AI recognition and following algorithm can be used on the wide-angle projection screen to track the marked position of the PTZ camera on the wide-angle projection screen.
  • the center of the imaging surface of the PTZ camera lens is updated in real time.
  • the coordinate position of the wide-angle projection screen and then update the angle formed by it and the wide-angle camera in real time, and update the angle that the PTZ camera needs to rotate to align the target object or area at this position in real time, so that the PTZ camera is always accurately aimed at the target object or area.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)

Abstract

本文提供了一种用于多机位摄像自动控制的系统与方法。所述方法包括，基于用户手动或者系统自动在两个或多个广角相机上分别标记(1)其他广角相机在本广角相机投影画面上的位置(2)所有场地中PTZ相机在本广角相机投影画面上的位置；(3)目标物体或区域在本广角相机上的位置，在不需要测量广角相机之间，或者广角相机与PTZ相机之间的距离的情况下，计算出场地中PTZ相机对准目标物体需要旋转的角度。

Description

用于多机位摄像的自动控制方法和系统
本申请要求于2020年9月10日递交的第202010943829.0号中国专利申请的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分。
技术领域
本申请涉及人机交互领域，更具体地，涉及用于多机位摄像的人机交互领域。
背景技术
多机位摄像系统一般由多个机位的摄像机(一般由摄像师操作)组成，摄像机通过线缆或无线信号连接到一个导播台(一般由导播操作)，并最终由导播台输出一路或多路信号。一次直播或拍摄活动一般由一个摄制团队完成。其中如何将一台或多台摄像机快速对准一个特定目标是实际录制或直播过程中的一个关键问题。现有摄像系统一般依赖1)导播与摄像师之间的默契与快速交流；2)摄像师的快速操作(转向、对准、缩放、对焦)。本发明的多机位摄像机控制系统，旨在让导播在没有摄像师配合的情况下，独自快速操控多个摄像机。
PTZ是Pan(水平移动)/Tilt(垂直移动)/Zoom(变焦)的简写。PTZ相机代表带有电动云台的可以全方位(左右/上下)移动及镜头变倍、变焦控制的相机。这类相机已经在安防或者拍摄中大量部署。PTZ相机相比传统相机的优势是可以用自身广角优势取得目标后，通过缩放变焦获得放大清晰的目标图像，并且在一定条件下控制并且跟踪目标。传统的PTZ相机多用遥控器或者专用控制台手动远程控制。
已有许多关于PTZ相机的自动化控制的探索，如：
1.专利JP2012520650A:讲述的是一个很普遍的广角相机与PTZ相机的系统，通过广角相机取得PTZ相机需要特写的位置，然后由PTZ相机特写放大。
2.专利US20040263476A1:讲述的是用类似游戏控制杆来控制PTZ相机。
自动化控制PTZ相机的已有探索主要集中在用户通过单个广角相机画面选定目标物体或区域，系统自动算出相对应的单个PTZ相机的控制信号，并且大部分系统和方法需要广角相机与PTZ相机位置相近，以确保广角相机拍摄到的画面与PTZ相机拍摄的画面相近。对于广角相机与PTZ相机位置相对较远的情况并没有过多研究。而对同一目标物体从不同角度的拍摄因为控制复杂，多由人工完成。
计算机视觉中有经典的运动恢复结构算法(Structure from Motion,SfM) 可以实现在未知相机位置和目标位置的前提下，通过拍摄画面的相关性，同时计算出在一个同三维坐标系中相机的位置和目标的位置。此方法也可以用在这个场景解决PTZ相机需要对准目标物体的问题。不过此方法有几个缺陷:(1)SfM需要相机位置在空间上是较为连续的，对前后给出的两个相机的画面的重合特征点的个数和质量有要求。但在一般的拍摄场景中，相机的位置一般都是不动的，而且机位的个数有限，且可能相距较远。(2)从多个角度对同一个物体进行拍摄时，如果一个角度是正面，一个角度是侧面，则SfM无法准确找到相同的特征点，误差会增大并不可控。(3)SfM的计算量较大，无法适用于快速确定目标位置并控制相机。
发明内容
根据本发明的一个方面，提供了一种用于多机位摄像的快速取景系统，所述系统包括：用户界面，其被配置为显示用于多机位摄像的两个或多个广角相机以及所述一个或多个PTZ相机的画面；以及中心处理单元，所述中心处理单元被配置为：接收关于如下内容的信息：在用户界面所显示的所述两个或多个广角相机中的每一个广角相机的画面上分别标记的(1)其他广角相机在该广角相机画面上的位置信息；和(2)所有PTZ相机在该广角相机画面上的位置信息；以及(3)在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每一个分别与广角相机、其他PTZ相机或者目标物体或区域之间的位置关系；至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
根据本发明的另一方面，提供了一种用于多机位摄像的快速取景方法，所述方法包括：接收关于如下内容的信息：在用户界面所显示的两个或多个广角相机中的每一个广角相机的画面上分别标记的(1)其他广角相机在该广角相机画面上的位置和(2)所有PTZ相机在该广角相机画面上的位置信息；以及(3)在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每一个分别与其他广角相机、所有PTZ相机或者目标物体或区域之间的位置关系；至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
附图说明
图1是根据本公开实施例的场景示意图。
图2是根据本公开实施例的图1中的广角相机A的示意成像画面。
图3是根据本公开实施例的图1中的广角相机B的示意成像画面。
图4示出了根据本公开实施例的详细方法流程图。
图5是在等距柱状投影中的α与γ对应关系。
图6是根据本公开实施例的场景示意图。
具体实施方式
图1是根据本公开实施例的场景示意图。在图一所示场景中，示出了包括广角相机A、广角相机B和PTZ相机C的系统。在该场景中，示出了三个示意物体，即圆柱状物体1、正方体物体2和球体物体3。因为相机A与B看到场景的角度不同，从而导致图像中各物体在不同相机中的成像画面区别极大。图2是根据本公开实施例的图1中的广角相机A的示意成像画面。图3是根据本公开实施例的图1中的广角相机B的示意成像画面，对于广角相机A和广角相机B中的画面而言，画面中物体形象变化极大，特征点匹配困难。类似地，在场地不同位置且相距较远的PTZ相机的成像画面也会区别较大。
本发明即针对上述场景，更具体地，本发明需要解决是，在拍摄场地面积较大，需要一个或多个PTZ相机在场地不同位置进行拍摄的情况下，如何在能拍摄到整个场地的多个广角相机画面上框定目标物体，使得在场地中的任意PTZ相机能自动旋转并对准目标物体的问题。本发明针对上述场景提出解决该问题的系统和方法。
根据本发明的实施例，提供一种用于多机位摄像的快速取景系统，所述系统包括：用户界面，其被配置为显示用于多机位摄像的两个或多个广角相机以及所述一个或多个PTZ相机的画面；以及中心处理单元，所述中心处理单元被配置为：接收关于如下内容的信息：在用户界面所显示的所述两个或多个广角相机中的每一个广角相机的画面上分别标记的(1)其他广角相机在该广角相机画面上的位置信息；和(2)所有PTZ相机在该广角相机画面上的位置信息；以及(3)在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每 一个分别与广角相机、其他PTZ相机或者目标物体或区域之间的位置关系；至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
在一个实施例中，图像处理算法包括SIFT算法或者图像识别与匹配算法。在另一个实施例中，所述图像处理算法包括识别目标物体或区域并根据目标物体或区域的特点自动调整PTZ相机的角度以使其精确对准目标物体的图像处理算法。
在一个实施例中，用户界面所显示的所述一个或多个广角相机的画面是将所述一个或多个广角相机中的每个广角相机捕获的画面进行光学去畸变以及等距柱状投影而获得的广角投影画面。进一步地，所述标记过程包括：在每个广角投影画面上进行标记。
在一个实施例中，对于其他广角相机和所有PTZ相机的标记是指示被标记相机的镜头中心在广角投影画面中的位置的点状标记。
在一个实施例中，所述标记包括由用户在用户界面上绘制的图案，并且所述中心处理单元还被配置为根据用户绘制的图案形成覆盖所述图案的最小矩形框。
在一个实施例中，所述两个或多个广角相机中的每一个与其余广角相机或PTZ相机及目标物体或区域之间的位置关系包括从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向其余每个广角相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度、从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向每个PTZ相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度和从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向目标物体或区域所对应矩形框正中心的射线分别在水平面和垂直面投影产生的射线之间的角度。
在一个实施例中，标记过程由用户在用户界面上完成。
在一个实施例中，中心处理单元基于用户在一广角相机中完成对待标记 物的首次标记后获得的位置信息来在即将进行标记的下一相机中生成关于待标记物的引导图案并将所述引导图案发送给用户界面，用户界面与所述待标记物相关地显示所述引导图案以辅助用户完成标记过程，其中所述待标记物是其余广角相机中的任一个、所述一个或多个PTZ相机中的任一个或者目标物体或区域。
在进一步的实施例中，当待标记物是目标物体且目标物体移动时，中心处理单元对目标物体进行识别与跟踪从而实时更新与所述目标物体相关联的引导图案。
在一个实施例中，对其他广角相机和所有PTZ相机的标记过程由中心处理单元自动完成。
在进一步的实施例中，对其他广角相机和所有PTZ相机的标记过程由中心处理单元基于对于放置在所述两个或多个广角相机以及所述一个或多个PTZ相机附近的识别物的自动识别完成。
在一个实施例中，中心处理单元被配置为，当PTZ相机或目标物体移动时，对PTZ相机或目标物体移动进行实时跟踪并实时更新PTZ相机或者目标物体与广角相机的相对位置关系。
根据本发明的实施例，提供了一种用于多机位摄像的快速取景方法，所述方法包括：接收关于如下内容的信息：在用户界面所显示的两个或多个广角相机中的每一个广角相机的画面上分别标记的(1)其他广角相机在该广角相机画面上的位置信息和(2)所有PTZ相机在该广角相机画面上的位置信息；以及(3)在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每一个分别与其他广角相机、所有PTZ相机或者目标物体或区域之间的位置关系；至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
在一个实施例中，用户界面所显示的所述一个或多个广角相机的画面是将所述一个或多个广角相机中的每个广角相机捕获的画面进行光学去畸变以 及等距柱状投影而获得的广角投影画面，并且其中所述标记过程包括：在每个广角投影画面上进行标记。
在一个实施例中，对于其他广角相机和所有PTZ相机的标记是指示被标记相机的镜头中心在广角投影画面中的位置的点状标记。
在一个实施例中，对于目标物体或区域的标记包括由用户在用户界面上绘制的图案，并且其中，所述方法还包括根据用户绘制的图案形成覆盖所述图案的最小矩形框。
在进一步的实施例中，所述两个或多个广角相机中的每一个与其余广角相机或PTZ相机及目标物体或区域之间的位置关系包括从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向其余每个广角相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度、从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向每个PTZ相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度和从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向目标物体或区域所对应矩形框正中心的射线分别在水平面和垂直面投影产生的射线之间的角度。
在进一步的实施例中，标记过程由用户在用户界面上完成。
在一个实施例中，所述方法还包括：基于用户在一广角相机中完成对待标记物的首次标记后获得的位置信息来在即将进行标记的下一相机中生成关于待标记物的引导图案；以及与所述待标记物相关地显示所述引导图案以辅助用户完成标记过程，其中所述待标记物是其余广角相机中的任一个、所述一个或多个PTZ相机中的任一个或者目标物体或区域。
在一个实施例中，所述方法还包括：当待标记物是目标物体且目标物体移动时，对目标物体进行识别与跟踪从而实时更新与所述目标物体相关联的引导图案。
在一个实施例中，对其他广角相机和所有PTZ相机的标记过程无需手动标记。
在一个实施例中，对其他广角相机和所有PTZ相机的标记过程基于对于放置在所述两个或多个广角相机以及所述一个或多个PTZ相机附件的识别物 的自动识别完成。
在一个实施例中，当PTZ相机或目标物体移动时，PTZ相机或目标物体移动被实时跟踪并且PTZ相机或者目标物体与广角相机的相对位置关系被实时更新。
更具体地，本文提供的系统可以让一位导播快速完成在场地不同位置的机位控制与信号输出。本发明通过UI/UX设计与图像算法实现大场景的多PTZ相机的单人快速定位。在本发明中在架构中集成了广角相机与PTZ相机，将广角相机及PTZ相机的捕获画面显示在屏幕上，通过与用户的交互设计，快速确定目标区域，将用户交互转化为针对在不同位置的PTZ相机的控制信号，并利用图像处理算法，最终实现对目标物体或区域的快速及准确的定位。本公开可以在不需要测量广角相机之间、或者广角相机与PTZ相机之间的距离的情况下，计算出场地中PTZ相机对准目标物体需要旋转的角度。
根据本发明的一方面，本示意架构包括两个或多个广角相机，一个或多个PTZ相机以及用于多机位摄像的快速取景系统。该快速取景系统包括中心处理单元以及用户界面。其中用户界面可以显示所有广角相机和所有PTZ相机捕获的画面，并且在一个实施例中，用户界面可以接收用户输入。整体操作流程总的来说分为两个环节:
1.准备环节。在该环节中，通过在每个广角相机画面中标记(1)其余广角相机在该广角相机画面的像素位置；以及(2)PTZ相机在该广角相机画面的像素位置。对于其他广角相机和所有PTZ相机的标记是指示被标记相机的镜头中心在广角投影画面中的位置的点状标记。
2.控制环节。用户在其中任意两个广角相机捕获的画面中指定目标物体或区域，中心处理单元则会自动计算出针对所有PTZ相机的控制信号并将PTZ相机旋转并粗略对准目标物体，再至少根据图像算法调整精确对准并自动调整缩放值大小。在一个实施例中，对于目标物体或区域的标记包括由用户在用户界面上绘制的图案。
图4示出了根据本公开实施例的详细方法流程图。本方法针对如下情形：架构中的所有广角相机的视场内均可看到各PTZ相机，并且至少两个广角相机的视场内可看到目标物体或区域。下述步骤中，步骤101、102为准备环节，而步骤103和步骤104是控制环节。
首先，需要确定在广角相机画面上确定从该广角相机正中心指向目标中心的射线与自广角相机成像表面正中心出发的垂直于相机成像表面的法线的角度。
更具体地，在步骤101中，将广角相机捕获的画面进行光学去畸变，然后将去畸变的画面通过标准计算视觉方法转换为等距柱状投影画面(后简称广角投影画面)，并在广角投影画面上，建立X-Y坐标系，以广角投影画面中心点(对应于广角相机镜头成像表面中心点)为该坐标系原点(0,0)。
在等距柱状投影中，其上的线段代表实际物体相对于固定参照物所形成的角度(例如，在通过等距柱状投影获得的平面世界地图上，水平或者垂直的线段代表经度或者维度的夹角)。在本实施例中，将广角相机横向视场角表示为w fov，纵向视场角表示为h fov，投影画面宽度表示为x max像素，高度表示为y max像素。则广角相机的从中心点(0，0)出发、垂直于相机成像表面的射线与从该相机中心点出发指向其所成像的真实世界中任一点(该点在广角投影画面上成像的坐标值表示(x，y))的射线在水平面的投影射线以及在垂直面的投影射线形成的夹角(α，γ)(如图5中所示)分别为:
Figure PCTCN2021073537-appb-000001
注意，上式中，α与y值无关，γ与x值无关，α与γ可分别根据x,y值单独计算，该特殊性质由等距柱状投影的特性决定。
接下来将描述在每个广角相机画面中获得其他广角相机和所有PTZ相机的相对位置的步骤。
在步骤102中，在所有广角相机中的每一个广角相机k的广角投影画面中标记其余广角相机中的每一个t镜头表面中心的位置，记为(x kt，y kt)，再由(x kt，y kt)计算出从广角相机k中的镜头表面中心点(0，0)出发、垂直于广角相机k成像表面的射线与从该相机中心点出发指向广角相机t镜头成像表面中心的射线在水平面的投影射线以及在垂直面的投影射线形成的夹角(α kt，γ kt)。并且在所有的广角相机k中的广角投影画面中标记所有PTZ相机中的每一个PTZ相机p的镜头成像表面中心的像素位置，记为(x kp，y kp)。 再由(x kp，y kp)计算出从广角相机k中的镜头成像表面中心点(0，0)出发、垂直于广角相机k成像表面的射线与从该相机中心点出发指向PTZ相机p的镜头成像表面中心的射线在水平面的投影射线以及在垂直面的投影射线形成的夹角(α kp，γ kp)。
在该发明中，对于每个目标物体或区域，用户需要在至少两个广角相机中分别标出。在第一个广角投影画面里标出目标物体或区域以后，用户往往较难在第二个广角投影画面中找到对应目标物体或区域。为止，提出以下引导标注方法帮助用户更快更准地在其余广角相机投影画面中标注出目标物体或区域。
在用户在第一个广角投影画面中标记了目标物体或区域后，用户界面可以在所有其余的广角投影画面中标注出相对于相对应目标物体位置可能存在的位置的引导图案(一般为一条线段或一段弧线)，用以帮助用户参考该可能存在的位置更快的作出合理的标注。
在一个实施例中，用户界面标注出相对应目标物体或区域的位置可能存在的位置可以由中心处理单元基于在第一个广角投影画面上确定从该广角相机正中心指向指示目标物体或区域的矩形框的中心的射线来计算得到。具体地，中心处理单元可以基于目标物体或区域所对应的上述矩形框的中心在其余广角投影画面中对应的直线或弧线射线投影来确定引导图案，并将该引导图像发送给用户界面从而由用户界面来标注出相对应目标物体或区域的位置。在一个实施例中，如果在第二个广角投影画面中标出的可能位置距离第一个广角投影画面中的目标物体或区域较大，则说明之前的步骤标注不准确，需要重新标注。
在用户完成第一个标注之后，在完成第二个标注之前，目标物体移动将导致地位精度降低，甚至错误。在又一个实施例中，在每个广角投影画面中进行控制环节的目标物标注后，中心处理单元会在该画面中对目标物体进行识别与跟踪，实时更新目标物体或区域所对应的上述矩形框的中心在该广角投影画面中的像素位置，同时更新其余广角投影画面中的引导图案，进而辅助整个标记过程能精准完成。
以上引导标注方法也可以用于前述其它标注过程，例如在每个广角相机 上标记其余广角相机的过程以及在每个广角相机上标注所有PTZ相机的过程。
在另一个实施例中，上述标注过程例如在每个广角相机上标记其余广角相机的过程以及在每个广角相机上标注所有PTZ相机的过程可以在没有用户输入的情况下自动完成。具体地，自动标注过程可以通过在广角相机、PTZ相机等空间正上方放置一个立体的标志物来实现，所述标志物用于标识各个相机。具体地，标志物可以为立体面二维码等，每个相机的标志物不同。系统启动后，中心处理单元会自动在每个广角投影画面中寻找并识别画面中的标志物，自动标记其他相机的位置，完成整个准备环节，而不需要用户做标志。此方法还有另一个好处，即可以在广角投影画面中通过判断识别物的图像大小(因为识别物尺寸已知)，估算出各个相机之前的绝对距离，从而可以完整计算出PTZ相机到最终目标物体或区域的距离，辅助PTZ相机进行快速对焦。在步骤103中，在可拍摄到目标物体或区域的任意两个广角相机(假设这两个相机是k 1，k 2)的广角投影画面中标记目标物体或区域的位置。
在一个实施例中，可以通过用户关于目标物体或区域的输入来标记目标物体或区域的位置。用户关于目标物体或区域的输入可以是用户在触屏控制台的广角相机或PTZ相机捕获的画面上针对目标物体或区域绘制图案。用户可以使用手指来进行绘制，所述图案可以是一段弧。在其它实施例中，用户可以使用触控笔或其它工具来进行绘制。在其它实施例中，所述图案还可以是圆圈、规则的矩形或者一段线段。
中心处理单元还被配置为根据用户绘制的图案形成覆盖所述图案的最小矩形框。所得到的两个画面上的矩形框分别具有坐标为
Figure PCTCN2021073537-appb-000002
Figure PCTCN2021073537-appb-000003
的中心点，宽高分别为
Figure PCTCN2021073537-appb-000004
和以及
Figure PCTCN2021073537-appb-000005
类似地，通过矩形框的中心点坐标
Figure PCTCN2021073537-appb-000006
与
Figure PCTCN2021073537-appb-000007
可以得到从广角相机k中的镜头成像表面中心点(0，0)出发、垂直于广角相机k 1，k 2成像表面的射线与从所述相机中心点出发指向矩形框中心点的射线在水平面的投影射线以及在垂直面的投影射线形成的夹角
Figure PCTCN2021073537-appb-000008
与
Figure PCTCN2021073537-appb-000009
在步骤104中，根据通过上述标记过程确定的各位置关系生成PTZ控制 信号进而控制PTZ相机的旋转从而使其粗略对准目标物体或区域，再至少根据图像算法使PTZ精确对准目标物体或区域并自动调整缩放值大小。
下面描述如何进行PTZ控制信号的计算。
在一个实施例中，PTZ控制信号包括PTZ相机旋转的水平移动角度和垂直移动角度。因为PTZ相机需要旋转的水平移动角度和垂直移动角度分别计算但是所采用的计算原理完全一样，因此以下只列出水平移动角度的确定过程。
图6是根据本公开实施例的场景示意图，其中示出了在水平面投影的广角相机与PTZ相机、目标物体或区域之间形成的角度关系，其中虚线延长线为从相机镜头表面中心点射出的与相机镜头成面垂直的法线辅助线。
其中α 3为自广角相机B镜头表面中心指向广角相机A镜头表面正中心的射线与自广角相机B镜头表面正中心指向目标物体或区域所对应矩形框中心的射线的夹角。
α 4为自广角相机B镜头表面正中心指向PTZ相机画面正中心的射线与自广角相机B镜头表面正中心指向目标物体或区域所对应矩形框中心的射线的夹角。
θ 3为自广角相机A镜头表面正中心指向广角相机B镜头表面正中心的射线与自广角相机A镜头表面正中心指向目标物体或区域所对应矩形框中心的射线的夹角。
θ 4为自广角相机A镜头表面正中心指向PTZ相机画面正中心的射线与自广角相机A镜头表面正中心指向目标物体或区域所对应矩形框中心的射线的夹角。
以图6中所示的场景示意图为例，通过上述准备环节和控制环节，可以得到以下角度：α 1，其为广角相机B镜头表面正中心指向PTZ相机画面正中心的射线与自广角相机投影画面正中心出发的垂直于该相机镜头表面的法线的角度，在准备环节中其被记为
Figure PCTCN2021073537-appb-000010
α 2，其为广角相机B镜头表面正中心指向目标物体或区域所对应矩形框的射线与自广角相机镜头表面正中心出发的垂直于该相机镜头表面的法线的角度，在准备环节中其被记为
Figure PCTCN2021073537-appb-000011
θ 1，其为广角相机A镜头表面正中心指向PTZ相机画面正中心的射线与自广角相 机镜头表面正中心出发的垂直于该相机镜头表面的法线的角度，在准备环节中其被记为
Figure PCTCN2021073537-appb-000012
以及θ 2，其为广角相机A镜头表面正中心指向目标物体或区域所对应矩形框正中心的射线与自广角相机镜头表面正中心出发的垂直于该相机镜头表面的法线的角度，在准备环节中其被记为
Figure PCTCN2021073537-appb-000013
另外，
Figure PCTCN2021073537-appb-000014
分别为在对应广角相机镜头表面正中心指向另一个广角相机的射线与自该广角相机镜头表面正中心出发的垂直于该相机投影画面的法线的角度。以上等同关系示出如下：
Figure PCTCN2021073537-appb-000015
Figure PCTCN2021073537-appb-000016
因此α 3，α 4，θ 3，θ 4可以用已知角分别求出：
Figure PCTCN2021073537-appb-000017
α 4＝α 1+α 2
Figure PCTCN2021073537-appb-000018
θ 4＝θ 1+θ 2
根据包括正弦定理、和差角公式等的三角函数公式，可以进一步确定β 1:
Figure PCTCN2021073537-appb-000019
其中:
Figure PCTCN2021073537-appb-000020
在实际操作过程中，为了算出最终的PTZ相机需要旋转的角度，还需要知道PTZ相机的初始朝向位置，在一个实施例中，需要知道自PTZ相机镜头成像表面中心点出发、垂直于该成像表面的法线与从其镜头表面正中心指向其中一个广角相机镜头成像表面正中心的射线之间形成的夹角β 0。在一个实施例中，该角度可以通过在安装时测量出PTZ相机画面中心对准其中一个广角相机时，PTZ相机的水平移动旋转角度读数得到。因而在图6所示的场景中，PTZ相机要最终对准目标物体或区域需要进行的水平移动旋转角度则为β 0+β 1。
另外，在不移动广角相机和PTZ相机位置的前提下，只需要重新操作以上的控制环节的步骤，则可以使PTZ相机对准另外的目标物体或区域。
通过以上方式可以确定PTZ相机旋转的水平移动角度并且基于同样的原因，垂直移动角度也可以被确定从而生成PTZ控制信号来控制PTZ相机通过水平移动角度和垂直移动角度对准目标物体或区域。
关于缩放值的计算，可以通过图像处理算法。在一个实施例中，中心处理单元基于用户在广角投影中截取的矩形框的大小占画面的比例大小来估算出目标物体占画面的比例大小。PTZ相机接受到画面比例信息后，会自动调整Z值大小将画面缩放为与用户在广角投影上相同的比例。
下面描述当目标物体移动时，如何控制PTZ相机的移动。
在某一些场景中，目标物体会移动。中央处理单元在广角投影画面中识别目标物体后，可以在广角投影画面上持续跟踪目标物体，持续输出上述控制环节中的目标物体的相对位置及角度，并实时更新PTZ控制信号从而控制其始终对准物体。
在另一些场景中，PTZ相机可为机位可移动的PTZ相机，如其可安装在大型机械臂上从而从不同角度拍摄物体，因此其空间位置会发生变化。对于这种情况中，可以在广角投影画面上使用AI识别与跟随算法跟踪已经标注出的PTZ相机在广角投影画面上的位置，在PTZ相机位置改变时，实时更新PTZ相机镜头成像表面正中心在广角投影画面的坐标位置，进而实时更新其与广角相机形成的角度，并实时更新PTZ相机在该位置要对准目标物体或区域需要旋转的角度，使PTZ相机始终精确对准目标物体或区域。

Claims (22)

  1. 一种用于多机位摄像的快速取景系统，所述系统包括：
    用户界面，其被配置为显示用于多机位摄像的两个或多个广角相机以及一个或多个PTZ相机的画面；和
    中心处理单元，其被配置为：
    接收关于如下内容的信息：在用户界面所显示的所述两个或多个广角相机中的每一个广角相机的画面上分别标记的其他广角相机在该广角相机画面上的位置信息；所有PTZ相机在该广角相机画面上的位置信息；以及在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；
    至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每一个分别与广角相机、其他PTZ相机或者目标物体或区域之间的位置关系；
    至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及
    基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
  2. 如权利要求1所述的系统，其中，用户界面所显示的所述一个或多个广角相机的画面是将所述一个或多个广角相机中的每个广角相机捕获的画面进行光学去畸变以及等距柱状投影而获得的广角投影画面，并且
    其中标记过程包括：
    在每个广角投影画面上进行标记。
  3. 如权利要求1或2所述的系统，其中对于其他广角相机和所有PTZ相机的标记是指示被标记相机的镜头中心在广角投影画面中的位置的点状标记。
  4. 如权利要求1所述的系统，其中，对于目标物体或区域的标记包括由用户在用户界面上绘制的图案，并且
    其中，中心处理单元还被配置为根据用户绘制的图案形成覆盖所述图案的最小矩形框。
  5. 如权利要求4所述的系统，其中所述两个或多个广角相机中的每一个与其余广角相机或PTZ相机及目标物体或区域之间的位置关系包括从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向其余每个广角相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度、从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向每个PTZ相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度和从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向目标物体或区域所对应矩形框正中心的射线分别在水平面和垂直面投影产生的射线之间的角度。
  6. 如权利要求1所述的系统，其中标记过程由用户在用户界面上完成。
  7. 如权利要求6所述的系统，其中中心处理单元基于用户在一广角相机中完成对待标记物的首次标记后获得的位置信息来在即将进行标记的下一相机中生成关于待标记物的引导图案并将所述引导图案发送给用户界面，用户界面与所述待标记物相关地显示所述引导图案以辅助用户完成标记过程，其中所述待标记物是其余广角相机中的任一个、所述一个或多个PTZ相机中的任一个或者目标物体或区域。
  8. 如权利要求7所述的系统，其中，当待标记物是目标物体且目标物体移动时，中心处理单元对目标物体进行识别与跟踪从而实时更新与所述目标物体相关联的引导图案。
  9. 如权利要求1所述的系统，其中对其他广角相机和所有PTZ相机的标记过程由中心处理单元自动完成。
  10. 如权利要求9所述的系统，其中对其他广角相机和所有PTZ相机的标记过程由中心处理单元基于对于放置在所述两个或多个广角相机以及所述一个或多个PTZ相机附近的识别物的自动识别完成。
  11. 如权利要求1或5所述的系统，其中，中心处理单元被配置为，当PTZ相机或目标物体移动时，对PTZ相机或目标物体移动进行实时跟踪并实时更新PTZ相机或者目标物体与广角相机的相对位置关系。
  12. 一种用于多机位摄像的快速取景方法，所述方法包括：
    接收关于如下内容的信息：在用户界面所显示的两个或多个广角相机中 的每一个广角相机的画面上分别标记的其他广角相机在该广角相机画面上的位置信息；所有PTZ相机在该广角相机画面上的位置信息；以及在用户界面所显示的所述两个或多个广角相机中的任意两个广角相机的画面上标记的目标物体或区域在该广角相机画面上的位置信息；
    至少基于所标记的部分位置信息确定所述两个或多个广角相机中的每一个分别与其他广角相机、所有PTZ相机或者目标物体或区域之间的位置关系；
    至少基于所确定的部分位置关系来确定PTZ相机对准目标物体或区域需要旋转的水平移动角度和垂直移动角度以控制PTZ相机粗略对准目标物体或区域；以及
    基于图像处理算法来确定PTZ相机的缩放大小并控制PTZ相机精确对准目标物体或区域。
  13. 如权利要求12所述的方法，其中，用户界面所显示的所述一个或多个广角相机的画面是将所述一个或多个广角相机中的每个广角相机捕获的画面进行光学去畸变以及等距柱状投影而获得的广角投影画面，并且
    其中标记过程包括：
    在每个广角投影画面上进行标记。
  14. 如权利要求12或13所述的方法，其中对于其他广角相机和所有PTZ相机的标记是指示被标记相机的镜头中心在广角投影画面中的位置的点状标记。
  15. 如权利要求12所述的方法，其中，对于目标物体或区域的标记包括由用户在用户界面上绘制的图案，并且
    其中，所述方法还包括根据用户绘制的图案形成覆盖所述图案的最小矩形框。
  16. 如权利要求15所述的方法，其中所述两个或多个广角相机中的每一个与其余广角相机或PTZ相机及目标物体或区域之间的位置关系包括从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向其余每个广角相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度、从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向每个PTZ相机镜头表面正中心的射线分别在水平面和垂直面投影产生的射线之间的角度和 从该广角相机镜头表面正中心出发、垂直于该镜头表面的法线与自每个广角相机镜头表面正中心指向目标物体或区域所对应矩形框正中心的射线分别在水平面和垂直面投影产生的射线之间的角度。
  17. 如权利要求12所述的方法，其中标记过程由用户在用户界面上完成。
  18. 如权利要求17所述的方法，所述方法还包括：
    基于用户在一广角相机中完成对待标记物的首次标记后获得的位置信息来在即将进行标记的下一相机中生成关于待标记物的引导图案；以及
    与所述待标记物相关地显示所述引导图案以辅助用户完成标记过程，其中所述待标记物是其余广角相机中的任一个、所述一个或多个PTZ相机中的任一个或者目标物体或区域。
  19. 如权利要求18所述的方法，其中，所述方法还包括：当待标记物是目标物体且目标物体移动时，对目标物体进行识别与跟踪从而实时更新与所述目标物体相关联的引导图案。
  20. 如权利要求12所述的方法，其中对其他广角相机和所有PTZ相机的标记过程无需手动标记。
  21. 如权利要求19所述的方法，其中对其他广角相机和所有PTZ相机的标记过程基于对于放置在所述两个或多个广角相机以及所述一个或多个PTZ相机附件的识别物的自动识别完成。
  22. 如权利要求12或16所述的方法，其中，当PTZ相机或目标物体移动时，PTZ相机或目标物体移动被实时跟踪并且PTZ相机或者目标物体与广角相机的相对位置关系被实时更新。
PCT/CN2021/073537 2020-09-10 2021-01-25 用于多机位摄像的自动控制方法和系统 WO2022052409A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010943829.0 2020-09-10
CN202010943829.0A CN111818270B (zh) 2020-09-10 2020-09-10 用于多机位摄像的自动控制方法和系统

Publications (1)

Publication Number Publication Date
WO2022052409A1 true WO2022052409A1 (zh) 2022-03-17

Family

ID=72860761

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073537 WO2022052409A1 (zh) 2020-09-10 2021-01-25 用于多机位摄像的自动控制方法和系统

Country Status (2)

Country Link
CN (1) CN111818270B (zh)
WO (1) WO2022052409A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111818270B (zh) * 2020-09-10 2021-02-19 视见科技(杭州)有限公司 用于多机位摄像的自动控制方法和系统
CN112672064B (zh) * 2021-03-18 2021-07-20 视云融聚（广州）科技有限公司 一种基于视频区域标签的算法调度方法、系统及设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100097472A1 (en) * 2008-10-21 2010-04-22 Honeywell International Inc. Method of efficient camera control and hand over in surveillance management
CN101707671A (zh) * 2009-11-30 2010-05-12 杭州普维光电技术有限公司 一种全景摄像机与ptz摄像机联动控制方法和装置
CN102148965A (zh) * 2011-05-09 2011-08-10 上海芯启电子科技有限公司 多目标跟踪特写拍摄视频监控系统
CN102231799A (zh) * 2011-06-29 2011-11-02 上海芯启电子科技有限公司 一种ptz摄像机瞬时光学就位系统及其方法
CN111242988A (zh) * 2020-01-14 2020-06-05 青岛联合创智科技有限公司 一种广角相机与长焦相机联动双云台跟踪目标的方法
CN111818270A (zh) * 2020-09-10 2020-10-23 视见科技(杭州)有限公司 用于多机位摄像的自动控制方法和系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102013217223A1 (de) * 2013-08-29 2015-03-05 Robert Bosch Gmbh Überwachungsanlage sowie Verfahren zur Darstellung eines Überwachungsbereichs
JP2017103619A (ja) * 2015-12-02 2017-06-08 ソニー株式会社 制御装置、制御方法、およびプログラム
US10950003B2 (en) * 2018-03-29 2021-03-16 Pelco, Inc. Method of aligning two separated cameras matching points in the view
KR101996907B1 (ko) * 2018-07-27 2019-07-08 비티에스 유한회사 객체 추적 장치

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100097472A1 (en) * 2008-10-21 2010-04-22 Honeywell International Inc. Method of efficient camera control and hand over in surveillance management
CN101707671A (zh) * 2009-11-30 2010-05-12 杭州普维光电技术有限公司 一种全景摄像机与ptz摄像机联动控制方法和装置
CN102148965A (zh) * 2011-05-09 2011-08-10 上海芯启电子科技有限公司 多目标跟踪特写拍摄视频监控系统
CN102231799A (zh) * 2011-06-29 2011-11-02 上海芯启电子科技有限公司 一种ptz摄像机瞬时光学就位系统及其方法
CN111242988A (zh) * 2020-01-14 2020-06-05 青岛联合创智科技有限公司 一种广角相机与长焦相机联动双云台跟踪目标的方法
CN111818270A (zh) * 2020-09-10 2020-10-23 视见科技(杭州)有限公司 用于多机位摄像的自动控制方法和系统

Also Published As

Publication number Publication date
CN111818270A (zh) 2020-10-23
CN111818270B (zh) 2021-02-19

Similar Documents

Publication Publication Date Title
JP5740884B2 (ja) 繰り返し撮影用ａｒナビゲーション及び差異抽出のシステム、方法及びプログラム
CN112444242B (zh) 一种位姿优化方法及装置
USRE43700E1 (en) Virtual reality camera
CN111199560B (zh) 一种视频监控的定位方法及视频监控系统
JP4095491B2 (ja) 距離測定装置、距離測定方法、及び距離測定プログラム
CN108932051B (zh) 增强现实图像处理方法、装置及存储介质
US20150035857A1 (en) Methods and apparatus for generating composite images
WO2022052409A1 (zh) 用于多机位摄像的自动控制方法和系统
JP2007192832A (ja) 魚眼カメラの校正方法。
JP2010219825A (ja) 三次元計測用画像撮影装置
CN110146030A (zh) 基于棋盘格标志法的边坡表面变形监测系统和方法
CN111935410B (zh) 用于多机位摄像的快速取景方法和系统
CN107038714A (zh) 多型视觉传感协同目标跟踪方法
JP2006352495A (ja) 遠隔指示システム
CN113124883A (zh) 基于3d全景相机的离线标点方法
KR20100060472A (ko) 위치 측정 방법 및 이를 이용한 휴대 단말기
KR102389762B1 (ko) 디지털 트윈 연동 증강현실 카메라를 통한 공간 형성 및 인식 시스템 및 그 방법
CN111783849B (zh) 一种室内定位方法、装置、电子设备及存储介质
WO2019100216A1 (zh) 3d建模方法、电子设备、存储介质及程序产品
WO2022228321A1 (zh) 视频大范围内物体识别定位方法及装置
TWI479857B (zh) PTZ camera automatic tracking method
JPH09329440A (ja) 複数枚の画像の各計測点の対応づけ方法
JP2005266520A (ja) 撮像装置および撮像方法
WO2022036512A1 (zh) 数据处理方法、装置、终端和存储介质
JPH11194027A (ja) 三次元座標計測装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21865473

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21865473

Country of ref document: EP

Kind code of ref document: A1