KR102331534B1 - Camera system and camera controlling method - Google Patents

Abstract

According to an embodiment of the present invention, a first camera for photographing a subject; a second camera photographing the subject and spaced apart from the first camera by a predetermined distance; a first driving unit and a second driving unit for panning and/or tilting the first camera and the second camera, respectively; and a controller for controlling the first and second drivers, wherein the controller includes: an object distance estimator for estimating a distance between the first camera and the subject; and a first angle estimator for estimating a first panning and/or tilting angle of the first camera from the distance estimated by the object distance estimator.

Description

Camera system and camera controlling method

The present invention relates to a camera system and a camera control method.

Recently, complementary information fusion research using two or more sensors has been conducted to improve the accuracy and reliability of algorithms such as change detection, motion detection, super-resolution image restoration, and object recognition and tracking in the fields of surveillance systems and medical imaging. is being actively carried out.

In an image information matching system using two or more sensors, since the sensors are physically spaced apart from each other, the photographed areas may not completely match.

Recently, research on an image processing method for matching photographed images is being actively conducted, but research for matching areas photographed by a plurality of sensors is nonexistent.

An embodiment of the present invention provides a camera system capable of increasing the area of a region where images captured by a plurality of cameras overlap, and a camera control method using the same.

According to an embodiment of the present invention, a first camera for photographing a subject; a second camera photographing the subject and spaced apart from the first camera by a predetermined distance; a first driving unit and a second driving unit for panning and/or tilting the first camera and the second camera, respectively; and a controller for controlling the first and second drivers, wherein the controller includes: an object distance estimator for estimating a distance between the first camera and the subject; and a first angle estimator for estimating a first panning and/or tilting angle of the first camera from the distance estimated by the object distance estimator.

In this embodiment, the control unit, the center coordinates of the image captured by the first camera, and the center coordinates of a region overlapping the image captured by the second camera among the images captured by the first camera a control vector estimator for estimating a vector between; an angle of view estimator for estimating an angle of view from pre-stored zoom track information of the first camera; and a second angle estimator for estimating a second panning and/or tilting angle from the vector estimated by the control vector estimator and the angle of view estimated by the angle of view estimator.

In this embodiment, the method further includes an image processing unit that processes the images photographed by the first camera and the second camera, wherein the image processing unit detects first feature points of the image photographed by the first camera a first feature point detection unit; a second feature point detector for detecting second feature points of the image captured by the second camera; a first feature point center coordinate estimator for estimating center coordinates of the first feature points; a matching point detection unit detecting matching points in which the first feature points and the second feature points overlap; and a matching point center coordinate estimator for estimating center coordinates of the matching points, wherein the control vector estimator may estimate a vector between the center coordinates of the first feature points and the center coordinates of the matching points.

In the present embodiment, the first camera includes a first panning and/or tilting angle estimated by the first angle estimator and a second panning estimated by the second angle estimator by the first driving unit. and/or it may be panned and/or tilted by an angle calculated by the following equation from the tilting angle.

<expression>

θ=a×θ ₁ + (1-a)×θ ₂

Here, θ ₁ is a first panning and/or tilting angle, θ ₂ is a second panning and/or tilting angle, and a is a constant of 0 or more and 1 or less.

In this embodiment, the first camera and the second camera may be a visible camera and a thermal camera, respectively.

According to another embodiment of the present invention, in the method of controlling a camera using a camera system, the camera system includes a first camera and a second camera, the first camera and the and a first driving unit and a second driving unit for panning and/or tilting each of a second camera, wherein the camera control method includes: estimating a distance between the first camera and the subject by an object distance estimator; and estimating, by a first angle estimator, a first panning and/or tilting angle of the first camera from the distance estimated by the object distance estimator; A vector between the center coordinates of the image captured by the first camera and the center coordinates of a region overlapping the image captured by the second camera among the images captured by the first camera by the control vector estimator estimating; estimating, by an angle of view estimator, an angle of view from pre-stored zoom track information of the first camera; and estimating, by a second angle estimator, a second panning and/or tilting angle of the first camera from the vector estimated by the control vector estimator and the angle of view estimated by the angle of view estimator. Disclosed is a method for controlling a camera.

The camera system and the camera control method according to the embodiment of the present invention may increase the overlapping area of the images captured by the first camera and the second camera, respectively.

1 is a block diagram schematically illustrating a camera system according to an embodiment of the present invention.
2 is a flowchart illustrating a camera control method according to an embodiment of the present invention.
3A and 3B are diagrams illustrating a method of estimating an object distance from zoom track information and a method of estimating a first panning and/or tilting angle, respectively.
4A and 4B are diagrams illustrating a method of estimating a vector between the center coordinates of first feature points and the center coordinates of matching points, respectively, and a method of estimating an angle of view from zoom track information.
5 is a flowchart illustrating a camera control method according to another embodiment of the present invention.
6A and 6B are views schematically illustrating the arrangement and overlapping regions of the cameras before the first and second cameras are panned and/or tilted by the first and second drivers, respectively.
7A and 7B are diagrams schematically illustrating an arrangement and overlapping area of a camera after the first camera and the second camera are panned and/or tilted by the first and second drivers, respectively.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

In the following embodiments, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

Terms used in the following examples are only used to describe specific examples, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the following examples, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, but one or more It should be understood that this does not preclude the possibility of addition or presence of other features or numbers, steps, operations, components, parts, or combinations thereof.

Embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented in any number of hardware and/or software configurations that perform specific functions. For example, embodiments of the present invention may be implemented directly, such as memory, processing, logic, look-up table, etc., capable of executing various functions by control of one or more microprocessors or other control devices. Circuit configurations may be employed. Similar to how components of an embodiment of the invention may be implemented as software programming or software elements, embodiments of the invention may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs. , C, C++, Java, assembler, etc. may be implemented in a programming or scripting language. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, embodiments of the present invention may employ conventional techniques for electronic configuration, signal processing, and/or data processing, and the like. Terms such as mechanism, element, means, and configuration may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in association with a processor or the like.

1 is a block diagram schematically illustrating a camera system according to an embodiment of the present invention.

Referring to FIG. 1 , a camera system 1 according to an embodiment of the present invention captures a subject, and a first camera 10 , a second camera 20 , and a first camera 10 are arranged to be spaced apart from each other by a predetermined distance. ) and the first driving unit 30 and the second driving unit 40 for panning and/or tilting the second camera 20, respectively, and a control unit for controlling the first driving unit 30 and the second driving unit (50).

The control unit 50 includes an object distance estimator 52 estimating a distance between the first camera 10 and the subject, and a second control unit of the first camera 10 from the distance estimated by the object distance estimator 52 . 1 A first angle estimator 551 for estimating a panning and/or tilting angle is included.

The control unit 50 controls the center coordinates of the image photographed by the first camera 10 and the region overlapping the image photographed by the second camera 20 among the images photographed by the first camera 10 . The control vector estimator 53 for estimating the vector between the center coordinates, the angle of view estimator 54 for estimating the angle of view from pre-stored zoom track information of the first camera 10, and the control vector estimator 53 ) and a second angle estimator 552 for estimating a second panning and/or tilting angle from the angle of view estimated by the angle of view estimator 54 .

The angle control unit 56 includes the first driving unit 30 and the second driving unit 40 by the angle calculated from the angle estimating unit 55 including the first angle estimating unit 551 and the second angle estimating unit 552 . ) may be controlled to pan and/or tilt the first camera 10 and the second camera 20 .

At least one of the first camera 10 and the second camera 20 may have a zoom function, and the controller 50 controls the zooming of the first camera 10 and/or the second camera 20 . It may further include a zoom control unit 51 to control.

The first camera 10 and the second camera 20 may be cameras of the same type or different types that provide image information by photographing the same scene, for example, the first camera 10 and the second camera 20 ) may be a visible camera and a thermal camera, respectively.

A visible light camera may generate a visible image different in the luminance distribution of an object by acquiring image information by detecting light, and the visible light camera may use a CCD or CMOS as an image sensor.

The thermal camera may detect radiation energy emitted by an object, measure the intensity of the thermal energy, and generate a thermal image representing different colors according to the intensity.

The camera system 1 may further include an image processing unit 60 that processes images captured by the first camera 10 and the second camera 20 , and the image processing unit 60 includes the first camera 10 . ) a first feature point detector 61 for detecting first feature points of an image photographed by and a matching point detection unit 63 that detects matching points in which the values and the second feature points overlap.

The first and second feature point detection units 61 and 62 use various feature point extraction algorithms to extract corners and edges from images captured by the first camera 10 and the second camera 20, respectively. , contours, or line intersections can be extracted as feature points.

The matching point detection unit 62 detects matching points in which the first feature points and the second feature points overlap, and the matching points overlap each image captured by the first camera 10 and the second camera 20 . can be placed in the area.

The image processing unit 60 includes a first key point center coordinate estimator 64 and a second key point center coordinate estimator for estimating center coordinates of the first and second key points detected by the first and second key point detection units 61 and 62, respectively. It may further include a matching point center coordinate estimator 66 for estimating the center coordinates of the matching points detected by the estimator 65 and the matching point detector 63 .

The control vector estimator 53 may estimate a vector between the center coordinates of the first and/or second feature points and the center coordinates of the matching points.

Although not shown, the image processing unit 60 may further include an image conversion unit, a fusion unit, and the like.

The camera system 1 may further include a memory (not shown), and the memory includes zoom track information indicating the distance and angle of view information to the object disposed at the focused position according to the degree of zooming. may be stored.

2 is a flowchart illustrating a camera control method according to an embodiment of the present invention, and FIGS. 3A and 3B are a method of estimating an object distance from zoom track information and a method of estimating a first panning and/or tilting angle, respectively. 4A and 4B are diagrams illustrating a method of estimating a vector between the center coordinates of the first feature points and the center coordinates of the matching points, respectively, and a method of estimating the angle of view from zoom track information.

_{Referring to FIG. 2 , the method for controlling a camera according to an embodiment includes estimating a distance r 1} between a first camera ( FIGS. 1 and 10 ) and a subject by an object distance estimator ( FIGS. 1 and 52 ). The method may include estimating the first panning and/or tilting angle θ ₁ from the ( S151 ) and the estimated distance r1 ( S152 ).

Referring to FIG. 3A , the object distance r1 may be estimated from zoom track information _{on the object distance r 1} according to the zooming of the first camera 10 . 3A is a graph illustrating a relationship between a zoom track and an object distance, and may be data obtained in advance and stored in a memory.

_{Referring to FIG. 3B , the first panning and/or tilting angle θ 1} is calculated from the object distance r1 and the distance r2 between the first camera 10 and the second camera 20, which are preset values. can The first panning and/or tilting angle θ ₁ may be calculated by the following equation.

<expression>

θ ₁ = tan ^-1 (r ₂ /2r ₁ )

2, 4A and 4B , in the camera control method, the center coordinates of the image IF1 photographed by the first camera ( FIGS. 1 and 10 ) by the control vector estimator ( FIGS. 1 and 53 ) Between the image IF2 captured by the second camera ( FIGS. 1 and 20 ) among the images IF1 captured by the ICP and the first camera 10 and the center coordinates MCP of the overlapping area MA In the step of estimating the vector V of ( S153 ) and the zoom track information for the angle of view (FOV) according to the zooming of the first camera 10 by the angle of view estimator ( FIGS. 1 and 54 ), the angle of view (FOV) It may include the step of estimating (S154).

The center coordinate ICP of the image IF1 photographed by the first camera 10 and the center coordinate MCP of the area MA overlapping the image IF2 photographed by the second camera 20 are As described above, it may be estimated by the image processing unit ( FIGS. 1 and 60 ). That is, the center coordinate ICP of the image IF1 captured by the first camera 10 may be the center coordinate of the first feature points, and the center coordinate MCP of the overlapping area MA is the center of the matching points. It can be coordinates.

4A illustrates a case in which the vector V estimated by the control vector estimator 53 includes only the x-axis component, but the present invention is not limited thereto and the vector V includes the x-axis component and the y-axis component. can include all of them.

4B is a graph illustrating a relationship between a zoom track and an angle of view (FOV), and may be data obtained in advance and stored in a memory.

_{The camera control method may include estimating a second panning and/or tilting angle θ 2} from the vector V and the angle of view (FOV) ( S155 ). The second panning and/or tilting angle θ ₂ may be calculated by the following equation.

<expression>

θ ₂ = FOV×C _V /C _t

Here, Cv denotes the number of columns corresponding to the vector V, and Ct denotes the number of columns in the entire image captured by the first camera.

Since the vector V of FIG. 4B includes only the x-axis component, the second panning and/or tilting angle θ ₂ is determined by the number of pixels in the x-axis direction of the vector V, that is, the number of columns as shown in the above equation. , when the vector V includes a y-axis component, the second panning and/or tilting angle θ ₂ corresponding to the y-axis component may be calculated from the number of pixels in the y-axis direction of the vector V.

The camera control method includes the first panning and/or tilting angle (θ ₁ ) and the second panning and/or tilting angle (θ ₂ ) by an angle (θ) calculated by the following equation from the first camera (10) may include the step of panning and/or tilting by the first driving unit ( FIGS. 1 and 30 ) ( S157 ).

<expression>

θ=a×θ ₁ + (1-a)×θ ₂

Here, θ ₁ is a first panning and/or tilting angle, θ ₂ is a second panning and/or tilting angle, and a is a constant of 0 or more and 1 or less.

a _{represents the weights for θ 1} and θ ₂ , and the panning and/or tilting angle θ of the first camera 10 varies according to the set a value, and thus the first camera 10 and the second camera ( 20), the area of the overlapping area of the images captured by each may vary depending on the value of a. When the value of a is 0, the second panning and/or tilting angle (θ ₂ ) becomes the panning and/or tilting angle (θ), and when the value of a is 1, the first panning and/or tilting angle (θ ₁ ) is It can be a panning and/or tilting angle θ.

That is, by adjusting the a value, the area of the overlapping area of the images captured by the first camera 10 and the second camera 20 can be maximized.

5 is a flowchart illustrating a camera control method according to another embodiment of the present invention.

_{Referring to FIG. 5 , a camera control method according to another embodiment includes estimating a distance r 1} between a first camera ( FIGS. 1 and 10 ) and a subject by an object distance estimation unit ( FIGS. 1 and 52 ). and estimating the first panning and/or tilting angle (θ ₁ ) from the ( S251 ) and the estimated distance ( S252 ).

The camera control method includes estimating a first vector between the center coordinates of the first feature points and the center coordinates of the matching points by the control vector estimator ( FIGS. 1 and 53 ) ( S253a ) and matching with the center coordinates of the second feature points estimating the second vector between the coordinates of the center of the points (S253b), and estimating the first angle of view of the first camera 10 by the angle of view estimator (FIGS. 1 and 54) (S254a) and the second vector 2 It may include estimating the second angle of view of the camera 20 ( S254b ).

_{A second panning and/or tilting angle θ 21} of the first camera 10 may be estimated from the first vector and the first angle of view (S255a), and the second camera 20 may be estimated from the second vector and the second angle of view. ) of the second panning and/or tilting angle θ ₂₂ may be estimated (S255b).

Camera control method, the second pan and / or the mean value of the tilting angle (θ ₂₁₎ and the second camera 20, a second pan and / or tilt angle (θ ₂₂₎ of a (θ ₂₎ of the camera 10 and calculating (S256), from the average value θ ₂ and the first panning and/or tilting angle θ ₁ by the angle θ calculated by the following equation, the first camera 10 and The method may include panning and/or tilting the second camera 20 by the first driving unit ( FIGS. 1 and 30 ) and the second driving unit ( FIGS. 1 and 40 ), respectively ( S257 ).

<expression>

θ=a×θ ₁ + (1-a)×θ ₂

Here, θ ₁ is the first panning and/or tilting angle, θ _{2 is the second panning and/or tilting angle θ 21} of the first camera 10 and the second panning and/or tilting angle of the second camera 20 The average value of the tilting angle θ ₂₂ , and a represents a constant of 0 or more and 1 or less.

a _{represents the weights for θ 1} and θ ₂ , and the panning and/or tilting angle θ of the first camera 10 and the second camera 20 varies according to the set a value, and thus the first camera ( 10) and the area in which the images captured by the second camera 20 overlap each other may vary according to the value a.

That is, by adjusting the a value, the area of the overlapping area of the images captured by the first camera 10 and the second camera 20 can be maximized.

In the camera control method, after panning and/or tilting the first camera 10 and the second camera 20 by the angle θ by the first driving unit 30 and the second driving unit 40, respectively, the second The method may include determining whether an area of an overlapping region between the image captured by the first camera 10 and the image captured by the second camera 20 is equal to or greater than a predetermined value ( S258 ).

The area of the overlapping region may be estimated through image processing by the image processing unit ( FIGS. 1 and 60 ).

When the area of the overlapping area is greater than or equal to a predetermined value, the first camera 10 and the second camera 20 are driven by the first driving unit 30 and the second driving unit 40, respectively, to increase the overlapping area. The control phase may end.

However, when the area of the overlapping region is equal to or less than a predetermined value, the step of estimating the first vector between the center coordinates of the first feature points and the center coordinates of the matching points again (S253a) and the central coordinates of the second feature points and the matching points The step S253b of estimating the second vector between the center coordinates is executed, and the subsequent steps may be repeated.

6A and 6B are views schematically illustrating the arrangement and overlapping regions of the cameras before the first and second cameras are panned and/or tilted by the first and second drivers, respectively.

Referring to FIGS. 6A and 6B , the optical axes of the first camera 10 and the second camera 20 are disposed substantially parallel to each other, and the image IF1 captured by the first camera 10 and the second camera 10 are 2 The image IF2 captured by the camera 20 may have an overlapping area MA having a predetermined area.

The area of the overlapping area MA may further decrease as the separation distance between the first camera 10 and the second camera 20 increases.

7A and 7B are diagrams schematically illustrating an arrangement and overlapping area of a camera after the first camera and the second camera are panned and/or tilted by the first and second drivers, respectively.

7A and 7B , FIG. 7A shows a first camera 10 and a second camera 20 according to a camera control method according to embodiments of the present invention, respectively, a first driving unit ( FIGS. 1 and 30 ) and a state of being panned and/or tilted by a predetermined angle θ by the second driving unit ( FIGS. 1 and 40 ), and FIG. 7B is an image IF1 captured by the first camera 10 photographed in this state. ) and an image IF2 captured by the second camera 20 are overlapped with each other, indicating an area MA.

Comparing FIGS. 6B and 7B , it can be seen that the overlapping area MA, that is, the area in which a plurality of pieces of information can be obtained from different cameras, is significantly increased.

The two images corresponding to the overlapping area MA may be matched through a predetermined image processing process, and through this, various information may be simultaneously acquired with respect to the overlapping area MA.

Although the above-described embodiments exemplify a case in which there are two cameras, the present invention is not limited thereto, and the number of cameras may be three or more, and each camera may be a camera of various types of the same type or of a different type.

The camera control method according to the embodiments of the present invention can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

Although the present invention has been described with reference to the embodiments shown in the drawings, which are merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10: first camera 20: second camera
30: first driving unit 40: second driving unit
50: control unit 60: image processing unit

Claims (6)

a first camera for photographing a subject;
a second camera photographing the subject and spaced apart from the first camera by a predetermined distance;
a first driving unit and a second driving unit performing at least one of panning and tilting with respect to each of the first camera and the second camera; and
a control unit for controlling the first driving unit and the second driving unit;
The control unit is
an object distance estimator for estimating a distance between the first camera and the subject from pre-stored zoom track information of the first camera;
a first angle estimator for estimating a first angle of the first camera from the distance estimated by the object distance estimator;
A control vector estimator for estimating a vector between the center coordinates of the image captured by the first camera and the center coordinates of a region overlapping the image captured by the second camera among the images captured by the first camera ;
an angle of view estimator for estimating an angle of view from pre-stored zoom track information of the first camera; and
a second angle estimator for estimating a second angle from the vector estimated by the control vector estimator and the angle of view estimated by the angle of view estimator;
The second angle is determined by the following equation.
<expression>
θ ₂ = FOV×C _V /C _t
Here, FOV is the angle of view, C _v is the number of columns corresponding to the vector V, and C _t is the number of columns of the entire image captured by the first camera. delete According to claim 1,
Further comprising an image processing unit for processing the image taken by the first camera and the second camera,
The image processing unit,
a first feature point detector for detecting first feature points of the image captured by the first camera;
a second feature point detector for detecting second feature points of the image captured by the second camera;
a first feature point center coordinate estimator for estimating center coordinates of the first feature points;
a matching point detection unit detecting matching points in which the first feature points and the second feature points overlap; and
and a matching point center coordinate estimator for estimating the center coordinates of the matching points.
The control vector estimator is configured to estimate a vector between the center coordinates of the first feature points and the center coordinates of the matching points. 4. The method of claim 1 or 3,
The first camera is configured by the first driving unit by an angle calculated by the following equation from the first angle estimated by the first angle estimator and the second angle estimated by the second angle estimator. A camera system that performs at least one of panning and tilting.
<expression>
θ=a×θ ₁ + (1-a)×θ ₂
Here, θ ₁ is a first angle, θ ₂ is a second angle, and a is a constant of 0 or more and 1 or less. 4. The method of claim 1 or 3,
The first camera and the second camera are a visible camera and a thermal camera, respectively. In the camera control method using the camera system,
The camera system includes a first camera and a second camera disposed to be spaced apart from each other by a predetermined distance while photographing a subject, a first driving unit and a second camera configured to perform at least one of panning and tilting for each of the first camera and the second camera 2 includes a drive,
The camera control method comprises:
estimating, by an object distance estimator, a distance between the first camera and the subject from pre-stored zoom track information of the first camera; and
estimating, by a first angle estimator, a first angle of the first camera from the distance estimated by the object distance estimator;
A vector between the center coordinates of the image captured by the first camera and the center coordinates of a region overlapping the image captured by the second camera among the images captured by the first camera by the control vector estimator estimating;
estimating, by an angle of view estimator, an angle of view from pre-stored zoom track information of the first camera; and
estimating, by a second angle estimator, a second angle of the first camera from the vector estimated by the control vector estimator and the angle of view estimated by the angle of view estimator;
The second angle is a camera control method determined by the following equation.
<expression>
θ ₂ = FOV×C _V /C _t
Here, FOV is the angle of view, C _v is the number of columns corresponding to the vector V, and C _t is the number of columns of the entire image captured by the first camera.

Images