JP2013009202A - Camera direction adjustment device and camera direction adjustment method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for simple and accurate adjustment of the optical axis direction of a digital camera.SOLUTION: The camera direction adjustment device comprises: image acquisition means for acquiring images successively taken by a camera subjected to adjustment of an optical axis direction; display means which comprises a display screen disposed at a position facing a target direction that is the target of the optical axis direction and a reference mark serving as a reference for adjustment; and control means for display and output of the acquired taken images on the display screen.

Description

  Embodiments described herein relate generally to a camera direction adjusting device and a camera direction adjusting method.

  Conventionally, for digital cameras installed in various electronic devices such as television receivers and industrial robots, when adjusting the optical axis direction (camera direction), which is the direction in which the digital camera captures images, the digital camera is used as a reference. An image obtained by capturing the mark is confirmed on a monitor, and the orientation of the digital camera is adjusted so that the reference mark is in a predetermined position.

JP 2005-334998 A

  However, in the above-described prior art, in order to adjust the camera direction with high accuracy, it is necessary to place the reference mark far from the digital camera, and it is difficult to easily adjust.

  The present invention has been made in view of the above, and an object of the present invention is to provide a camera direction adjusting device and a camera direction adjusting method capable of easily and accurately adjusting the optical axis direction of a digital camera. .

  In order to solve the above-described problems and achieve the object, the camera direction adjustment apparatus according to the embodiment includes an image acquisition unit that acquires captured images sequentially captured by a camera to be adjusted in the optical axis direction; Display means having a display screen arranged at a position facing the target direction as a target and a reference mark as a reference for adjustment, and control means for displaying and outputting the acquired captured image on the display screen Prepare.

  In addition, the camera direction adjustment method of the embodiment includes a display screen arranged at a position facing a target direction in the optical axis direction of the camera to be adjusted and a reference mark as a reference for adjustment. A camera direction adjustment method for a camera direction adjustment apparatus, comprising: acquiring captured images sequentially captured by the camera; and displaying the acquired captured images on the display screen.

FIG. 1 is an external view illustrating an example of a schematic configuration of a camera direction adjusting apparatus according to an embodiment. FIG. 2 is an external perspective view illustrating an example of a television device. FIG. 3 is a block diagram illustrating a hardware configuration of the camera direction adjusting apparatus according to the embodiment. FIG. 4 is an external view illustrating the front external appearance of the display device. FIG. 5 is a conceptual diagram illustrating an example of a camera view display by a loop generated in imaging of a digital camera. FIG. 6 is a conceptual diagram illustrating the relationship between the optical axis direction with respect to the target direction and the camera view. FIG. 7 is a conceptual diagram illustrating a reference coordinate system. FIG. 8 is a conceptual diagram illustrating projective transformation. FIG. 9 is a flowchart illustrating an example of calibration.

  Hereinafter, a camera direction adjusting device and a camera direction adjusting method of an embodiment will be described in detail with reference to the accompanying drawings. FIG. 1 is an external view showing an example of a schematic configuration of a camera direction adjusting apparatus 1 according to the embodiment.

  As shown in FIG. 1, the camera direction adjusting device 1 includes a display device 10 having a display screen 101 such as an LCD (Liquid Crystal Display), and a control box 11 that controls the display device 10. The control box 11 is connected to the television device 2 via a communication cable 12 and is connected to the display device 10 via a communication cable 13. The display surface of the display screen 101 is disposed in advance at a position directly facing the target direction D1 that is the target in the optical axis direction D2 of the digital camera 22 (see FIG. 2) provided in the television device 2. Specifically, the center D of the display surface of the display screen 101 and the target direction D1 are arranged at a right angle.

  FIG. 2 is an external perspective view showing an example of the television apparatus 2. As shown in FIG. 2, the television device 2 includes a housing 20, a support unit 21, a digital camera 22, and a display unit 23. The housing 20 includes a digital camera 22 and a display unit 23 on the front, and is supported by the support unit 21. The digital camera 22 is provided in the upper part of the display unit 23, and images the optical axis direction D2 illustrated in FIG. 1 by a CMOS image sensor, a CCD image sensor, or the like. An image captured by the digital camera 22 is output to the control box 11 via a communication cable 12 connected to a communication connector provided at a lower portion (not shown) of the housing 20. The display unit 23 is an LCD or the like that displays a video signal input from a tuner or a video input unit (both not shown). In the embodiment, the case where the optical axis direction D2 of the digital camera 22 provided in the television apparatus 2 is adjusted is illustrated, but the digital camera 22 that performs the adjustment is configured to output the captured image to the control box 11. Any electronic apparatus may be provided as long as it is provided. For example, the digital camera 22 may be provided in an electronic device such as an assembly robot.

  FIG. 3 is a block diagram illustrating a hardware configuration of the camera direction adjusting apparatus 1 according to the embodiment. As shown in FIG. 3, the camera direction adjusting apparatus 1 includes a CPU 111, a RAM 112, a ROM 113, an HDD 114, a communication controller 115, a display controller 117, a communication unit 116, an operation input controller 118, and an operation input unit. 119.

  The CPU 111 executes various programs installed and stored in advance in the ROM 113, the HDD 114, and the like, and controls the operation of each unit constituting the camera direction adjustment apparatus 1. The RAM 112 is a rewritable volatile storage device that functions as a work area for the CPU 111 and functions as a stack, a buffer, and the like during various processes. The ROM 113 is a non-rewritable non-volatile storage device, and stores a program and various setting information for controlling the camera direction adjusting apparatus 1. The HDD 114 is a rewritable nonvolatile storage device, and stores programs, various data, and the like related to the control of the camera direction adjusting apparatus 1. The camera direction adjusting apparatus 1 can include a storage device such as an SSD (solid state drive) or a semiconductor memory (for example, a flash memory) as a rewritable nonvolatile storage device (storage unit). .

  The communication controller 115 controls data communication in the communication unit 116 under the control of the CPU 111. The communication unit 116 performs data communication according to a predetermined communication protocol with an external device such as the television apparatus 2 connected via the communication cable 12. Specifically, the communication unit 116 acquires images sequentially captured by the digital camera 22 of the television apparatus 2 under the control of the communication controller 115. The display controller 117 generates display data under the control of the CPU 111, and outputs the generated display data to the display device 10, thereby displaying on the display screen 101 of the display device 10 connected via the communication cable 13. Control the output. Specifically, the display controller 117 causes the display screen 101 to output the image captured by the digital camera 22 of the television apparatus 2 acquired by the communication unit 116. The operation input controller 118 outputs a signal received from the operation input unit 119 configured as a keyboard, a touch panel, a click button, a push button, a switch, or the like according to the operation of the operator to the CPU 111. For example, in the CPU 111, a process of displaying and outputting an image captured by the digital camera 22 of the television apparatus 2 acquired by the communication unit 116 on the display screen 101 in accordance with a signal output from the operation input controller 118 by an operator's operation. To start.

  FIG. 4 is an external view illustrating the front external appearance of the display device 10. When adjusting the optical axis direction D2 of the digital camera 22 of the television apparatus 2 to the target direction D1 with the digital camera 22 and the display device 10 facing each other, as shown in FIG. Originally, a reference mark image 102 as a reference for adjustment and a camera view 103 which is an image sequentially captured by the digital camera 22 are displayed on the display screen 101. The reference mark image 102 may have any shape as long as it has a predetermined shape displayed at a predetermined position outside the camera view 103. For example, the reference mark image 102 may be an outer frame of the display screen 101, a frame of the display device 10, or the like. May be. In the illustrated example, the reference mark image 102 is obtained by displaying a rectangle whose four inner angles are all 90 degrees (hereinafter referred to as a basic rectangle) on the display screen 101 outside the camera view 103. It is assumed that the positions of the four vertices of the basic rectangle displayed on the display screen 101 are known. The reference mark image 102 may be colored with a predetermined color so as to facilitate image processing described later. The background color of the reference mark image 102 may be changed so that the contrast is clear. The camera view 103 is displayed on the display screen 101 with the center D on the display screen 101 as a reference (center of the camera view 103). The distance between the display device 10 and the digital camera 22 when adjusting the optical axis direction D2 of the digital camera 22 to the target direction D1 is such that the camera field of view 221 of the digital camera 22 includes the reference mark image 102 and the camera view 103. It shall be adjusted to

  As illustrated in FIG. 4, when the digital camera 22 and the display device 10 face each other and images sequentially captured by the digital camera 22 are displayed on the camera view 103, the display device 10 is imaged and captured by the digital camera 22. A loop (recursion) of displaying an image on the camera view 103 → imaging the display device 10 with the digital camera 22 → occurs. FIG. 5 is a conceptual diagram showing an example of the display of the camera view 103 by a loop that occurs in the imaging of the digital camera 22. As shown in FIG. 5, the camera view 103 displays an image in which the reference mark image 102 is looped inward and a recursive reference mark image. Here, while the optical axis direction D <b> 2 of the digital camera 22 does not coincide with the target direction D <b> 1, the looped image displayed on the camera view 103 has a distorted shape with respect to the reference mark image 102. This distortion increases as it goes from the outside to the inside because an error caused by the mismatch between the optical axis direction D2 and the target direction D1 is amplified by the loop.

  In addition, a different loop image is displayed on the camera view 103 depending on which position displacement or rotation displacement occurs between the optical axis direction D2 and the target direction D1. FIG. 6 is a conceptual diagram illustrating the relationship between the optical axis direction D2 with respect to the target direction D1 and the camera view 103. As shown in FIG. 6, the imaging surface of the digital camera 22 and the display surface of the display screen 101 are parallel to each other, and the horizontal and vertical directions of both surfaces coincide with each other. An image similar to the reference mark image 102 with the camera view 103 as the center when the feet of the perpendicular line drawn from the center to the display screen 101 match, that is, in the ideal case where the target direction D1 and the optical axis direction D2 match. Appears as a loop inward. Even if the imaging surface of the digital camera 22 and the display surface of the display screen 101 are parallel to each other and the horizontal and vertical directions of both surfaces coincide with each other, the digital camera 22 has a center D on the display screen 101. When there is a position shift in which the vertical line drawn from the center is shifted to the display screen 101, an image similar to the reference mark image 102 is looped and displayed inward with the shift position as the center. In addition, when there is a rotational deviation in which the digital camera 22 rotates around the target direction D1, an image similar to the reference mark image 102 is displayed in a loop so that it gradually rotates inward with the camera view 103 as the center. Is done.

  Therefore, the operator easily adjusts the optical axis direction D2 to the target direction by adjusting the looped image displayed from the outside toward the inside displayed in the camera view 103 to be the ideal case illustrated in FIG. It can be adjusted to D1. In addition, since the distortion increases due to the loop, the accuracy can be adjusted without increasing the distance between the digital camera 22 and the display device 10.

  Next, a case where the position / orientation (camera parameter) of the digital camera 22 is calculated from the basic rectangular loop image and the optical axis direction D2 is adjusted to the optical axis direction D2 will be described. The calculation of camera parameters is hereinafter referred to as calibration. Also, internal parameters such as the focal length of an optical lens (not shown) of the digital camera 22 are known, and camera parameters obtained by calibration are external parameters, that is, the three-dimensional position of the digital camera 22 with respect to the reference coordinate system, 3 Assume that the posture is defined by the unit vector of the book.

  In the camera direction adjusting apparatus 1, the CPU 111 performs image processing of the image captured, detects the reference mark image 102 included in the captured image, and an image obtained by recurring the reference mark image 102, and calculates parameters. . Specifically, by performing image processing on the captured image corresponding to the camera view 221 illustrated in FIG. 4, a plurality of images (square images) corresponding to the basic rectangle are extracted. The extracted square image becomes smaller in size from the outside to the inside, and extraction by image processing becomes difficult. Therefore, for example, K square images having a predetermined size or more are taken out. The square image is extracted by performing straight line fitting on each side after detecting an edge from the captured image.

  The method for extracting K square images is arbitrary, but it is efficient to perform the processing in the following procedure. First, a captured image of the digital camera 22 is acquired without displaying the camera view 103 on the display screen 101. At this time, the square image existing on the captured image is only the basic rectangle of the reference mark image 102, and the extraction is easy. As will be described in detail later, the conversion from the display screen 101 to the imaging surface is represented by projective conversion, and is unique from the correspondence between the four vertices (the position of the four vertices on the display screen 101 and the projection onto the image). It is decided. Therefore, this projective transformation is obtained using a square image obtained by extracting the basic rectangle of the reference mark image 102. Next, a captured image of the digital camera 22 is acquired in a state where the camera view 103 is displayed on the display screen 101, and the above-described loop image is observed. Since the outermost square image has already been extracted, the second and subsequent square images are to be extracted. The conversion between two adjacent square images is the same, and is composed of a projective transformation from the display screen 101 to the imaging surface and a scale (enlargement) transformation from the imaging surface to the display screen 101. Since it has already been obtained, a square image may be extracted considering only scale conversion.

The CPU 111 calculates camera parameters from the basic rectangle displayed on the display screen 101 and the video (K square images extracted by image processing) on the image of the basic rectangle. FIG. 7 is a conceptual diagram illustrating a reference coordinate system. Although the method for setting the reference coordinate system for the camera parameters is arbitrary, in the embodiment, the origin of the reference coordinate system is the left end of the display screen 101, the display surface of the display screen 101 is the XY plane and the vertical direction of the display screen 101 is Z Axis. Further, the position (V1) of the digital camera 22 is set to t = (t _x , t _y , t _z ) ^T (T is a transposition symbol), and the attitude of the digital camera 22 (the attitude of the imaging surface (V2)) is set to an orthonormal basis i. , J, k. Further, a matrix M = (i ^T , j ^T , k ^T ) ^T composed of these three vectors is defined. Since M represents the attitude of the digital camera 22, it is called an attitude matrix. The position t and posture matrix M of the digital camera 22 are camera parameters to be obtained. The projection point x = (x, y) ^{T on} the image of the point X = (X, Y, Z) ^{T in} the three-dimensional space is given by the following equation (1).

Since the plane of the display screen 101 coincides with the XY plane, Z = 0. That is, the projected point (x, y) ^T of the point (X, Y, 0) ^T on the display screen 101 is given by the following equation (2).

In the following, a homogeneous coordinate expression is used to simplify the expression. That is, the point (X, Y) on the display screen 101 and the point (x, y) on the image are represented by X = (X, Y, 1) ^T and x = (x, y, 1) ^T , respectively. When this is used, the above-described equation can be expressed by the following equation (3).

ただし、

である。 here,

However,

It is.

The point X (X, Y, 1) ^T on the display screen 101 is subjected to the two-dimensional projective transformation represented by the expression (3) and is projected to the point x = (x, y, 1) ^T of the image.

FIG. 8 is a conceptual diagram illustrating projective transformation. As shown in FIG. 8, the point on the outermost square image of the camera view 103 is x ^(1), and the points on the second and third square images are x ⁽²⁾ and x ⁽³⁾ below. As shown, the k-th square image from the outside is represented as x ^(k) . Similarly, the square image on the display screen 101 is also expressed as X ⁽¹⁾ , X ⁽²⁾ , X ⁽³⁾ from the outside. X ⁽¹⁾ is a basic rectangle displayed outside the camera view 103 on the display screen 101, and X ⁽²⁾ , X ⁽³⁾ ,... Are square images displayed in the camera view 103 on the display screen 101. It is. Since the projection onto the image of the k-th square image X ^(K) from the outside on the display screen 101 is x ^(k) , from Equation (3),

(Formula (4)). The second outermost square image X ⁽²⁾ on the display screen 101 is an enlarged display of the outermost square image x ⁽¹⁾ in the camera view 103 on the display screen 101. When generalized, the k-th outer square image X ^(k) on the display screen 101 is an enlargement of the (k-1) -th outer square image x ^(k-1) on the camera view 103.

となる（式（５））。ただし、Ｓは拡大を表す行列であり、係数ｓを用いて次の式（６）で表現される。

(Equation (5)). However, S is a matrix representing enlargement, and is expressed by the following equation (6) using the coefficient s.

  The following equation (7) is obtained from equations (4) and (5).

here,

It is. P ′, like P, represents a two-dimensional projective transformation. The posture matrix M of the digital camera 22 is obtained from this relational expression and K square images x ^(k) (k = 1, 2,... K) extracted by the above-described image processing.

The four vertices of the k-th square image are represented as x ^(k) ₁ , x ^(k) ₂ , x ^(k) ₃ , x ^(k) ₄ . From the k-th square image and the correspondence between the vertices of the (k−1) -th square image inside one, and Equation (7),

Get. The two equations described above are obtained from the correspondence between the vertices. Since there are four vertices, eight equations are obtained from a pair of square images. Further, since there are K-1 sets of square images adjacent to each other in the K square images, a total of 8 × (K−1) equations are obtained. By projecting these equations, a projective transformation matrix P ′ is obtained. However, P ′ is projective transformation, and its elements have a constant multiple indefiniteness. That is, for example, when w = t ₃ , the following equation (8)

H ₁₁ to h ₃₂ are uniquely determined. Since the first column (r ₁₁ , r ₂₁ , r ₃₁ ) of the attitude matrix M is a unit vector,

  And the following equation (9) is obtained.

  From this equation (9) and equation (8),

As described above, the elements of the first column and the second column of the attitude matrix M are obtained. However, w ′ = w / s. The third column of M (r ₁₃ , r ₂₃ , r ₃₃ ) is

It is obtained from the relational expression In the above relational expression, x represents an outer product of vectors. Two attitude matrices are calculated based on the sign of w ′, and an appropriate one is selected from the physical meaning. For example, since i = (r ₁₁ , r ₁₂ , r ₁₃ ) representing the horizontal direction of the imaging surface of the digital camera 22 substantially matches the X-axis direction, the sign of w ′ can be uniquely determined.

The position t = (t _x , t _y , t _z ) of the digital camera 22 is calculated using equation (3). Substituting the vertex X ⁽¹⁾ _i of the basic rectangle on the display screen 101 and the projection point x ⁽¹⁾ _i into the equation (3), the following equation (10) is obtained.

Equation (10) represents two equations. Using 4 vertices, we get 8 equations. Since the attitude matrix M of the digital camera 22 has already been obtained, this is also used to solve the eight simultaneous equations for t = (t _X , t _Y , t _Z ) ^T to obtain the position of the digital camera 22.

  The procedure described above enables calibration of the digital camera 22, that is, calculation of the position and orientation of the digital camera 22 with respect to the display screen 101.

The validity of the calculated camera parameters can be evaluated by the following method. First, the camera view 103 is displayed together with the rectangular image X ⁽¹⁾ of the basic rectangle so that the center D on the display screen 101 coincides with the foot of the perpendicular drawn from the calculated position t of the digital camera 22 to the plane of the display screen 101. Further, X ⁽¹⁾ is converted as shown in the following equation (11).

  In order to simplify the expression, the superscript “(1)” is omitted. The projection of X ′ onto the image is given by the following equation (12).

  On the other hand, an attitude matrix of an ideal digital camera 22 (referred to as an ideal camera) in which three attitude vectors coincide with the X, Y, and Z axes of the reference coordinate system is expressed by the following equation (13).

  From this equation (13) and equation (3), the projection point x ″ when the basic rectangle is imaged by the ideal camera is expressed by the following equation (14).

  From Expression (12) and Expression (14), x ′ and x ″ match if the calculated camera parameters are correct. That is, when the basic rectangle is converted by the equation (11), the converted image of the basic rectangle becomes the same as the projected image when the basic rectangle is captured by the ideal camera, and the repeated loop image is illustrated in FIG. Like the ideal case. That is, the validity of the calculated camera parameter can be evaluated from the similarity of the observed repeated loop images and the invariance of the direction of each side.

  It is also possible to improve accuracy by performing recalculation until such an ideal repeated loop image is observed. FIG. 9 is a flowchart illustrating an example of calibration. As shown in FIG. 9, after the digital camera 22 is arranged in front of the camera direction adjusting device 1 (S1), the reference mark image 102 is displayed on the display screen 101 by the operation of the operator (S2). Next, the CPU 111 acquires a captured image captured by the digital camera 22 (S3), and performs the above-described image processing (S4) and camera parameter calculation (S5). Next, the CPU 111 determines the end of calibration by determining the validity of the above-described camera parameters (matching x ′ and x ″) (S6).

  When it is determined in the end determination that the camera parameters are valid (S6: YES), the CPU 111 displays the end of adjustment of the digital camera 22 on the display device 10 (S8), and the end of calibration is indicated to the operator. Informs and ends the process. If it is determined in the end determination that the camera parameters are not appropriate (S6: NO), the CPU 111 deforms the shape of the reference mark image 102 by the equation (11), and displays the deformed basic rectangle on the display screen 101. The basic rectangle is updated (S7). After updating the basic rectangle, the CPU 111 returns the process to S3. Therefore, the camera direction adjusting apparatus 1 performs calculation using the deformed square image. In this way, by updating the basic rectangle, the repeated square image approaches the ideal shape, so each side of the square image becomes a horizontal line or a vertical line, and straight line extraction by image processing becomes easy and extraction The accuracy can be increased.

  Note that the CPU 111 may notify the operator by displaying the calculated camera parameters on the display screen 101. In this case, the operator can adjust the digital camera 22 from the notified camera parameters. Further, the CPU 111 may adjust the digital camera 22 by controlling an actuator for adjusting the posture of the digital camera 22 based on the calculated camera parameters.

  Note that the program executed by the camera direction adjusting apparatus 1 of the embodiment is provided by being incorporated in advance in a ROM or the like. A program executed by the camera direction adjusting apparatus 1 according to the embodiment is a file in an installable format or an executable format, and is a computer such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). The information may be provided by being recorded on a recording medium that can be read by the user.

  Furthermore, the program executed by the camera direction adjusting apparatus 1 of the embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. Moreover, you may comprise so that the program run with the camera direction adjustment apparatus 1 of embodiment may be provided or distributed via networks, such as the internet.

  The program executed by the camera direction adjusting apparatus 1 of the embodiment has a module configuration including the above-described functional configuration. As actual hardware, a CPU (processor) reads the program from the ROM and executes it. The functional configuration is loaded on the main storage device and generated on the main storage device.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 1 ... Camera direction adjustment apparatus, 2 ... Television apparatus, 10 ... Display apparatus, 11 ... Control box, 12 ... Communication cable, 13 ... Communication cable, 20 ... Housing, 21 ... Support part, 22 ... Digital camera, 23 ... Display unit 101 ... Display screen 102 ... Reference mark image 103 ... Camera view 111 ... CPU 112 ... RAM 113 ... ROM 114 ... HDD 115 ... Communication controller 116 ... Communication unit 117 ... Display controller 118: Operation input controller, 119: Operation input unit, 221: Camera field of view, D: Center, D1: Target direction, D2: Optical axis direction

Claims (5)

Image acquisition means for acquiring captured images sequentially captured by a camera to be adjusted in the optical axis direction;
Display means having a display screen arranged at a position facing the target direction of the optical axis direction and a reference mark as a reference for adjustment;
Control means for displaying and outputting the acquired captured image on the display screen;
A camera direction adjusting device. The reference mark is arranged outside the captured image displayed and output on the display screen.
The camera direction adjusting apparatus according to claim 1. The reference mark is a rectangular image surrounding the captured image on the display screen.
The camera direction adjustment apparatus according to claim 1 or 2. Detecting means for detecting the reference mark included in the captured image and a reference mark recursed by imaging the display screen displaying the captured image with the camera;
Calculation means for calculating a parameter indicating the position and orientation of the camera based on the reference mark and the position of the recursive reference mark;
A notification means for notifying that the calculated parameter indicates a predetermined value;
The camera direction adjustment apparatus according to any one of claims 1 to 3. Camera direction adjusting method of camera direction adjusting apparatus having display means having display screen arranged at position facing to target direction in optical axis direction of camera to be adjusted and reference mark as reference for adjustment Because
Acquiring captured images sequentially captured by the camera;
Displaying and outputting the acquired captured image on the display screen;
Camera direction adjustment method including

Images