JP6403628B2 - Mask processing apparatus and method, imaging apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an imaging apparatus and method having a rotation drive mechanism for pan rotation and tilt operations, and capable of superimposing a mask that covers an area to be hidden in an image on a captured image, and such The present invention relates to a mask processing apparatus and method used in an imaging apparatus and method. The present invention also relates to a program and a recording medium.

  Conventionally, as an imaging device that has a rotation drive mechanism for pan rotation and tilt operation and can superimpose a mask pattern that covers an area to be hidden in an image on a captured image, the pan rotation and tilt operation is performed. In some cases, a mask pattern to be superimposed on a captured image is moved in conjunction with the area to be hidden and kept hidden.

In such an imaging apparatus, a region to be hidden in a three-dimensional coordinate space with the position of the camera unit as the origin is determined from information indicating the direction of the vertex constituting the region to be hidden with a specific imaging direction as a reference direction. Calculate and save the coordinates of the vertices that make up. When a direction different from the reference direction is imaged after storage, a region to be hidden is calculated based on a difference in angle with respect to the reference direction, and a mask pattern for the region to be hidden is superimposed on the captured image.
When only a rectangular pattern can be selected as the mask pattern, the mask pattern is superimposed on a rectangular area that circumscribes the area to be hidden.

Japanese Patent No. 3996805 (paragraph 0026, FIG. 9)

  However, in such a configuration, for example, when the area represented by the saved coordinates is a rectangle that is long in the horizontal direction, the area that is desired to be hidden using the saved coordinates becomes a long area in the diagonal direction, and the diagonal direction In particular, a rectangular area circumscribing a long area becomes extremely large, and an area to be originally displayed may be hidden, which is a problem.

  The present invention has been made to solve the above-described problem, and an object of the present invention is to make it possible to reliably hide an area to be hidden so that the area to be originally displayed is not hidden as much as possible.

The mask processing apparatus according to the first aspect of the present invention comprises:
A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image A mask processing apparatus for an imaging apparatus, comprising: a camera unit including: a rotation driving mechanism that performs pan driving and tilt driving with respect to the camera unit;
An imaging condition acquisition unit that acquires direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion unit for converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction unit that changes the shape of the rectangular area and generates the coordinates of the vertices of the rectangular area generated by the change;
A spherical coordinate calculation unit that converts the coordinates of the vertex generated by the specified region correction unit into coordinates on a virtual sphere,
A mask area storage unit for storing coordinates on the virtual sphere calculated by the spherical coordinate calculation unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation unit that reads coordinates on the virtual sphere and converts them into coordinates on a second virtual plane;
The coordinates on the second virtual plane obtained by the conversion in the plane coordinate calculation unit are converted into coordinates in the second captured image, and circumscribed in the area defined by the coordinates obtained by the conversion. A mask processing area calculation unit that calculates a rectangular area to be used as a mask processing area,
The designated area correction unit matches a longer length with a shorter length of a horizontal side and a vertical side of a rectangular region defined by coordinates on the first virtual plane. It is characterized by changing as follows.

The mask processing apparatus according to the second aspect of the present invention comprises:
A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image Including a camera part;
A mask processing apparatus of an imaging apparatus having a rotation drive mechanism that performs pan driving and tilt driving with respect to the camera unit,
An imaging condition acquisition unit that acquires direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion unit for converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction unit that divides a rectangular area into a plurality of rectangular small areas and generates coordinates of each vertex of the plurality of small areas;
A spherical coordinate calculation unit that converts the coordinates of the vertex for each of the plurality of small regions generated by the designated region correction unit into coordinates on a virtual sphere,
A mask area storage unit for storing coordinates on the virtual sphere for each of the plurality of small areas calculated by the spherical coordinate calculation unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation unit that reads coordinates on the virtual sphere for each of the plurality of small regions and converts the coordinates to coordinates on a second virtual plane;
The coordinates on the second virtual plane for each of the plurality of small regions obtained by the conversion in the plane coordinate calculation unit are converted into coordinates in the second captured image, and obtained by this conversion. A mask processing area calculation unit that calculates a plurality of rectangular areas circumscribing each of the plurality of small areas defined by the given coordinates as a plurality of mask processing areas;
The designated area correction unit divides a rectangular area defined by coordinates on the first virtual plane into a plurality of rectangular small areas,
The division is performed so that, in each of the small regions, a ratio of a longer length to a shorter length of a horizontal side and a vertical side is equal to or less than a predetermined threshold value. It is characterized by.

  According to the present invention, since the imaging direction is different between the setting of the mask area and the subsequent imaging, the area to be masked becomes very large, so that the area that is not originally intended to be hidden is hidden. Can be prevented.

It is a block diagram which shows the imaging device of Embodiment 1 of this invention. It is the schematic which shows the designation | designated method of the mask area | region performed using the display apparatus of FIG. It is a block diagram which shows the mask processing apparatus of Embodiment 1 of this invention. It is a figure which shows the relationship between the picked-up image and a virtual spherical surface in the case of the setting of a mask area | region in the three-dimensional coordinate space which makes the camera part of an imaging device the origin at the time of an imaging direction corresponding to a Z-axis direction. (A) And (b) is a figure which shows the conversion from the coordinate on the picked-up image of the designated mask area | region to the coordinate on a virtual plane. (A) And (b) is a figure which shows correction | amendment of a mask area | region. It is a figure which shows the projection from a virtual plane to a virtual spherical surface of a mask area | region. It is a figure which shows the example of the imaging direction different from a reference direction. It is a figure which shows the rotational movement to the imaging direction of the vertex of a mask area | region. It is a figure which shows the rotational movement of the vertex of a mask area | region from the imaging direction to a reference direction. It is a figure which shows the projection from the virtual spherical surface to a virtual plane, and the mapping from a virtual plane to a captured image of the vertex of a mask area | region. (A) And (b) is a figure which shows the coordinate transformation from a virtual plane to a captured image, and the determination method of a mask area | region. (A) And (b) is a figure which shows the example of correction | amendment of the designated area | region in Embodiment 2 of this invention. It is a block diagram which shows the mask processing apparatus of Embodiment 3 of this invention. (A) to (e) are diagrams showing an example of correction of a designated region and calculation of a mask processing region using the corrected mask region in Embodiment 3, and processing in a conventional example. (A), (b) and (c) is a figure which shows the example from which the division | segmentation of the designated area | region in Embodiment 3 differs. It is a block diagram which shows the mask processing apparatus of Embodiment 4 of this invention. FIG. 10 is a diagram illustrating an example of mask area correction in the fourth embodiment. FIG. 20 is a diagram showing another example of mask area correction in the fourth embodiment. It is a block diagram which shows the example of the computer which performs the process of this invention.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention.
The illustrated imaging apparatus includes a rotation drive mechanism 11, a zoom lens 12, an imaging element 13, a pre-processing unit 15 as a first image processing unit, a post-processing unit 17 as a second image processing unit, A video signal output terminal 18, a control unit 19, and a mask processing device 20 are provided. A display device 22 is connected to the video signal output terminal 18, and an operation input unit 23 provided in the display device 22 is connected to the control unit 19.

  The zoom lens 12 condenses light from the subject and forms an optical image on the imaging surface of the imaging element 13.

The image sensor 13 is composed of a CCD or a CMOS image sensor, converts an optical image formed by the zoom lens 12 into an electric signal, and outputs an analog image signal Da representing a captured image.
The zoom lens 12 and the image sensor 13 constitute a camera unit 14.

  The control unit 19 controls the imaging direction designation value SDR, that is, the pan angle designation value φd and the tilt angle designation value θd, in order to control the imaging direction by the camera unit 14 (the direction of the optical axis of the zoom lens 12). Output to the rotation drive mechanism 11. The pan angle specified value φd and the tilt angle specified value θd are expressed as relative values with the imaging direction when the camera unit 14 is facing a certain direction as a reference direction.

  The rotation drive mechanism 11 changes the pan angle and tilt angle of the camera unit 14 according to the designated value SDR of the imaging direction output from the control unit 19, that is, the designated value φd of the pan angle and the designated value θd of the tilt angle. .

  The control unit 19 also outputs a designated value FLD of the focal length of the zoom lens 12 to the zoom lens 12. The optical magnification of the zoom lens can be changed by changing the focal length. The optical magnification of the zoom lens 12 is expressed by using the focal length when the zoom lens 12 is in the widest angle state as a reference focal length. That is, the optical magnification corresponds to the ratio of the focal length at the time of imaging to the reference focal length.

  The zoom lens 12 changes the focal length according to the designated value FLD of the focal length output from the control unit 19.

The preprocessing unit 15 performs CDS (correlated double sampling) processing, gain processing, A / D conversion processing, and the like on the analog image signal Da from the image sensor 13, and outputs a digital image signal Db representing the captured image. To do.
The imaging device 13 and the preprocessing unit 15 constitute an imaging unit 16 that images a subject and outputs a digital image signal Db representing the captured image.

  The control unit 19 further receives and outputs the mask region designation information MRD from the operation input unit 23 of the display device 22. The mask area is an area in the captured image that is desired to be covered when outputting for display or the like for privacy protection or for security reasons. An example of an area to be covered for privacy protection is a window portion of a residence. An example of an area to be covered for security reasons is a key operation unit used for password entry at the entrance of a management area where entry is restricted.

  The mask area is designated by designating a partial area of the image picked up by the image pickup apparatus. For example, as shown in FIG. 2, the captured image is displayed on the display screen 24 of the display device 22, and the position of the vertex of the mask region Mj in the captured image is determined using the operation input unit 23 (FIG. 1), for example. It is specified.

  The designated mask area Mj is, for example, a rectangular area. In this case, the mask area designation information MRD is composed of coordinates Jn (Jnx, Jny) (n = 1, 2, 3, 4) of the four vertices of the designated mask area (mask designation area) Mj.

  The rectangular mask area Mj is, for example, a rectangular area composed of sides extending in the horizontal direction and the vertical direction of the display screen 24. In this case, when the user designates the positions of two vertices at the diagonal position of the mask area Mj, for example, the upper left vertex and the lower right vertex, this also automatically positions the other two vertices of the mask area Mj. Specified.

  The mask processing device 20 includes the mask area designation information MRD generated by the designation of the mask area Mj and the designation value SDR of the imaging direction at the time of imaging of the image used at the time of designation of the mask area Mj, that is, panning The mask area indicated by the mask area specifying information MRD is corrected from the specified angle φd and tilt angle specified value θd, and the specified focal length FLD, coordinate conversion is performed, and the mask area after the coordinate conversion is corrected. The coordinates Rn of the four vertices are registered. The process of registering the vertex coordinate Rn is referred to as setting of a mask area constituted by the vertex or registration of a set mask area. The registered coordinates Rn are used for determining a region to be masked in a captured image obtained by subsequent imaging.

  At the time of imaging after setting the mask area, the mask processing device 20 specifies the specified value SDR of the imaging direction at the time of imaging, that is, the specified value φd of the pan angle and the specified value θd of the tilt angle, the specified value FLD of the focal length, From the coordinates Rn of the four vertices of the set mask area, a mask processing area in the captured image is calculated and output to the post-processing unit 17.

  When setting the mask area, the post-processing unit 17 applies correction to the mask area designated using the operation input unit 23 or the designated mask area as described later by the mask processing apparatus 20. The image indicating the mask area after correction (for example, the image indicating the outline of the area) MPA generated in step S is superimposed on the captured image output from the preprocessing unit 15.

The post-processing unit 17 performs the imaging after setting the mask region based on the information MPB indicating the mask processing region output from the mask processing device 20 for the image output from the pre-processing unit 15. Perform mask processing.
The mask processing is performed by superimposing a fill pattern for filling, replacing with a separately prepared image, mosaic processing, or the like on a portion corresponding to the mask processing area in the image output from the preprocessing unit 15.

  The post-processing unit 17 superimposes the information indicating the mask area specified as described above on the captured image output from the pre-processing unit 15 or performs the mask processing described above, and then outputs the image signal to the video signal output terminal 18. Processing corresponding to the characteristics of the connected display device 22 or other output device, for example, YCbCr conversion, color matrix conversion, and gradation conversion is performed, and the video signal Dc generated by the processing is output from the video signal output terminal 18 To do.

Hereinafter, an example of the mask processing apparatus 20 will be described with reference to FIG.
The mask processing device 20 shown in FIG. 3 includes an imaging condition acquisition unit 31, a designated region conversion unit 32, a mask region registration unit 33, a mask region storage unit 34, a plane coordinate calculation unit 35, and a mask processing region calculation. Unit 36 and a designated region inverse transform unit 37.

  The imaging condition acquisition unit 31 outputs the specified value SDR of the imaging direction, that is, the specified value φd of the pan angle and the specified value θd of the tilt angle of the camera unit 14 output from the control unit 19 when setting the mask area. The designated value FLD of the focal length supplied to the mask area registration unit 33 and output from the control unit 19 is supplied to the designated area conversion unit 32.

  The imaging condition acquisition unit 31 outputs an imaging direction designation value SDR output from the control unit 19, that is, a panning angle designation value φd and a tilt angle designation, which are output from the control unit 19 at the time of imaging after setting the mask area. The value θd is supplied to the plane coordinate calculation unit 35, and the designated focal length value FLD output from the control unit 19 is supplied to the mask processing region calculation unit 36.

  In the following description, the imaging direction designated by the designated value SDR of the imaging direction is indicated by the same symbol “SDR”, and the pan angle and tilt angle designated by the pan angle designated value φd and the tilt angle designated value θd. Are denoted by the same reference numerals “φd” and “θd”. The same applies to other codes. Further, in order to distinguish between the imaging direction when setting the mask area and the imaging direction when imaging after setting the mask area, the former is represented by reference numerals “SDRm”, “φm”, and “θm”, and the latter is “ It may be represented by “SDRv”, “φv”, “θv”.

  The designated area conversion unit 32 forms a mask designated area converted into the reference focal length from the mask area designation information MRD output from the control unit 19 and the designated value FLD of the focal length when setting the mask area. The coordinates Pn (n = 1, 2, 3, 4) of the four vertices to be calculated are calculated and output to the mask area registration unit 33.

  When setting the mask area, the mask area registration unit 33 performs a process to be described later on the coordinates Pn of the four vertices constituting the mask designation area output from the designation area conversion unit 32, and 4 obtained as a result of the process. The vertex coordinates Rn are stored in the mask area storage unit 34 as the coordinates of the four vertices of the set mask area.

  The plane coordinate calculation unit 35 reads the coordinates Rn of the four vertices of the set mask area stored in the mask area storage unit 34 at the time of imaging after setting the mask area, and is output from the imaging condition acquisition unit 31 The plane coordinates Tn (n = 1, 2, 3, 4) converted to the reference focal length are calculated from the pan angle specified value φd and the tilt angle specified value θd.

  The mask processing area calculation unit 36 performs conversion according to the designated value FLD of the current focal length output from the imaging condition acquisition unit 31 with respect to the plane coordinate Tn output from the plane coordinate calculation unit 35, and performs current imaging. The mask processing area in the image is calculated, the start coordinates (SUx, SUy) and the end coordinates (EUx, EUy) are calculated, and output to the post-processing unit 17 as information MPB indicating the mask processing area.

  Hereinafter, the operations of the designated region conversion unit 32, the mask region registration unit 33, the mask region storage unit 34, the plane coordinate calculation unit 35, and the mask processing region calculation unit 36 will be described in more detail.

  As described above, the mask area is specified by specifying the area to be hidden in the captured image using the operation input unit 23. The area is specified by inputting the position of the vertex of the area.

The designated area conversion unit 32 is provided with mask area designation information MRD generated by designation of the mask area from the control unit 19 and is used for designation of the mask area from the imaging condition acquisition unit 31. A designated value FLD of a focal length at the time of imaging is given.
As the mask area designation information MRD, information indicating the coordinates Jn (Jnx, Jny) (n = 1, 2, 3, 4) of the four vertices of the mask designation area in the captured image in the captured image is given.

  The designated area conversion unit 32 uses the given mask area designation information MRD and the designated focal length value FLD to set the coordinates Pn (Pnx, Pny, Pny, 4 vertexes) of the mask designated area converted to the reference focal length. Pnz) (n = 1, 2, 3, 4) is calculated and output to the mask area registration unit 33.

  The mask designation area converted into the reference focal length is a mask designation area when the focal length of the zoom lens 12 is set as the reference focal length (when the zoom lens 12 is at the widest angle).

  Hereinafter, a process in the designated area conversion unit 32 and a process in a mask area registration unit 33 described later will be described with reference to FIG.

  FIG. 4 shows a virtual spherical surface that is located in a three-dimensional coordinate space with the position of the camera unit 14 as the origin and the X, Y, and Z axes orthogonal to each other as the coordinate axes, the origin is the center, and the radius is 1. SP is shown. In the following, the direction of the Z-axis is the reference direction.

When the focal length of the zoom lens 12 is equal to the reference focal length, the captured image is formed on a plane that is in contact with the virtual spherical surface SP. Accordingly, the coordinates Pn (n = 1, 2, 3, 4) of the four vertices constituting the mask designation area converted into the reference focal length are coordinates on the plane that is in contact with the virtual spherical surface SP.
When the imaging direction (the direction of the center of the captured image) matches the reference direction (Z-axis direction), the coordinates Pn of the four vertices are on the plane PLr that is perpendicular to the Z-axis and touches the virtual spherical surface SP. Coordinates.

Since the virtual plane PLr is perpendicular to the Z axis and touches the virtual spherical surface SP, Z = 1 in the virtual plane PLr. The virtual plane PLr has its coordinate axes Px and Py parallel to the X axis and Y axis of the three-dimensional coordinate space.
Since the virtual plane PLr is perpendicular to the Z-axis, the virtual plane PLr is perpendicular to the imaging direction (here, assumed to be the Z-axis direction) when the captured image JL is captured, and thus the captured image (captured image is formed). The horizontal coordinate axis Px and the vertical coordinate axis Py of the virtual plane PLr are parallel to the horizontal coordinate axis Jx and the vertical coordinate axis Jy of the captured image JL, respectively. is there.
However, in the virtual plane PLr, the position of the origin and the direction of the coordinate axis are different from the captured image.

  The relationship between the coordinate axis of the captured image JL and the coordinate axis of the virtual plane PLr will be described with reference to FIGS. 5 (a) and 5 (b). 5A and 5B are views of the captured image JL and the virtual plane PLr of FIG. 4 as viewed from below.

As shown in FIG. 5A, the coordinates on the captured image JL are such that the horizontal coordinate axis Jx extends rightward and the vertical coordinate axis Jy extends downward, with the upper left corner of the captured image JL as the origin.
On the other hand, as shown in FIG. 5B, the coordinates on the virtual plane PLr have the intersection with the Z axis as the origin, the horizontal coordinate axis Px extends leftward, and the vertical coordinate axis Py extends upward. .

Conversion (projection) from the coordinates Jn (Jnx, Jny) of the four vertices on the captured image JL to the coordinates Pn (Pnx, Pny, Pnz) on the virtual plane PLr is expressed by the following equation (1).

In equation (1),
Hs is the horizontal size of the captured image JL, that is, the number of pixels in the horizontal direction.
Vs is the vertical size of the captured image JL, that is, the number of pixels in the vertical direction.
Fw is a focal length (reference focal length) when the zoom lens 12 is in the widest angle state.
Fm is a designated value Fd of the focal length when the image used for designating the mask area is captured.
The correction coefficients Cfx and Cfy are adjustment values determined in advance and will be described later.

  By the calculation of Expression (1), the coordinates of the four vertices J1 (J1x, J1y), J2 (J2x, J2y), J3 (J3x, J3y), and J4 (J4x, J4y) constituting the mask designation area in the captured image JL are obtained. The coordinates P1 (P1x, P1y, 1), P2 (P2x, P2y, 1), P3 (P3x, P3y, 1), and P4 (P4x, P4y, 1) on the virtual plane PLr are obtained. If the mask designation area is limited to a rectangle composed of sides parallel to the horizontal and vertical coordinate axes, P1x = P4x, P2x = P3x, P1y = P2y, and P3y = P4y.

  The mask area registration unit 33 corrects the mask designation area defined by the plane coordinates P1 to P4 of the plurality of vertices, and changes the vertex coordinates of the mask designation area after correction to the coordinates (spherical coordinates) on the virtual spherical surface SP. The converted coordinates are stored in the mask area storage unit 34.

The mask area registration unit 33 includes a designated area correction unit 331 and a spherical coordinate calculation unit 332.
The designated area correction unit 331 corrects the mask designated area so as to make the shape square. This square has the shorter side of the mask designation area before correction as one side.
For such correction, the designated area correction unit 331 firstly determines the coordinates CP = (CPx at the center of the mask designation area from the coordinates Pn (n = 1, 2, 3, 4) of the four vertices constituting the mask designation area. , CPy, 1) is calculated by the following equation (2).
CPx = (P1x + P3x) / 2
CPy = (P1y + P3y) / 2
(2)

Next, the designated area correction unit 331 compares the horizontal side length (horizontal length) of the mask designated area with the vertical side length (vertical length), and based on the comparison result. Make corrections.
For comparison, first, the horizontal length LNX and the vertical length LNY are calculated by the following equation (3) (FIG. 5B). ABS () in the following equation (3) is a process for calculating an absolute value in parentheses.
LNX = ABS (P1x-P3x)
LNY = ABS (P1y-P3y)
(3)

Subsequently, the designated area correction unit 331 compares the horizontal length LNX with the vertical length LNY, and the mask designation area has a square whose side is the shorter of the length LNX and the length LNY. The coordinates Pn of the vertices of the mask designation area are corrected so as to be (FIGS. 6A and 6B). This correction is expressed by the following equation (4). In the following equation (4), the corrected coordinates are indicated by Pc1x to Pc4x and Pc1y to Pc4y.
if (LNX> LNY) {
Pc1x = CPx−LNY / 2
Pc4x = Pc1x
Pc3x = CPx + LNY / 2
Pc2x = Pc3x
} Else {
Pc1y = CPy-LNX / 2
Pc2y = Pc1y
Pc3y = CPy + LNX / 2
Pc4y = Pc3y
}
(4)
Note that the coordinates in the Z-axis direction are not changed by the correction, and the corrected coordinates Pcnz are equal to the uncorrected coordinates Pnz.

  By the above correction, it is possible to prevent a mask area that is long in the horizontal direction or the vertical direction from being set. As a result, since the imaging direction is different between the setting of the mask area and the subsequent imaging, the area to be masked becomes extremely large, thereby preventing the area that is not originally intended to be hidden from being covered. be able to.

The spherical coordinate calculation unit 332 outputs the coordinates Pcn (n = 1, 2, 3, 4) of the four vertices constituting the corrected designated area, which are output from the designated area correction unit 331, to the virtual spherical surface shown in FIG. Project to SP and calculate spherical coordinates Qn (Qnx, Qny, Qnz) (n = 1, 2, 3, 4). This calculation is expressed by equation (5).

  In the above description, the coordinate Qn is assumed on the assumption that the imaging direction when the image used for specifying the mask area is in the reference direction, that is, the Z-axis direction of the three-dimensional coordinate space with the camera unit 14 as the origin. (N = 1, 2, 3, 4) is calculated.

  As shown in FIGS. 8 and 9, when the image capturing direction SDRm at the time of capturing the image used for specifying the mask area is in a direction other than the reference direction (Z-axis direction), that is, the pan angle φm and the tilt angle. When at least one of θm is not zero, the vertex coordinate of the mask area designated on the captured image JL is defined as a virtual plane PLa that is perpendicular to the imaging direction SDRm and touches the virtual spherical surface SP. After projecting onto the virtual plane PLa and correcting the shape of the region to a square, it projects onto the virtual spherical surface SP.

  As described above, the spherical coordinates Rn (Rnx, Rny, Rnz) formed by such projection are the actual imaging with respect to the spherical coordinates Qn obtained on the assumption that the imaging direction is the Z-axis direction. It is obtained by applying rotation according to the pan angle Φm and the tilt angle θm representing the direction SDRm. In that case, the coordinates obtained as a result of the projection from the captured image JL to the virtual plane PL and the projection from the virtual plane PL to the virtual spherical surface PS are projected assuming that the imaging direction matches the Z-axis direction, and then It can be obtained by adding rotation. This rotation is relative, and can be regarded as a process of rotating the Z axis by the same magnitude as the pan angle φm and the tilt angle θm and by the opposite direction. The process of adding rotation is performed by calculating the product of the rotation matrix MTm having the pan angle φm and the tilt angle θm as elements and the matrix having the spherical coordinates Qnx, Qny, and Qnx as elements. This calculation is expressed by the following equation (6).

The rotation matrix MTm is calculated by the following equation (7).

In FIG. 8, coordinates on the virtual plane PLa corresponding to the spherical coordinates Rn (Rnx, Rny, Rnz) after rotation are indicated by a symbol Pdn (Pdnx, Pdny, Pdnz).
The mask area registration unit 33 registers the coordinates Rn in the three-dimensional coordinate space calculated as described above in the mask area storage unit 34 as the coordinates of the four vertices of the set mask area.

  It should be noted that the specified mask area obtained as a result of the correction by the specified area correction unit 331 is converted into a corresponding area on the captured image by the specified area inverse conversion unit 37, and information indicating the area after the reverse conversion is converted into a post-processing unit. 17, information indicating the region after reverse conversion by the post-processing unit 17, for example, a contour line indicating the region after reverse conversion is superimposed on the captured image and displayed on the display device 22, and the display device 22 is viewed. After receiving approval from the user, the above registration is performed in the mask area storage unit 34. The approval by the user is performed using the operation input unit 23, for example.

  In the imaging apparatus, the coordinates of the four vertices of the set mask area registered in the mask area storage unit 34 as described above are read and used at the time of subsequent imaging.

  Hereinafter, the processing in the plane coordinate calculation unit 35 and the processing in the mask processing region calculation unit 36 at the time of imaging after setting the mask region will be described in more detail.

The plane coordinate calculation unit 35 reads the coordinates Rn of the vertex of the set mask area from the mask area storage unit 34 when the imaging apparatus is imaging, and determines the imaging direction SDRv at that time from the imaging condition acquisition unit 31. In response to the indicated information, that is, the pan angle specified value φv and the tilt angle specified value θv, the coordinates Rn of the apex of the setting mask region are converted into coordinates Tn (Tnx, Tny, Tnz). FIG. 10 shows an example of the imaging direction SDRv, the pan angle φv, and the tilt angle θv.
Since the virtual plane TL is perpendicular to the imaging direction SDRv, it can be viewed as being parallel to the captured image UL.

  In the above coordinate conversion, the virtual plane TL perpendicular to the imaging direction SDRv is rotated in the same direction as the pan angle φv and tilt angle θv as shown in FIG. By moving the plane TL in the direction of the Z-axis and simultaneously rotating the coordinate Rn of the apex of the set mask area with the same angle as the pan angle φv and the tilt angle θv, the rotation is reversed on the spherical surface. After the movement, the coordinate Sn after the movement is obtained, the virtual plane TL perpendicular to the imaging direction SDRv is moved by giving the same rotation as described above, and the coordinate Sn after the movement is projected onto the virtual plane TL after the movement. The coordinates Tn on the virtual plane TL are obtained.

  As described above with respect to equation (6), the rotation is relative, and the above processing calculates the coordinate Sn after the rotational movement from the coordinate Rn by rotating the Z axis in the imaging direction SDRm. In the three-dimensional coordinate space in which the Z-axis coincides with the imaging direction SDRm, it can be regarded as a process for converting the spherical coordinate Sn to the planar coordinate Tn.

For the above conversion, the plane coordinate calculation unit 35 gives the above rotation based on the current imaging direction SDRv (pan angle φv and tilt angle θv) of the camera unit 14 output from the imaging condition acquisition unit 31. The rotation matrix MTv is calculated by the following equation (8).

The plane coordinate calculation unit 35 performs an operation for obtaining a product of the coordinates R1 to R4 of the vertexes of the set mask area and the rotation matrix MTv stored in the mask area storage unit 34. This calculation is expressed by the following equation (9).

  As shown in FIG. 11, the coordinates S1 to S4 obtained by this calculation are obtained by moving the current imaging direction SDRv in the Z-axis direction, in other words, the center of the captured image with the Z-axis of the virtual spherical surface SP. This is the coordinates of the vertex of the setting mask area on the virtual spherical surface SP when moved to the intersection.

Further, the plane coordinate calculation unit 35 projects the coordinates S1 to S4 on the virtual spherical surface SP onto the virtual plane TL in the three-dimensional coordinate space shown in FIG.
The virtual plane TL is at the same position as the virtual plane PLr used when setting the mask area. That is, the virtual plane TL is a plane that is perpendicular to the Z axis and is in contact with the virtual spherical surface SP in the three-dimensional coordinate space. Therefore, the virtual plane TL is at the position of Z = 1. The virtual plane TL also has an origin at the intersection with the Z axis, and its coordinate axes Tx and Ty are parallel to the X axis and the Y axis.

Conversion from coordinates Sn (Snx, Sny, Snz) (n = 1, 2, 3, 4) of the four vertices on the virtual sphere to coordinates Tn (Tnx, Tny, Tnz) of the four vertices on the virtual plane TL ( (Projection) is expressed by the following equation (10).
Tnx = Snx / ABS (Snz)
Tny = Sny / ABS (Snz)
Tnz = 1
(N = 1, 2, 3, 4)
(10)

The mask processing area calculation unit 36 calculates the corresponding coordinates Un in the captured image UL from the vertex coordinates Tn on the virtual plane TL calculated by the plane coordinate calculation unit 35, and circumscribes the area defined by the coordinates Un. A rectangular area to be processed is determined as a mask processing area.
The rectangular area referred to here is composed of a side extending in the horizontal direction and a side extending in the vertical direction of the captured image. The “rectangular area circumscribing the area defined by the coordinates Un” is the smallest rectangular area that includes the entire area defined by the coordinates Un within the interior.

  For the above processing, the mask processing area calculation unit 36 first sets the coordinates Tn = (Tnx, Tny, Tnz) (n = 1, 2, 3, 4) on the virtual plane TL to the coordinates Un on the captured image UL. Conversion to (Unx, Uny) (n = 1, 2, 3, 4).

  This conversion is the reverse of the conversion from the coordinates on the captured image JL to the coordinates on the virtual plane PLr described with reference to FIGS. 5 (a) and 5 (b).

As shown in FIG. 12A, the coordinates on the virtual plane TL are centered on the center of the plane (intersection with the Z axis), the horizontal coordinate axis Tx extends leftward, and the vertical coordinate axis Ty faces upward. It extends to.
On the other hand, as shown in FIG. 12B, the coordinates on the captured image UL are such that the horizontal coordinate axis Ux extends rightward and the vertical coordinate axis Uy extends downward with the upper left corner of the captured image UL as the origin. Yes.

As described above, the virtual plane TL can be viewed as being parallel to the captured image UL.
As for the coordinates on the captured image UL, the coordinates of the upper left corner are (0, 0, 1) and the coordinates of the lower right corner (Hs-1, Vs-1, 1). Here, Hs is the horizontal size of the captured image UL, that is, the number of pixels in the horizontal direction, and Vs is the vertical size of the captured image UL, that is, the number of pixels in the vertical direction.

The conversion (projection) of the coordinates of the four vertices of the virtual plane TL to the coordinates on the captured image UL is expressed by the following equation (11).

In the above equation (11),
Fv is a focal length used for imaging the captured image UL,
Fw is a reference focal length, that is, a focal length when the zoom lens 12 is in the widest angle state,
Cfx and Cfy are predetermined correction coefficients.

  The mask processing area calculation unit 36 further obtains the coordinates of the upper left vertex and the lower right vertex of the mask processing area as the start point coordinates (SUx, SUy) and the end point coordinates (EUx, EUy), respectively. .

Specifically, the mask processing area calculation unit 36 uses the coordinates Un (n = 1, 2, 3, 4) of the four vertices of the mask processing area on the captured image UL as shown in the following formula (12). , The horizontal minimum value SUx and the maximum value EUx, the vertical minimum value SUy and the maximum value EUy are obtained, and the combination of the horizontal minimum value and the vertical minimum value is determined as the start point coordinates (SUx , SUy), and the combination of the maximum value in the horizontal direction and the maximum value in the vertical direction is defined as the end point coordinates (EUx, EUy) of the mask processing area.
SUx = min (U1x, U2x, U3x, U4x)
SUy = min (U1y, U2y, U3y, U4y)
EUx = max (U1x, U2x, U3x, U4x)
EUy = max (U1y, U2y, U3y, U4y)
(12)

  The correction coefficients Cfx and Cfy used in the equations (1) and (11) set the imaging direction so that the pan angles φm and φv are 0 degrees, the tilt angles θm and θv are 90 degrees, and the zoom lens 12 Are set as the reference focal length Fw, the entire captured image is set as the mask designation area, and the coordinates Pn (n = 1, 2, 3, 4) output from the designation area conversion unit 32 are When the coordinates Un (n = 1, 2, 3, 4) are calculated without performing correction by the designated area correction unit 331, the start point coordinates (SUx, SUy) of the mask processing area obtained by Expression (12) ) Becomes (0,0), and the end point coordinates become (Hs-1, Vs-1) in advance.

The mask processing area calculation unit 36 further clips the start point coordinates (SUx, SUy) of the mask processing area with the minimum value and clips the end point coordinates (EUx, EUy) with the maximum value.
Both the minimum value in the horizontal direction and the minimum value in the vertical direction are zero.
The maximum value in the horizontal direction is Hs-1, and the maximum value in the vertical direction is Vs-1.
This process is expressed by the following equation (13).
SUbx = min {max (SUx, 0), (Hs-1)}
SUby = min {max (SUy, 0), (Vs-1)}
EUbx = min {max (EUx, 0), (Hs-1)}
EUby = min {max (EUy, 0), (Vs−1)}
(13)

  The start point coordinates (SUbx, SUby) and end point coordinates (EUbx, EUby) of the mask processing area that has undergone the above clipping process are output to the post-processing unit 17 as information MPB indicating the mask processing area. The post-processing unit 17 performs mask processing on the mask processing area defined by the start point coordinates (SUbx, SUby) and the end point coordinates (EUbx, EUby).

  According to the present embodiment, since the imaging direction is different between the area designation and the imaging after the area setting, it is possible to prevent an area that is not originally intended to be hidden from being covered. In addition, the correction of the mask designation area in the designation area correction unit 331 is performed only by the process of comparing the difference between the horizontal length and the vertical length of the rectangle constituting the mask designation area and performing addition / subtraction based on the comparison result. The desired effect can be obtained with few operations.

Embodiment 2. FIG.
In the first embodiment, the designated area correction unit 331 corrects the mask designated area (the area defined by the coordinates Pn calculated by the designated area conversion unit 32) to a square area. Of the horizontal and vertical sides of the designated area, the ratio of the longer length to the shorter length (hereinafter referred to as the aspect ratio) is limited to a predetermined upper limit UR or less. It is also possible to employ a configuration for performing correction so as to do this.
The configuration for correcting the mask designation area to a square described in the first embodiment corresponds to the case where the upper limit value UR is set to 1.

  In this case, among the horizontal side and the vertical side of the rectangular area defined by the coordinates Pn on the virtual plane PL calculated by the specified area conversion unit 32 by the specified area correction unit 331 in FIG. The longer length is changed so that the ratio to the shorter length matches the predetermined upper limit value UR.

Specifically, the designated area correction unit 331 performs the processes shown in FIGS. 13A and 13B instead of the processes shown in FIGS. 6A and 6B.
That is, the horizontal length LNX is compared with the vertical length LNY, and as shown in FIG. 13A, if LNX / LNY> UR, the horizontal length LNX is reduced and corrected. The ratio of the subsequent horizontal length LNXc to the vertical length LNY is made equal to the upper limit value UR. In this case, the position of the center of the rectangular area is not changed. Therefore, correction is performed so that the distance from the center to the left and right sides becomes (LNY / 2) × UR.

  On the other hand, as shown in FIG. 13B, if LNY / LNX> UR, the vertical length LNY is reduced and the corrected vertical length LNYc to the horizontal length LNX is It is made equal to the upper limit value UR. Also in this case, the position of the center of the rectangular area is not changed. LNY / LNX = UR. Therefore, correction is performed so that the distance from the center to the upper limit side becomes (LNX / 2) × UR.

  As described above, even when the mask designation area is a rectangular area other than a square and whose aspect ratio is limited, as described in the first embodiment, at the time of area designation and at the time of imaging after setting the area. Since the image capturing directions are different, it is possible to prevent an area that is not originally intended to be hidden from being hidden.

Embodiment 3 FIG.
FIG. 14 is a diagram showing a configuration of a mask processing apparatus 20b used in the imaging apparatus according to Embodiment 3 of the present invention. The mask processing apparatus 20b in FIG. 14 can be used in the imaging apparatus in FIG. 1 instead of the mask processing apparatus 20 in FIG. 14 is generally the same as the mask processing apparatus 20 in FIG. 3, but the mask area registration unit 33, the mask area storage unit 34, the plane coordinate calculation unit 35, and the mask processing area calculation unit 36 in FIG. In addition, a mask region registration unit 33b, a mask region storage unit 34b, a plane coordinate calculation unit 35b, a mask processing region calculation unit 36b, and a specified region reverse conversion unit 37b are provided instead of the specified region reverse conversion unit 37. It is different. The mask area registration unit 33b includes a designated area correction unit 331b and a spherical coordinate calculation unit 332b.

  As described above, in the first embodiment, when one mask area is designated, one mask area is set correspondingly. Instead of doing this, if the specified mask area is long in the horizontal direction or long in the vertical direction, the mask area is divided into a plurality of rectangular areas (small areas having a limited aspect ratio). ) And the small areas generated by the division are registered as setting mask areas. The rectangle whose aspect ratio is limited here includes a square.

  In order to perform such processing, the designated area correction unit 331b determines that the mask designation area (rectangular area defined by the coordinates Pn on the virtual plane PL calculated by the designated area conversion unit 32) is horizontal or vertical. When the direction is long and the aspect ratio is larger than a predetermined threshold value URc, the mask designation area is divided into a plurality of small rectangular areas with limited aspect ratios, and a plurality of small areas generated by the division are divided. Generate vertex coordinates for each of the regions.

Such processing is performed as follows.
First, the designated area correction unit 331b calculates the horizontal length LNX and the vertical length LNY shown in FIG. 5B by the above equation (3).

Subsequently, the designated area correction unit 331b compares the horizontal length LNX with the vertical length LNY.
Ratio of the shorter one of the length LNX and the length LNY to the longer one (aspect ratio)
MAX (LNX, LNY) / MIN (LNX, LNY)
And whether or not it is larger than the threshold value URc is determined.
When the aspect ratio is equal to or less than the threshold value URc, the coordinates Pn of the apex of the designated area are supplied to the spherical coordinate calculation unit 332 without correcting the designated area.
When the aspect ratio is larger than the threshold value URc, the designated area is divided.

  In this case, the region is divided so that the aspect ratio of the divided region (small region) is equal to or less than a predetermined threshold URd, and the coordinates of each vertex of the small region are supplied to the spherical coordinate calculation unit 332. .

FIG. 15A shows an example of a region Ma having an aspect ratio larger than the threshold value URc.
FIG. 15D shows a plurality of, for example, four small areas Md1 to Md4 formed by dividing the area of FIG.
The same processing as described in the first embodiment is performed on each of the small regions Md1 to Md4 formed by division.
That is, the spherical coordinate calculation unit 332b converts the vertex coordinates Pcn for each of the plurality of small regions generated by the designated region correction unit 331b into coordinates Rn on the virtual spherical surface SP, and stores them in the mask region storage unit 34b. Remember.
As a result, the coordinates Rn on the virtual spherical surface SP for each of the plurality of small areas are registered in the mask area storage unit 34b as the coordinates of the four vertices of the set mask area.

  The small area obtained as a result of the division by the designated area correction unit 331b is converted into a corresponding area on the captured image by the designated area inverse transformation unit 37b, and information indicating the area after the inverse transformation is converted to the post-processing unit 17. The information indicating the region after inverse transformation in the post-processing unit, for example, the contour line indicating the region after inverse transformation is superimposed on the captured image and displayed on the display device 22, and the user viewing the display device 22 After receiving the approval, it is stored in the mask area storage unit 34b.

At the time of imaging after setting the mask area, the coordinates of the vertices of the plurality of set mask areas are read, and a mask processing area is calculated for each.
In other words, the plane coordinate calculation unit 35b reads the coordinates Rn on the virtual spherical surface for each of the plurality of small areas from the mask area storage unit 34b, and converts them to the coordinates Tn on the virtual plane TL.
In this conversion, as described in the first embodiment, rotation and projection of coordinates according to the imaging direction and focal length in the imaging (imaging after setting) are performed.

An example of a region defined by the coordinate Tn after conversion for a plurality of small regions is indicated by reference numerals Me1 to Me4 in FIG. The region shown in FIG. 15E is different in shape from the region shown in FIG. In particular, in FIG. 15D, the four small areas Md1 to Md4 are connected in the horizontal direction, but in FIG. 15E, the areas Me1 to Me4 corresponding to the four small areas Md1 to Md4 are connected in the oblique direction. ing.
This is due to the deformation of the region that occurs as a result of the difference in pan angle or tilt angle between when the mask region is designated and when imaging is performed after setting.

The mask processing area calculation unit 36b converts the coordinates Tn on the virtual plane TL for each of the plurality of small areas obtained by the conversion in the plane coordinate calculation unit 35b into coordinates Un in the captured image UL. A plurality of rectangular areas circumscribing each of the plurality of small areas defined by the coordinates Un obtained by the conversion are calculated as a plurality of mask processing areas.
An example of a plurality of mask processing areas is indicated by symbols Mf1 to Mf4 in FIG.

  If the region corresponding to the mask region Ma that is long in the horizontal direction is stored in the mask region storage unit as it is without dividing the region by the designated region correction unit 331b, as in the case described above, Due to the difference in the imaging direction from the time of imaging after setting, the mask area represented by the vertex coordinates Tn on the image obtained by imaging after setting becomes as indicated by the symbol Mb in FIG. 15B. In this case, the mask processing area constituted by a rectangle circumscribing the area shown in FIG. 15B becomes extremely large as indicated by reference numeral Mc in FIG.

  As can be seen from a comparison between FIG. 15C and FIG. 15F, the mask processing area Mc shown in FIG. 15C includes a plurality of mask processing areas Mf1 to Mf4 shown in FIG. It is wider than the region where all of the above are combined, and includes many regions Mca that are not originally intended to be hidden.

  In the present embodiment, the combination of the mask processing areas Mf1 to Mf4 at the time of imaging after setting has an elongated shape as shown in FIG. 15 (f), and the mask processing area shown in FIG. 15 (c). It can be avoided that it becomes large like Mc.

  In the above example, equal division is performed so that all parts of the designated area belong to one of the divided small areas. For example, the equal division can be performed by setting both the thresholds URc and URd to be equal to √2≈1.4 and determining the number of divisions as follows.

First, the maximum integer N that does not exceed the above aspect ratio (MAX (LNX, LNY) / MIN (LNX, LNY) is obtained.
next,
(MAX (LNX, LNY) / {MIN (LNX, LNY) × N}
And {MIN (LNX, LNY) × (N + 1)} / (MAX (LNX, LNY)
If the former is less than the latter, the designated area is divided into N. Otherwise, the designated area is divided (N + 1). In this way, the designated area can be equally divided so that all of its parts belong to any of the divided small areas, and the aspect ratio of the small areas formed by the division can be minimized. .

  As described above, the plurality of rectangular regions with limited aspect ratios may be square. If the rectangular area is a square area, all parts of the original specified area will be divided if the longer side of the specified area before division is an integral multiple of the shorter side length. It belongs to only one of the subsequent small areas.

  If the length of the longer side of the designated area before division is not an integral multiple of the length of the shorter side, as shown in FIG. 16 (a) or (b), a part of the original designated area is It may be configured not to belong to any of the divided small areas. As shown in FIG. 16C, a part of the original designated area belongs to the adjacent small areas after the division. You may make it happen.

In the example shown in FIG. 16A, both end portions Mde of the original elongated designated area Ma are cut off (excluded) when dividing into a plurality of square small areas Md1 to Md4.
In the example shown in FIG. 16B, when dividing into a plurality of square small areas Md1 to Md4, a gap Mdg is opened between the small areas Md1 to Md4, and a portion located in the gap Mdg is cut off (excluded). ing).
In the example shown in FIG. 16C, the end portions of the adjacent small regions Md1 to Md5 overlap each other when dividing into a plurality of square small regions Md1 to Md5. In FIG. 16C, a portion where adjacent small regions overlap each other is indicated by a symbol Mdo.

  Note that when the designated area is a rectangular small area other than a square and whose aspect ratio is limited, as in the above, part of the original designated area does not belong to any of the divided small areas. Alternatively, a part of the original designated area may belong to an adjacent small area after being divided.

  By dividing the designated area as described above, it becomes possible to prevent a long mask area from being set in the horizontal direction or the vertical direction, and the imaging direction differs between the setting of the mask area and the subsequent imaging. In addition, the area to be masked becomes extremely large, so that it is possible to prevent the area that is not originally intended to be hidden from being obscured.

Note that the configuration features of the third embodiment and the configuration features of the second embodiment may be combined. That is, in the above example, if the aspect ratio is equal to or less than the threshold value URc, the coordinates Pn are output to the spherical coordinate calculation unit 332 as they are. With this configuration, the threshold value URc is set to a value larger than the upper limit value UR of Embodiment 2, and if the aspect ratio is equal to or less than the threshold value URc, it is next determined whether or not it is larger than the upper limit value UR. When the aspect ratio is larger than the threshold value URc, the designated area is divided as described above. On the other hand, if the aspect ratio is larger than the upper limit value UR, the shape of the designated area is changed so that the aspect ratio is equal to or less than the upper limit value UR. This process is the same as that described in the second embodiment.
In this way, an optimal correction can be applied according to the shape of the designated area.

Embodiment 4 FIG.
FIG. 17 is a diagram showing a configuration of a mask processing apparatus 20c used in the imaging apparatus according to Embodiment 4 of the present invention. A mask processing apparatus 20c in FIG. 17 can be used in the imaging apparatus in FIG. 1 instead of the mask processing apparatus 20 in FIG. The mask processing apparatus 20c of FIG. 17 is generally the same as the mask processing apparatus 20 of FIG. 3, but differs in that a mask processing area calculation unit 36c is provided instead of the mask processing area calculation unit 36 of FIG. .

  The mask processing area calculation unit 36c corrects the start point coordinates (SUx, SUy) and the end point coordinates (EUx, EUy) of the mask processing area calculated by the process expressed by the above formula (12) as necessary. Add

This correction is a correction for making the mask processing area square. Specifically, the length of the side in the horizontal direction is compared with the length of the side in the vertical direction, and correction is performed to change the length of the shorter side to the length of the longer side. Examples of such correction are shown in FIGS.
For this processing, first, a difference DLN between the horizontal side length and the vertical side length is calculated by the following equation (14).
DLN = (EUx−SUx) − (EUy−SUy) (14)

In the example shown in FIG. 18, the length in the horizontal direction is longer, and therefore the difference DLN is greater than zero. In this case, vertical correction is performed as shown in the following equation (15). However, no correction is performed when the vertical start point coordinate SUy is less than 0 or the end point coordinate EUy is greater than Vs-1. In the following equation (15), the corrected SUx, SUy, EUx, EUy are represented by SUcx, SUcy, EUcx, EUcy.
SUcy = SUy-DLN / 2
SUcx = SUx
EUcy = EUy + DLN / 2
EUcx = EUx
(15)

In the example shown in FIG. 19, the length in the vertical direction is longer, and therefore the difference DLN is smaller than zero. In this case, correction in the horizontal direction is performed as shown by the following equation (16). However, no correction is performed when the horizontal start point coordinate SUx is less than 0 or the end point coordinate EUx is greater than Hs-1. In the following equation (16), the corrected SUx, SUy, EUx, EUy are represented by SUcx, SUcy, EUcx, EUcy.
SUcx = SUx + DLN / 2
SUcy = SUy
EUcx = EUx-DLN / 2
EUcy = EUy
(16)

Further, the mask processing area calculation unit 36 clips the start point coordinates (SUcx, SUcy) of the mask processing area with the minimum value and clips the end point coordinates (EUcx, EUcy) with the maximum value.
Both the minimum value in the horizontal direction and the minimum value in the vertical direction are zero.
The maximum value in the horizontal direction is Hs-1, and the maximum value in the vertical direction is Vs-1.
This process is expressed by the following equation (17).
SUdx = min {max (SUcx, 0), (Hs-1)}
SUdy = min {max (SUcy, 0), (Vs-1)}
EUdx = min {max (EUcx, 0), (Hs-1)}
EUdy = min {max (EUcy, 0), (Vs−1)}
(17)

  The start point coordinates (SUdx, SUdy) and the end point coordinates (EUdx, EUdy) of the mask processing area that has undergone the above clipping process are output to the post-processing unit 17. The post-processing unit 17 performs mask processing on the mask area defined by the start point coordinates (SUdx, SUdy) and the end point coordinates (EUdx, EUdy).

  As described above, the mask processing area calculation unit 36c compares the length of the horizontal side of the rectangle in the mask area with the length of the vertical side after the pan rotation of the camera unit 14 and the tilting operation. Since the correction is made to a square with one side as one side, the mask area is slightly larger than the case where correction is not performed, and an effect of more reliably covering the vicinity of the boundary of the mask area can be obtained. .

  Further, since the mask processing area calculation unit 36c is configured to correct the mask area after the tilting operation after the pan rotation of the camera unit 14 to a square, the mask area maintaining the shape of the square when the mask area is set is selected. It can be provided to the user, and an effect of not causing the user to feel uncomfortable due to the shape mismatch is obtained.

  In the above example, the corrected mask processing area is square, but the aspect ratio (the length in the horizontal direction and the length of the mask processing area is similar to that described in the second embodiment. Modification may be made so that the ratio of the larger one of the lengths in the vertical direction to the smaller one is equal to or less than a predetermined upper limit value.

  The features of the mask processing region calculation unit 36c described in the fourth embodiment may be combined with the features described in the third embodiment. That is, when the mask designation area is divided into a plurality of small areas as in the third embodiment, each of the plurality of mask processing areas generated corresponding to the plurality of small areas is described in the fourth embodiment. As described above, the shape may be corrected. The shape correction is not limited to the deformation to the square, but may be a correction other than the square so that the rectangle has a limited aspect ratio.

  Although the present invention has been described as a mask processing apparatus and an imaging apparatus including the mask processing apparatus, the mask processing method implemented by the mask processing apparatus and the imaging method implemented by the imaging apparatus are also one aspect of the present invention. Form part.

  A part or all of the processing performed in part or all of the configuration of the mask processing apparatus or the imaging apparatus, or part or all of the processing performed in the mask processing method or the imaging method can be executed by a computer including a processor. . Therefore, a program for causing a computer to execute part or all of the processing performed in part or all of the configuration of the mask processing apparatus or imaging apparatus, and the processing performed in the mask processing method or imaging method, and A computer-readable recording medium on which the program is recorded also forms part of the present invention.

An example of the computer is shown in FIG. The computer in FIG. 20 executes the functions of the post-processing unit 17, the control unit 19, and the mask processing apparatus 20 in FIG.
A processor 51, a program memory 52, a data memory 53, an image input interface 54, a control input interface 55, and an output interface 56 are provided, and these are connected by a data bus 57.
For example, a digital image signal Db output from the preprocessing unit 15 of FIG. 1 is supplied to the image input interface 54.
The control input interface 55 is supplied with mask region designation information MRD from the operation input unit 23.

  The processor 51 operates in accordance with a program stored in the program memory 52 and, as described in the first, second, third, or fourth embodiment, based on the mask area designation information MRD input via the control input interface 55. The same processing as that of the control unit 19 and the mask processing device 20, 20b, or 20c is performed, and the same processing as that of the post-processing unit 17 is performed on the captured image Db supplied from the image input interface 54.

That is, as the same processing as the control unit 19, the rotational drive mechanism 11 and the camera unit 14 are controlled, and the mask region designation information MRD is output. Calculation is performed, and mask processing for the captured image Db is performed as the same processing as the post-processing unit 17.
The image Dc generated as a result of the mask process is output from the output interface 56 and supplied to the display device 22, for example.

  The contents of processing by the processor 51 are the same as those described in the first, second, third, or fourth embodiment. Data generated in the course of processing is held in the data memory 53.

  Note that a plurality of computers shown in FIG. 20 may be provided, and each unit may perform processing of each part of the mask processing apparatus or the imaging apparatus. The same applies to a case where a part or all of the mask processing method or the imaging method is executed by a computer.

  DESCRIPTION OF SYMBOLS 11 Rotation drive mechanism, 12 Zoom lens, 13 Imaging device, 14 Camera part, 15 Pre-processing part, 16 Imaging part, 17 Post-processing part, 18 Video signal output terminal, 19 Control part, 20 Mask processing apparatus, 31 Acquisition of imaging conditions 32, designated area conversion section, 33, 33b mask area registration section, 34, 34b mask area storage section, 35, 35b plane coordinate calculation section, 36, 36b, 36c mask processing area calculation section, 37, 37b reverse conversion of specified area 331, 331b designated area correction unit, 332, 332b spherical coordinate calculation unit.

Claims (20)

A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image A mask processing apparatus for an imaging apparatus, comprising: a camera unit including: a rotation driving mechanism that performs pan driving and tilt driving with respect to the camera unit;
An imaging condition acquisition unit that acquires direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion unit for converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction unit that changes the shape of the rectangular area and generates the coordinates of the vertices of the rectangular area generated by the change;
A spherical coordinate calculation unit that converts the coordinates of the vertex generated by the specified region correction unit into coordinates on a virtual sphere,
A mask area storage unit for storing coordinates on the virtual sphere calculated by the spherical coordinate calculation unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation unit that reads coordinates on the virtual sphere and converts them into coordinates on a second virtual plane;
The coordinates on the second virtual plane obtained by the conversion in the plane coordinate calculation unit are converted into coordinates in the second captured image, and circumscribed in the area defined by the coordinates obtained by the conversion. A mask processing area calculation unit that calculates a rectangular area to be used as a mask processing area,
The designated area correction unit matches a longer length with a shorter length of a horizontal side and a vertical side of a rectangular region defined by coordinates on the first virtual plane. The mask processing apparatus is characterized by being modified as follows. The designated area correction unit, when changing the length of the longer of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane, The mask processing apparatus according to claim 1, wherein a position of a center of the rectangular mask designation area defined by coordinates on the first virtual plane is maintained. The mask processing area calculation unit has a ratio of a longer length to a shorter length of a horizontal side and a vertical side of the rectangular mask processing area equal to or less than a predetermined threshold value. as such, mask processing apparatus according to claim 1 or 2, characterized in that changing at least one length of the horizontal side length and the vertical sides. The mask processing area calculation unit is configured to change a shorter length of a horizontal side and a vertical side of the rectangular mask processing area so as to coincide with a longer length. The mask processing apparatus according to claim 3 . The mask processing region calculation unit maintains the position of the center of the rectangular mask processing region when changing the length of a horizontal side or a vertical side of the rectangular mask processing region. The mask processing apparatus according to claim 3 or 4 . A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image A mask processing apparatus for an imaging apparatus, comprising: a camera unit including: a rotation driving mechanism that performs pan driving and tilt driving with respect to the camera unit;
An imaging condition acquisition unit that acquires direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion unit for converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction unit that divides a rectangular area into a plurality of rectangular small areas and generates coordinates of each vertex of the plurality of small areas;
A spherical coordinate calculation unit that converts the coordinates of the vertex for each of the plurality of small regions generated by the designated region correction unit into coordinates on a virtual sphere,
A mask area storage unit for storing coordinates on the virtual sphere for each of the plurality of small areas calculated by the spherical coordinate calculation unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation unit that reads coordinates on the virtual sphere for each of the plurality of small regions and converts the coordinates to coordinates on a second virtual plane;
The coordinates on the second virtual plane for each of the plurality of small regions obtained by the conversion in the plane coordinate calculation unit are converted into coordinates in the second captured image, and obtained by this conversion. A mask processing area calculation unit that calculates a plurality of rectangular areas circumscribing each of the plurality of small areas defined by the given coordinates as a plurality of mask processing areas;
The designated area correction unit divides a rectangular area defined by coordinates on the first virtual plane into a plurality of rectangular small areas,
The division is performed so that, in each of the small regions, a ratio of a longer length to a shorter length of a horizontal side and a vertical side is equal to or less than a predetermined threshold value. A mask processing apparatus. The mask processing apparatus according to claim 6 , wherein the plurality of rectangular small areas are square areas. The mask processing region calculation unit determines in advance a ratio of a longer length to a shorter length of a horizontal side and a vertical side of each of the plurality of rectangular mask processing regions. The mask processing apparatus according to claim 6, wherein at least one of the horizontal side length and the vertical side length is changed so as to be equal to or less than a predetermined threshold value. For each of the plurality of rectangular mask processing regions, the mask processing region calculation unit is configured so that the shorter one of the horizontal side and the vertical side matches the longer side. The mask processing apparatus according to claim 8 , wherein the mask processing apparatus is changed to: The mask processing area calculation unit determines the position of the center of the rectangular mask processing area when changing the length of the horizontal side or the vertical side of each of the plurality of rectangular mask processing areas. The mask processing apparatus according to claim 8 , wherein the mask processing apparatus is maintained. The first virtual plane and the second virtual plane are at the position of the captured image when the focal length of the zoom lens is a reference focal length,
The virtual spherical surface is a virtual spherical surface having a radius of 1 with the position of the camera unit as the origin in a three-dimensional coordinate space,
The first virtual plane is a plane perpendicular to the first imaging direction and in contact with the virtual spherical surface;
The second virtual plane is a plane perpendicular to the second imaging direction and in contact with the virtual spherical surface,
The spherical coordinate calculation unit projects the vertex coordinates generated by the designated region correction unit on the first virtual plane onto the virtual spherical surface, and rotates by an angle corresponding to the first imaging direction. To obtain the coordinates on the virtual sphere,
The plane coordinate calculation unit rotates the coordinates on the virtual sphere by an angle that is the same as the angle corresponding to the second imaging direction and has the opposite direction, and projects it onto the second virtual plane. it is, mask processing apparatus according to any one of claims 1 to 10, and calculates the coordinates on the second virtual plane. The mask processing apparatus according to any one of claims 1 to 5 and 11 ,
An image processing apparatus comprising: an image processing unit that masks and outputs a portion corresponding to the mask processing region calculated by the mask processing region calculation unit of the captured image generated by the camera unit. The mask processing apparatus according to any one of claims 6 to 11 ,
An image processing unit that masks and outputs portions corresponding to the plurality of mask processing regions calculated by the mask processing region calculation unit out of the captured images generated by the camera unit. apparatus. A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image A mask processing method implemented by an imaging apparatus having a camera unit including a rotation driving mechanism that performs pan driving and tilt driving with respect to the camera unit,
An imaging condition acquisition step of acquiring direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion step of converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction step for changing the shape of the rectangular area and generating the coordinates of the vertices of the rectangular area generated by the change;
A spherical coordinate calculation step for converting the coordinates of the vertex generated in the specified region correction step into coordinates on a virtual sphere,
A storage step of storing the coordinates on the virtual sphere calculated in the spherical coordinate calculation step in a mask area storage unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation step of reading coordinates on the virtual sphere and converting them to coordinates on a second virtual plane;
The coordinates on the second virtual plane obtained by the conversion in the plane coordinate calculating step are converted into coordinates in the second captured image, and circumscribed in the area defined by the coordinates obtained by the conversion. And a mask processing area calculation step for calculating a rectangular area to be used as a mask processing area,
In the designated area correction step, the longer one of the horizontal side and the vertical side of the rectangular area defined by the coordinates on the first virtual plane is made equal to the shorter one. The mask processing method characterized by changing as follows. The mask processing method according to claim 14 ;
An image processing method comprising: an image processing step of masking and outputting the mask processing region calculated in the mask processing region calculation step in the captured image generated by the camera unit. A zoom lens that can change a focal length, collects light from a subject to form an optical image, and an imaging device that converts the optical image formed by the zoom lens into an electrical signal and outputs a captured image A mask processing method implemented by an imaging apparatus having a camera unit including a rotation driving mechanism that performs pan driving and tilt driving with respect to the camera unit,
An imaging condition acquisition step of acquiring direction information indicating an imaging direction of the camera unit and focal length information indicating a focal length of the zoom lens;
The camera unit is designated as a mask region in a first captured image obtained by performing imaging with the focal length of the zoom lens as the first focal length and the imaging direction as the first imaging direction. A designated area conversion step of converting the coordinates of the rectangular area into coordinates on the first virtual plane;
When the ratio of the longer side to the shorter side of the horizontal side and the vertical side of the rectangular region defined by the coordinates on the first virtual plane is larger than a predetermined upper limit value, A designated area correction step of dividing a rectangular area into a plurality of rectangular small areas and generating coordinates of vertices of each of the plurality of small areas;
A spherical coordinate calculation step for converting the coordinates of the vertex for each of the plurality of small regions generated in the specified region correction step into coordinates on a virtual sphere;
A storage step of storing coordinates on the virtual sphere for each of the plurality of small regions calculated in the spherical coordinate calculation step in a mask region storage unit;
When the camera unit generates a second captured image by performing imaging with the focal length of the zoom lens as the second focal length and the imaging direction as the second imaging direction, from the mask area storage unit A plane coordinate calculation step of reading coordinates on the virtual spherical surface for each of the plurality of small regions and converting the coordinates to coordinates on a second virtual plane;
The coordinates on the second virtual plane for each of the plurality of small areas obtained by the conversion in the plane coordinate calculation step are converted into coordinates in the second captured image, and obtained by this conversion. A mask processing area calculating step for calculating a plurality of rectangular areas circumscribing each of the plurality of small areas defined by the coordinates as a plurality of mask processing areas,
The designated area correction step divides a rectangular area defined by coordinates on the first virtual plane into a plurality of rectangular small areas;
The division is performed so that, in each of the small regions, a ratio of a longer length to a shorter length of a horizontal side and a vertical side is equal to or less than a predetermined threshold value. A mask processing method characterized by the above. The mask processing method according to claim 16 ,
An image processing step of masking and outputting portions corresponding to the plurality of mask processing regions calculated in the mask processing region calculation step of the captured image generated by the camera unit. Method. The program for making a computer perform the process of each step of the mask processing method of Claim 14 or 16 . The program for making a computer perform the process of each step of the imaging method of Claim 15 or 17 . A computer-readable recording medium on which the program according to claim 18 or 19 is recorded.

Images