JP2020065218A - Image processing program, image processing apparatus, image processing system, and image processing method - Google Patents

Abstract

To suppress a decrease in reliability due to false detection of an object when an image is captured by using an imaging device capable of remotely controlling the angle of view.SOLUTION: An image processing apparatus 20 acquires an image P1 from a camera 10 (S1) and sets a non-detection area A1 for the image P1 (S2). After that, the image processing apparatus 20 controls to change the angle of view of the camera 10 (S3) and acquires an image P2. The image processing apparatus 20 can calculate a non-detection area A2 after changing the angle of view corresponding to the non-detection area A1 (S4), and detects an object by applying the non-detection area A2 to the image P2.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing program, an image processing device, an image processing system, and an image processing system capable of suppressing a decrease in reliability due to erroneous detection of an object when an image is captured using an imaging device capable of controlling an angle of view. The present invention relates to an image processing method.

  Conventionally, in many cases, a fixed surveillance camera is installed at a predetermined position, and an image of an object captured by the surveillance camera is transmitted to an image processing apparatus. At this time, in order to control the viewing angle of the surveillance camera, a pan control for moving the angle of view of the camera left and right, a tilt control for moving the angle of view of the camera up and down, and enlargement / reduction of the angle of view of the camera. The zoom control is performed.

Therefore, a technique is known in which a target area in an image is moved according to pan, tilt, and zoom control of a camera (hereinafter referred to as “PTZ control”). For example, Japanese Patent Application Laid-Open No. 2004-242242 discloses a technique of cutting out an image of a target area in an image frame and generating an output image by moving the target area in a direction opposite to the movement of the camera and cutting out the target area. By using this prior art, it becomes possible to set the target area in the image of the fixed surveillance camera and move the target area when the PTZ control of the surveillance camera is performed.

JP, 2005-179988, A

  However, the image captured by the surveillance camera may include an object other than the detection target. For example, when the plant grows and affects the field of view of the surveillance camera, the shaking of the plant is erroneously detected as if it were a moving body. For this reason, even if the above-mentioned Patent Document 1 is used, a target region of an unintended object is cut out, which causes a problem that reliability is lowered.

  In particular, when the surveillance camera is PTZ-controlled, the area to be originally excluded from the detection target increases, so that the false detection of the object may increase. For this reason, when capturing an image using a PTZ controllable surveillance camera, how to suppress a decrease in reliability due to erroneous detection of an object is an important issue.

  The present invention has been made in order to solve the above-mentioned problems (problems) of the prior art, and accompanies erroneous detection of an object when an image is captured using an imaging device capable of remotely controlling the angle of view. An object of the present invention is to provide an image processing program, an image processing device, an image processing system, and an image processing method capable of suppressing deterioration of reliability.

  In order to solve the above problems, the present invention is an image processing program in an image processing apparatus that performs image processing of an image captured by an image capturing apparatus capable of remotely controlling an angle of view, which is a non-detection region for the image. A non-detection area setting procedure for setting the non-detection area and a non-detection area after changing the view angle corresponding to the non-detection area set by the non-detection area setting procedure when the view angle of the imaging device is remotely controlled. And a calculation procedure to be executed by a computer.

  Further, the present invention, in the above-mentioned invention, further causes the computer to execute a first object detection procedure of detecting an object with an area excluding the non-detection area after the change of the angle of view calculated by the calculation procedure as a target area. It is characterized by

  Further, the invention is characterized in that, in the above-mentioned invention, a computer is further caused to execute an angle-of-view control procedure for performing pan, tilt, and zoom control of the imaging device.

  Further, the present invention is characterized in that, in the above-mentioned invention, the computer is further caused to execute a non-detection area correction procedure for correcting the non-detection area after the view angle change calculated by the calculation procedure.

  The present invention, in the above invention, further causes a computer to execute a second object detection procedure of detecting an object with an area excluding the non-detection area corrected by the non-detection area correction procedure as a target area. Characterize.

  Further, in the present invention according to the above-mentioned invention, the imaging device can image in all directions, and the calculation procedure converts two-dimensional coordinates in the image of the non-detection region into polar coordinates centered on the imaging device. The two-dimensional coordinates of the non-detection area after the change of the angle of view are calculated from the polar coordinates.

  Further, the present invention is an image processing apparatus that performs image processing of an image captured by an image capturing apparatus capable of remotely controlling an angle of view, and a non-detection area setting unit that sets a non-detection area for the image, And a calculator that calculates a non-detection area after changing the angle of view corresponding to the non-detection area set by the non-detection area setting unit when the angle of view of the imaging device is remotely controlled. And

  The present invention also provides an image processing system in which an imaging device and an image processing device that remotely controls an angle of view of the imaging device and performs image processing of an image captured by the imaging device are communicably connected. The image processing apparatus includes a view angle control unit that controls a view angle of the image pickup apparatus, a non-detection area setting unit that sets a non-detection area for the image, and a view angle of the image pickup apparatus is remotely controlled. In this case, a calculation unit that calculates a non-detection region after changing the angle of view, which corresponds to the non-detection region set by the non-detection region setting unit, is provided.

  Further, the present invention is an image processing method in an image processing device for performing image processing of an image captured by an imaging device capable of remotely controlling an angle of view, the non-detection region having a non-detection region set for the image. A setting step, and a calculation step of calculating a non-detection area after changing the angle of view corresponding to the non-detection area set in the non-detection area setting step when the angle of view of the imaging device is remotely controlled. It is characterized by

  According to the present invention, when an image is captured by using an image capturing device capable of remotely controlling the angle of view, it is possible to suppress a decrease in reliability due to erroneous detection of an object.

FIG. 1 is an explanatory diagram of the concept of the image processing system according to the first embodiment. FIG. 2 is a configuration diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 3 is a flowchart illustrating the processing procedure of the image processing apparatus according to the first embodiment. FIG. 4 is a flowchart showing details of the object detection process shown in FIG. FIG. 5 is an explanatory diagram of the imaging range of the camera and the lattice. FIG. 6 is an explanatory diagram of the angle of view of the camera. FIG. 7 is an explanatory diagram regarding management of coordinates by three-dimensional coordinates. FIG. 8 is an explanatory diagram of the concept of the image processing system according to the second embodiment. FIG. 9 is an explanatory diagram showing the correspondence relationship between the non-detection region and the lattice. FIG. 10 is an explanatory diagram of the correction of the non-detection area. FIG. 11 is a configuration diagram illustrating the configuration of the image processing apparatus according to the second embodiment. FIG. 12 is a flowchart illustrating the processing procedure of the image processing apparatus according to the second embodiment. FIG. 13 is an explanatory diagram of correction of the non-detection area in the modification. FIG. 14 is an explanatory diagram of correction after changing the angle of view in the modified example. FIG. 15 is a diagram illustrating an example of the hardware configuration.

  Embodiments of an image processing program, an image processing apparatus, an image processing system, and an image processing method will be described below with reference to the accompanying drawings.

  First, the concept of the image processing system according to the first embodiment will be described. FIG. 1 is an explanatory diagram of the concept of the image processing system according to the first embodiment. As shown in FIG. 1, the image processing system according to the first embodiment performs image processing of a camera 10 which is an imaging device, a remote control of an angle of view of the camera 10, and an image processing of an image captured by the camera 10. It has a configuration in which the device 20 is communicably connected.

  In FIG. 1, the camera 10 is installed so as to be able to capture an image of a road, and a row of trees along the road may be included in the image capturing range. The image processing device 20 rotates the camera 10 in the horizontal direction to move the angle of view left and right, pan control, rotates the camera 10 in the vertical direction to move the angle of view up and down, and expands the angle of view of the camera 10. Zoom control that reduces the size is possible.

  Further, the image processing device 20 detects a predetermined object by performing image processing on the image captured by the camera 10. For example, the image processing device 20 can detect a car or an animal on the road. Here, the image processing device 20 can set a non-detection region for the image. Specifically, by setting the position of the image of the tree in the image in the non-detection area, it is possible to detect the object by excluding the position of the tree.

  By setting the non-detection area for the image, it is possible to avoid detection of objects other than the detection target. However, if the angle of view of the camera 10 is changed, the coordinates of the non-detection area in the image before the change cannot be used as they are.

  Therefore, the image processing apparatus 20 converts the non-detection area set for the image into polar coordinates centered on the camera 10 and manages the same, and also manages the imaging direction of the camera 10. Then, when an object is detected, it is determined whether or not the detected object is located in the non-detection area by using the position in the image of the object and the imaging direction of the camera 10, and the object is located in the non-detection area. In this case, it is assumed that no object was detected. In other words, when the angle of view of the camera 10 is changed, the image processing apparatus 20 detects the object by calculating the changed non-detection area. The image capturing direction of the camera 10 can be managed by comparing the image captured by the camera 10 with the immediately preceding image for each frame and calculating the amount of change in the image capturing direction.

  In FIG. 1, the image processing apparatus 20 acquires the image P1 from the camera 10 (S1) and sets the non-detection area A1 for the image P1 (S2). After that, the image processing apparatus 20 changes the angle of view for panning control of the camera 10 to the right (S3), and acquires the image P2.

  The image processing apparatus 20 converts the non-detection area A1 set in the image P1 into polar coordinates centered on the camera 10 and manages the non-detection area A2 after changing the angle of view corresponding to the non-detection area A1. It can be calculated (S4). Therefore, it is possible to detect the object by applying the non-detection area A2 to the image P2. Details of the process of calculating the non-detection area after the change of the angle of view will be described later.

  As described above, the image processing device 20 sets the non-detection area for the image captured by the camera 10, and when the angle of view of the camera 10 is remotely controlled, the image processing apparatus 20 sets the non-detection area before changing the angle of view. A corresponding non-detection area after changing the angle of view is calculated. Therefore, when an image is captured using the camera 10 whose angle of view can be remotely controlled, it is possible to suppress a decrease in reliability due to erroneous detection of an object.

  Next, the configuration of the image processing device 20 will be described. FIG. 2 is a configuration diagram showing the configuration of the image processing apparatus 20. As shown in FIG. 2, the image processing device 20 includes a communication unit 21, a display unit 22, an input unit 23, a storage unit 24, and a control unit 25. The communication unit 21 is a communication interface for making a communication connection with the camera 10 and the like. The display unit 22 is a display device such as a liquid crystal display device. The input unit 23 is an input device such as a keyboard and a mouse.

  The storage unit 24 is a storage device such as a hard disk device or a non-volatile memory, and stores the non-detection area data 24a. The non-detection area data 24a is data that specifies the non-detection area set for the image. Specifically, the non-detection area data 24a indicates whether or not each grid obtained by dividing the celestial sphere centering on the camera 10 is set in the non-detection area. As for the method of dividing the celestial sphere, the angle in the pan direction is the longitude and the angle in the tilt direction is the latitude, and the longitude and latitude are divided at equal intervals. The dividing method is not limited to this, and any method can be used.

  The control unit 25 is a control unit that controls the entire image processing apparatus 20, and includes an angle-of-view control unit 25a, a non-detection area setting unit 25b, a calculation unit 25c, and an object detection unit 25d. Actually, the programs corresponding to these functional units are stored in a ROM or a non-volatile memory (not shown), and these programs are loaded into the CPU and executed to execute the view angle control unit 25a and the non-detection area setting. The processes corresponding to the unit 25b, the calculation unit 25c, and the object detection unit 25d will be executed.

  The view angle control unit 25a is a processing unit that remotely controls the view angle of the camera 10. Specifically, the view angle control unit 25a controls pan, tilt, and zoom of the camera 10. The control of the angle of view by the angle-of-view control unit 25a may be performed by accepting an operation by an operator or may be automatically performed. For example, the angle of view of the camera 10 may be set to change at regular intervals or cycles. Further, the angle of view may be controlled based on the detection result or the like.

  Further, the view angle control unit 25a manages the image capturing direction of the camera 10 by comparing the image captured by the camera 10 with the immediately preceding image for each frame and calculating the amount of change in the image capturing direction. The imaging direction of the camera 10 is indicated by the pan angle and tilt angle from the reference position. The angle-of-view control unit 25a updates the image capturing direction by adding the amount of change to the image capturing direction up to that point, after calculating the amount of change in the image capturing direction by comparison with the immediately preceding image. The image capturing direction may be held by the image control unit 25a or may be stored in the storage unit 24.

  The non-detection area setting unit 25b is a processing unit that sets a non-detection area for an image captured by the camera 10. When the non-detection area is set for the image, the non-detection area setting unit 25b stores the data in the celestial sphere at a position corresponding to the angle of view of the current image, and the non-detection area data 24a. To update. At this time, the data of the grid of the celestial sphere other than the set position such as outside the angle of view is maintained.

  The calculation unit 25c is a processing unit that calculates the non-detection region after changing the view angle corresponding to the non-detection region set by the non-detection region setting unit 25b. Specifically, the calculation unit 25c uses the imaging direction and zoom magnification of the camera 10 and the non-detection area data 24a to project a grid of the non-detection area on the current image to determine the position of the non-detection area in the image. It can be calculated.

  Here, the non-detection area is used to exclude the detection result of the object. Therefore, instead of calculating the position of the non-detection area in the image every time the angle of view is changed, it is determined whether or not the position of the object is within the non-detection area when the object is detected from the image. May be In this case, the calculation unit 25c converts the position of the object in the image into polar coordinates centered on the camera 10 and compares the polar coordinates of the conversion result with the non-detection area indicated by the non-detection area data 24a. Become. In the first embodiment, a description will be mainly given of a configuration that determines whether or not the position of the object is within the non-detection region when the object is detected from the image.

  The object detection unit 25d performs a process of detecting an object from the image captured by the camera 10. At this time, the object detection unit 25d sets the area excluding the non-detection area as the detection target area. Specifically, the object detection unit 25d identifies whether or not the target object exists in the image by image processing, and when the target object exists, the position in the image is calculated by the calculation unit 25c. It is output to determine whether the position of the object is within the non-detection area. Then, when the target object is identified from the image and the position of the object is not in the non-detection region, the object detection unit 25d outputs the detection result that the object is detected. For example, the object detection unit 25d can output a notification when a predetermined object is detected. In addition, the object detection unit 25d can store the detection result of the object in the storage unit 24.

  Next, the processing procedure of the image processing apparatus 20 will be described. FIG. 3 is a flowchart showing a processing procedure of the image processing apparatus 20. First, the angle-of-view control unit 25a of the image processing apparatus 20 acquires an image from the camera 10 (step S101), calculates the amount of change in the imaging direction by comparing it with the immediately preceding image (step S102), and outputs the image from the camera 10. The imaging direction is updated (step S103).

  The non-detection area setting unit 25b determines whether or not a setting operation of the non-detection area has been received (step S104). When the non-detection area setting operation is accepted (step S104; Yes), and the non-detection area is designated on the image, the calculation unit 25c converts the non-detection area on the designated image into polar coordinates (step S105). . Then, the non-detection area setting unit 25b sets the corresponding grid on the celestial sphere as the non-detection area and updates the non-detection area data 24a (step S106).

  After step S106 or when the setting operation of the non-detection area is not received (step S104; No), the object detection unit 25d executes the object detection process (step S107).

  Then, the object detection unit 25d determines whether to end the object detection (step S108). As the condition for ending the object detection, any condition can be used, such as when a specific object is detected, when a predetermined time has elapsed from the start of the detection, when a designated time is reached, or the like.

  When the object detection is not completed (step S108; No), the process proceeds to step S101 to acquire the next image. Then, when it is determined that the object detection is to be ended (step S108; Yes), the processing is ended.

  FIG. 4 is a flowchart showing details of the object detection process shown in FIG. The object detection unit 25d performs image processing on the entire image captured by the camera 10 and identifies whether or not the target object exists in the image (step S201). If the target object is present in the image (step S202; Yes), the object detection unit 25d outputs the position of the identified object in the image to the calculation unit 25c. The calculator 25c converts the identified position of the object on the image into polar coordinates, and compares the position of the object with the position of the non-detection region (step S203).

  As a result of the comparison, if the position of the object is not in the non-detection region (step S204; No), the object detection unit 25d outputs the detection result that the target object is detected (step S205), and returns to the original processing. . On the other hand, when there is no target object in the image (step S202; No) or when the position of the object is in the non-detection area (step S204; Yes), the object detection unit 25d determines that the target object is non-detection ( (Step S206), the process returns to the original process.

  Next, the calculation of the non-detection area after changing the angle of view by the calculation unit 25c will be described with reference to FIGS. FIG. 5 is an explanatory diagram of the imaging range and the lattice of the camera 10. As shown in FIG. 5, the camera 10 is capable of capturing an image in all horizontal directions by pan control of 360 °, and tilt control of 90 ° or more. By the pan control and the tilt control, the image pickup range of the camera 10 is a half-sphere to a whole-sphere. Then, each grid obtained by dividing the imaging range centered on the camera 10 serves as a unit for setting the non-detection area. As an example, the grid is formed by dividing the longitude and latitude at equal intervals, with the angle in the pan direction as the longitude and the angle in the tilt direction as the latitude. The dividing method is not limited to this, and any method can be used.

  FIG. 6 is an explanatory diagram of the angle of view of the camera 10. FIG. 6A shows the angle of view at the reference position where the camera 10 is not under the pan control and tilt control, and FIG. 6B is the angle of view under the pan control and tilt control. There is.

Here, the two-dimensional coordinate value in the camera which is the image captured by the camera 10, the focal length corresponding to the zoom magnification of the camera 10, the three-dimensional coordinate value in the real space corresponding to the coordinate value of the image, the coordinate value of the image The latitude / longitude direction of the real space corresponding to, the distance to the position of the real space corresponding to the coordinate value of the image, and the tilt / pan angle from the horizontal reference direction of the camera are determined as follows.
(X, Y): Two-dimensional coordinate value of the camera image (however, the center of the image is (0,0))
Z: focal length (x, y, z) corresponding to the zoom magnification of the camera 10: three-dimensional coordinate value of the real space corresponding to the coordinate value of the camera image θ, φ: real space corresponding to the coordinate value of the camera image Latitude / longitude direction r: Distance to the position in real space corresponding to the coordinate value of the camera image θ ₀ , φ ₀ : Tilt angle and pan angle from the horizontal reference direction of the camera

の対応関係となる。
そして、カメラ画像の２次元の座標値（Ｘ，Ｙ）に対応する距離Ｚの位置の平面に置いたカメラ画像上の３次元の座標値は、（Ｘ，Ｙ，Ｚ）であり、３次元の座標値と緯度・経度方向の対応関係は次の通りである。

また、カメラ画像における各画素（ピクセル）の方向は、

となる。 In FIG. 6A, the two-dimensional coordinate value (X, Y) of the camera image is the coordinate value obtained by projecting the three-dimensional coordinate value (x, y, z) onto the plane of z = Z. Because

It becomes the correspondence of.
The three-dimensional coordinate value on the camera image placed on the plane at the distance Z corresponding to the two-dimensional coordinate value (X, Y) of the camera image is (X, Y, Z). The correspondence between the coordinate values of and the latitude / longitude directions is as follows.

Also, the direction of each pixel in the camera image is

Becomes

そして、３次元の座標値と緯度・経度方向の対応関係は次の通りである。

また、カメラ画像における各画素（ピクセル）の方向は、

となる。 In FIG. 6B, the three-dimensional coordinate values (X, Y, Z) on the camera image placed on the plane at the distance Z corresponding to the two-dimensional coordinate values (X, Y) of the camera image are tilted. The three-dimensional coordinate values (x, y, z) after rotation when considering the correspondence with the three-dimensional coordinate values by rotating at the pan angle are as follows.

The correspondence between the three-dimensional coordinate values and the latitude / longitude directions is as follows.

Also, the direction of each pixel in the camera image is

Becomes

である。
そのズーム倍率に対してｎ倍のズーム倍率のときの焦点距離Ｚは、

である。
ここで、現在の画像中央が向く方向に対応する過去のある時点で画像のピクセルの２次元の座標値、現在のチルト角度、現在のパン角度、過去のチルト角度、過去のパン角度、過去の焦点距離を次のように定める。
（Ｘ’，Ｙ’）：現在の画像中央が向く方向に対応する過去のある時点で画像のピクセルの２次元の座標値
θ₀：現在のチルト角度
φ₀：現在のパン角度
θ₀’：過去のチルト角度
φ₀’：過去のパン角度
Ｚ’：過去の焦点距離 Next, how to obtain the focal length Z will be described. Angle from the front to the horizontal? ? _When the object at ₁ is reflected at the position of (X ₁ , 0), the focal length Z _{1 at the} zoom magnification is

Is.
The focal length Z when the zoom magnification is n times the zoom magnification is

Is.
Here, the two-dimensional coordinate value of the pixel of the image at a certain point in the past corresponding to the direction in which the current image center faces, the current tilt angle, the current pan angle, the past tilt angle, the past pan angle, and the past pan angle. The focal length is set as follows.
(X ', Y'): Two-dimensional coordinate value of the pixel of the image at a certain point in the past corresponding to the direction in which the current image center is facing θ ₀ : Current tilt angle φ ₀ : Current pan angle θ ₀ ': Past tilt angle φ ₀ ': Past pan angle Z': Past focal length

By changing the angle of view of the camera 10, the following formula is obtained.

  Next, updating of the pan angle, tilt angle, and zoom magnification when changing the angle of view will be described. If the angle of view of the camera 10 is changed and a conversion formula for changing the angle of view can be calculated by image processing, the pan angle, tilt angle, and zoom magnification after the angle of view change can be obtained from the conversion formula.

となる。 First, if Z is the focal length, the three-dimensional coordinate value (x, y, z) projected onto the z = Z plane corresponding to the two-dimensional coordinate value (X, Y) on the image is (X , Y, Z). Then, the coordinate values after the coordinate conversion of the three-dimensional coordinate values (x, y, z) by pan, tilt, and roll are:

Becomes

そして、ズーム変更による変更後の射影平面をＺ’としたとき、射影変換後の画像上の位置（Ｘ’，Ｙ’）は、

となり、射影変換は次の式のように８個の係数（ａ_xx，ａ_xy，ａ_yx，ａ_yy，ｂ_x，ｂ_y，ｃ_x，ｃ_y）で表すことができる。

_{(X x, x y, x} z), when a _{(y x, y y, y} z), (z x, z y, z z) X -axis direction of the coordinate transformation, Y-axis, Z-axis direction, The X-axis direction, the Y-axis direction, and the Z-axis direction must satisfy the following conditions.

Then, when the projection plane after the change due to the zoom change is Z ′, the position (X ′, Y ′) on the image after the projective transformation is

Therefore, the projective transformation can be represented by eight coefficients (a _xx , a _xy , a _yx , a _yy , b _x , b _y , c _x , _cy ) as in the following equation.

The pixel (X ′, Y ′) of the past image corresponding to the pixel (X, Y) of the current image is calculated by projective transformation, and the past tilt angle θ ₀ ′ and the past pan angle φ are calculated using these coefficients. _The current tilt angle θ ₀ , the current pan angle φ ₀ , and the current focal length Z can be calculated from ₀ ′ and the past focal length Z ′.

である。そして、現在のチルト角度θ₀と現在のパン角度φ₀は、

となる。条件式を係数を用いて書き直すと、

であり、現在の焦点距離Ｚは次のように求められる。

First, since the direction in which the image center faces corresponds to the tilt angle / pan angle of the camera 10, the pixel (X ′, Y ′) in the past image corresponding to the direction in which the current image center faces is

Is. The current tilt angle θ ₀ and the current pan angle φ ₀ are

Becomes If you rewrite the conditional expression using coefficients,

And the current focal length Z is obtained as follows.

  As described above, the image processing device 20 converts the two-dimensional coordinates in the image into the three-dimensional polar coordinates in the real space centered on the camera 10, and converts the three-dimensional polar coordinates into the coordinates in the image after the angle of view is changed. Correspondence can be calculated. Therefore, the image processing apparatus 20 stores the data of all celestial spheres in the non-sensing region data 24a, manages the non-sensing regions on the celestial sphere, and refers to a portion corresponding to the angle of view of the camera 10 when detecting an object. . Therefore, even if the non-detection area goes out of the angle of view after the non-detection area is set, the non-detection area is not lost.

FIG. 7 is an explanatory diagram regarding management of coordinates by three-dimensional coordinates. In FIG. 7, the image P1 is captured at the reference position (θ ₀ = 0, φ ₀ = 0) of the camera 10, and the camera 10 is pan-controlled rightward to capture the image P2.

  When the coordinate value of the pixel in the image P1 is converted along with the pan control, the coordinate value of the corresponding pixel in the image P2 is obtained. In the image P2, the corresponding pixel exists within the angle of view, but if the image Pn obtained by continuing the pan control is similarly converted, the coordinate value is outside the angle of view, and the corresponding pixel does not exist.

  If the two-dimensional coordinates in the image after changing the angle of view are directly calculated from the two-dimensional coordinates in the image before changing the angle of view, the information on the coordinates outside the angle of view is lost. On the other hand, the calculation unit 25c of the image processing device 20 converts the coordinate values of the pixels in the image P1 into the three-dimensional polar coordinates in the real space, and therefore does not lose information about the coordinates outside the angle of view.

For example, when the coordinates of the non-detection area are initially set (φ ₀ = 0), the pan control is continued, and the camera 10 makes one rotation (φ ₀ = 360), the two-dimensional image before the change is obtained every time the angle of view is changed. In the configuration in which the coordinates are changed to the two-dimensional coordinates after the change, the coordinates of the non-detection area initially set (φ ₀ = 0) are lost when the angle of view exits, and the same after rotation (φ ₀ = 360). However, the non-detection area cannot be excluded regardless of the angle of view.

On the other hand, in the configuration in which the coordinates of the non-detection area initially set (φ ₀ = 0) are converted into three-dimensional polar coordinates in the real space, the coordinates of the non-detection area are specified even after rotation (φ ₀ = 360). Therefore, the non-detection area can be excluded.

  When the two-dimensional coordinates on the image are converted into the three-dimensional polar coordinates in the real space for use, it is necessary to accumulate the history of pan, tilt, and zoom. The history of pan, tilt, and zoom may be accumulated based on the control content of the camera 10, or may be calculated and accumulated from the image of the camera 10. When accumulating the history of pan, tilt, and zoom from the image of the camera 10, an arbitrary corresponding point in the image before and after the change of the view angle is used regardless of whether or not the non-detection area is included in the view angle. It is sufficient to calculate and accumulate how much pan, tilt and zoom have been performed.

  Further, in the configuration of calculating and accumulating how much pan, tilt, and zoom are performed by using arbitrary corresponding points in the image before and after the change of the angle of view, the angle of view that does not depend on the remote control from the image processing device 20 can be obtained. It is possible to cope with fluctuations. For example, even when the camera 10 is directly operated, or when the camera 10 has a physical force to change the view angle of the camera 10, the change in the view angle can be specified.

  As described above, in the first embodiment, when the image processing apparatus 20 sets the non-detection area in the image captured by the camera 10 that is the image capturing apparatus and remotely controls the angle of view of the camera 10. , The non-detection area after the view angle change corresponding to the non-detection area set before the view angle change is calculated. Therefore, when an image is captured using the camera 10 whose angle of view can be remotely controlled, it is possible to suppress a decrease in reliability due to erroneous detection of an object.

  Further, the image processing apparatus 20 can perform an object detection process of detecting an object by using an area excluding the non-detection area after the calculated view angle change as a target area. Further, the image processing device 20 can perform pan, tilt, and zoom control of the camera 10.

  Further, the camera 10 can capture images in all directions, and the image processing apparatus 20 sets the two-dimensional coordinates in the image of the non-detection area and the value corresponding to the zoom magnification at the time of capturing the image to polar coordinates centered on the camera 10. It is possible to convert the two-dimensional coordinates of the non-detection region after changing the angle of view from the polar coordinates.

  In the second embodiment, an image processing system that corrects a non-detection area will be described. When the non-detection area is set in a certain image and the view angle is changed, the non-detection area after the view angle change can be calculated as shown in the first embodiment.

  However, a region that is not included in the image before changing the angle of view and is newly included in the image after changing the angle of view is not detected in the original image even if it is a region that should be excluded from the detection target as a non-detection region. It cannot be excluded because it is not set in the detection area.

  On the other hand, if the operator is requested to make a setting each time a new area is included in the image due to the change of the angle of view, the work load on the operator increases. Therefore, when the non-detection area is set using an image with a certain angle of view, the operator's burden can be reduced by performing correction so that the set non-detection area is reflected outside the angle of view. it can.

  FIG. 8 is an explanatory diagram of the concept of the image processing system according to the second embodiment. In FIG. 8, the image processing apparatus 120 acquires the image P1 from the camera 10 (S1) and sets the non-detection area A1 for the image P1 (S2). After that, the image processing apparatus 120 changes the angle of view to control the camera 10 to tilt downward (S3), and acquires the image P3. The image processing apparatus 120 calculates the non-detection area A3 after changing the angle of view, which corresponds to the non-detection area A1 set in the image P1 (S4).

  The non-detection area A1 in the image P1 is a grid selected so as to include an area showing a row of trees on the image, in other words, the area showing a row of trees is approximated by a rectangle. For this reason, there are locations that are not trees but are included in the non-detection area A1, and locations that are trees and are not included in the non-detection area.

  Further, as a result of the tilt control, the image P3 includes a region showing row trees in a wider range than the image P1. However, since the non-detection area A3 is calculated from the non-detection area set in the image P1, the area of the tree lined newly included in the image P3 is not included in the non-detection area A3.

  FIG. 9 is an explanatory diagram showing the correspondence between the non-detection region A1 and the lattice, and FIG. 10 is an explanatory diagram of correction of the non-detection region A3. First, as shown in FIG. 9, the coordinates of the image P1 correspond to the spherical grid, and the non-detection area A1 is set by selecting the grid.

  Further, as shown in FIG. 10, the celestial sphere lattice also corresponds to the coordinates of the image P3 after changing the angle of view. Therefore, of the grids included in the non-detection area A1, the grids included in the image P3 form the non-detection area A3.

  Further, the image processing apparatus 120 determines the correction target area B3 from the image P3 and sets the correction target area B3 as the non-detection area to obtain the non-detection area A3a. The condition of the correction target area B3 may be “a grid continuous with the non-detection area A3 and the type of the area is the same as the non-detection area A3”.

  The type of region can be specified, for example, by using segmentation by a learning model using a neural network. Then, as the type of the region of the lattice, the type of the region having the largest area included in the lattice may be used. The area type of the grid may be specified in advance for all the grids of the celestial sphere and may be associated with the non-detection area data 24a. Further, the ratio of the types of regions included in the grid may be associated with the non-detection region data 24a.

  Specifically, in the images P1 and P3, the types of regions such as row of trees, roads, and sidewalks are obtained. Then, as shown in FIG. 10, since the non-detection area A3 is a type of tree, the grid that is continuous with the non-detection area A3 and has a type of tree is the correction target area B3. The correction target area B3 is included in the non-detection area A3a by the correction.

  Next, the configuration of the image processing device 120 will be described. FIG. 11 is a configuration diagram showing the configuration of the image processing apparatus 120. As illustrated in FIG. 11, the image processing apparatus 120 differs from the image processing apparatus 20 illustrated in FIG. 2 in that the control unit 25 further includes an area type identification unit 25e and a correction processing unit 25f. Since other configurations and operations are the same as those of the image processing apparatus 20 shown in FIG. 2, the same components are designated by the same reference numerals and description thereof will be omitted.

  The area type identification unit 25e performs processing for identifying the area type for the image acquired from the camera 10. The region can be identified for any image regardless of whether or not the image is captured by the camera 10 at the reference position. Further, as described above, any method such as segmentation by a learning model using a neural network can be used for identification.

  The correction processing unit 25f performs a process of correcting the non-detection area. Specifically, the correction processing unit 25f determines the correction target area using a condition such as “a grid that is continuous with the non-detection area and the type of the non-detection area is the same as the type”, and determines the correction target area. Perform the correction to be included in the non-detection area. Then, the object detection unit 25d of the image processing apparatus 120 detects an object by using the region excluding the non-detection region corrected by the correction processing unit 25f as the target region.

  Next, a processing procedure of the image processing apparatus 120 will be described. FIG. 12 is a flowchart showing the processing procedure of the image processing apparatus 120. In the processing procedure shown in FIG. 12, correction of the non-detection area (step S106a) is added before the object detection processing (step S107). The other processing procedure is the same as the processing procedure shown in FIG.

  In the processing procedure of FIG. 12, after step S106 or when the setting operation of the non-detection area is not accepted (step S104; No), the correction processing unit 25f corrects the non-detection area (step S106a), and the object detection is performed. The unit 25d detects an object by using a region excluding the non-detection region corrected by the correction processing unit 25f as a target region (step S107).

  Next, a modification of correction of the non-detection area will be described. In the description so far, the configuration has been described in which the non-detection area is managed in units of grids obtained by dividing the celestial sphere, but by correcting the non-detection area based on the type of area, the non-detection area is managed in unit of the area. It is also possible to do so.

  FIG. 13 is an explanatory diagram of correction of the non-detection area in the modification. In FIG. 13, the image processing apparatus 120 performs a process of identifying the type of area for the image P1 (S1c). For this identification, for example, segmentation by a learning model using a neural network may be used. The image P1c in which the type of the area is identified is divided in the identified area, and the type is associated with each area. Specifically, the image P1c is divided into areas such as a row of trees, a roadway, and a sidewalk.

  Then, when the non-detection area A1 is set (S2), the image processing apparatus 120 corrects the non-detection area by using the ratio of each type of the areas included in the non-detection area A1 (S2c). In FIG. 13, the area of the row of trees occupies most of the non-detection area A1. Therefore, the image processing apparatus 120 sets the area of the row of trees that is continuous with the non-detection area A1 as the post-correction non-detection area A1c.

  FIG. 14 is an explanatory diagram of correction after changing the angle of view in the modified example. The image processing device 120 changes the angle of view (S3) and acquires the image P3. Then, the image processing apparatus 120 calculates the non-detection area A3 after changing the angle of view, which corresponds to the non-detection area A1 set in the image P1 (S4).

  Further, the image processing apparatus 120 performs a process of identifying the type of area for the image P3. Then, the non-detection area is corrected by using the ratio of each area included in the non-detection area A3 (S4c). In FIG. 14, most of the non-detection area A3 is occupied by a row of trees. Therefore, the image processing apparatus 120 sets the area of the row of trees that is continuous with the non-detection area A3 as the corrected non-detection area A3c.

  In this modification, the correction processing unit 25f performs a process of correcting the non-detection region set by the non-detection region setting unit 25b and the non-detection region after the change of the view angle calculated by the calculation unit 25c. Specifically, the correction processing unit 25f obtains the ratio of the detection region and the non-detection region for each type, and if the ratio of the non-detection region is equal to or more than a predetermined ratio, the region of the type is determined as a grid of the non-detection region. A continuous area is defined as a non-detection area. Then, the object detection unit 25d of the image processing apparatus 120 detects an object by using the region excluding the non-detection region corrected by the correction processing unit 25f as the target region.

  Next, the correspondence relationship between the main hardware configurations of the image processing apparatus and the computer will be described. FIG. 15 is a diagram illustrating an example of the hardware configuration.

  A general computer has a configuration in which a CPU 91, a ROM (Read Only Memory) 92, a RAM 93, a non-volatile memory 94 and the like are connected by a bus 95. A hard disk device may be provided instead of the non-volatile memory 94. For convenience of explanation, only a basic hardware configuration is shown.

  Here, the ROM 92 or the non-volatile memory 94 stores a program or the like required for booting an operating system (hereinafter, simply referred to as “OS”), and the CPU 91 stores the ROM 92 or the non-volatile memory when the power is turned on. The OS program is read from the local memory 94 and executed.

  On the other hand, various application programs executed on the OS are stored in the non-volatile memory 94, and the CPU 91 executes the application programs while using the RAM 93 as a main memory to execute a process corresponding to the application. It

  The functions of the image processing apparatus 20 and the image processing apparatus 120 described in the first and second embodiments are also stored as a program in the non-volatile memory 94 or the like as with other application programs, and the CPU 91 loads this program. Will be executed. For example, a screen processing program including routines corresponding to the view angle control unit 25a, the non-detection area setting unit 25b, the calculation unit 25c, and the object detection unit 25d illustrated in FIG. By loading and executing the processing program, processes corresponding to the view angle control unit 25a, the non-detection area setting unit 25b, the calculation unit 25c, and the object detection unit 25d are generated. The non-detection area data 24a and the like are stored in the non-volatile memory 94.

  As described above, in the second embodiment, as in the first embodiment, the non-detection area after changing the angle of view can be calculated, and when the image is captured using the camera 10 that can remotely control the angle of view. Therefore, it is possible to suppress a decrease in reliability due to erroneous detection of an object. Furthermore, in the second embodiment, by correcting the non-detection region, it is possible to reduce the work load on the operator and prevent erroneous detection and omission of detection.

  In the second embodiment, the type of the area included in the image is identified, and the non-detection area is corrected using the type of the area included in the non-detection area. However, the present invention is not limited to this. However, the non-detection area can be corrected by using any method.

  Further, in the above-described first and second embodiments, the configuration in which the two-dimensional non-detection area set for the image is converted into polar coordinates and stored as the non-detection area data 24a is illustrated. The set two-dimensional non-detection area may be directly stored as the non-detection area data 24a. In this case, when calculating the non-detection area after changing the angle of view, the non-detection area data 24a is converted into polar coordinates and the non-detection area in the image after changing the angle of view is calculated from the current imaging direction and the like. become. Alternatively, whether or not the position of the object in the image after changing the angle of view is converted into polar coordinates based on the current imaging direction and the non-detection region data 24a is converted into polar coordinates, and whether the position of the object is within the non-detection region or not. You may judge whether.

  Further, in the above-described first and second embodiments, the case where the image processing device remotely controls the camera has been described as an example, but as long as the angle of view control of the camera is PTZ control, control by a device other than the image processing device is performed. The image processing apparatus may not have the control function of the camera.

  Further, in the above-described first and second embodiments, the configuration for calculating the non-detection region after the change of the angle of view is illustrated, but it is also possible to calculate for the detection region after the change of the angle of view.

  It should be noted that each of the configurations illustrated in the above-described embodiments is a functional schematic, and does not necessarily have to be physically configured as illustrated. That is, the form of distribution / integration of each device is not limited to that shown in the figure, and all or a part of them may be functionally or physically distributed / integrated in arbitrary units according to various loads and usage conditions. Can be configured.

  The image processing program, the image processing device, the image processing system, and the image processing method of the present invention reduce reliability due to erroneous detection of an object when an image is captured using an imaging device capable of remotely controlling the angle of view. It is useful for suppressing.

10 camera 20, 120 image processing device 21 communication unit 22 display unit 23 input unit 24 storage unit 24a non-detection region data 25 control unit 25a view angle control unit 25b non-detection region setting unit 25c calculation unit 25d object detection unit 25e region type identification Part 25f Correction processing part 91 CPU
92 ROM
93 RAM
94 non-volatile memory 95 bus

In order to solve the above problems, the present invention is an image processing program in an image processing apparatus that performs image processing of an image captured by an image capturing apparatus capable of remotely controlling an angle of view, which is a non-detection region for the image. A non-detection area setting procedure for setting the non-detection area and a non-detection area after changing the view angle corresponding to the non-detection area set by the non-detection area setting procedure when the view angle of the imaging device is remotely controlled. And a non-detection area correction procedure for correcting the non-detection area after changing the angle of view calculated by the calculation procedure .

Further, the present invention is an image processing apparatus that performs image processing of an image captured by an image capturing apparatus capable of remotely controlling an angle of view, and a non-detection area setting unit that sets a non-detection area for the image, If the angle of view of the imaging device is remotely controlled, and a calculation unit for calculating a non-detection region after the angle of view changes corresponding to the non-detection area set by the non-detection area setting unit, calculated by the calculation unit And a non-detection area correction unit that corrects the non-detection area after the changed view angle .

The present invention also provides an image processing system in which an imaging device and an image processing device that remotely controls an angle of view of the imaging device and performs image processing of an image captured by the imaging device are communicably connected. The image processing apparatus includes a view angle control unit that controls a view angle of the image pickup apparatus, a non-detection area setting unit that sets a non-detection area for the image, and a view angle of the image pickup apparatus is remotely controlled. If the, the non-detection and calculation unit for calculating a non-detection region after the angle of view changes corresponding to the non-detection area set by the area setting unit, a non-detection region after the angle of view changes calculated by the calculating section And a non-detection area correction unit that corrects

Further, the present invention is an image processing method in an image processing device for performing image processing of an image captured by an imaging device capable of remotely controlling an angle of view, the non-detection region having a non-detection region set for the image. A setting step; a calculation step of calculating a non-sensing area after changing the angle of view corresponding to the non-sensing area set in the non-sensing area setting step when the angle of view of the imaging device is remotely controlled ; And a non-detection area correction step of correcting the non-detection area after changing the angle of view calculated by the calculation step .

Claims (9)

An image processing program in an image processing device for performing image processing of an image captured by an imaging device capable of remotely controlling an angle of view,
A non-detection area setting procedure for setting a non-detection area for the image,
And a calculation procedure for calculating a non-detection area after changing the angle of view corresponding to the non-detection area set by the non-detection area setting procedure when the angle of view of the imaging device is remotely controlled. An image processing program characterized by:   2. The computer is further caused to execute a first object detection procedure for detecting an object with an area excluding the non-detection area after the change of the angle of view calculated by the calculation procedure as a target area. Image processing program.   The image processing program according to claim 1 or 2, further causing a computer to execute an angle-of-view control procedure for performing pan, tilt, and zoom control of the imaging device.   The image processing program according to claim 1, 2, or 3, further causing a computer to execute a non-detection area correction procedure for correcting the non-detection area after changing the angle of view calculated by the calculation procedure.   The image processing according to claim 4, further comprising: causing the computer to further execute a second object detection procedure of detecting an object by using an area excluding the non-detection area corrected by the non-detection area correction procedure as a target area. program. The imaging device can image in all directions,
In the calculation procedure, the two-dimensional coordinates in the image of the non-detection area are converted into polar coordinates centered on the imaging device, and the two-dimensional coordinates of the non-detection area after the change of the angle of view are calculated from the polar coordinates. The image processing program according to any one of claims 1 to 5, which is characterized in that. An image processing device for performing image processing of an image captured by an image capturing device capable of remotely controlling an angle of view,
A non-detection area setting unit for setting a non-detection area for the image,
And a calculation unit that calculates a non-detection region after changing the angle of view corresponding to the non-detection region set by the non-detection region setting unit when the angle of view of the imaging device is remotely controlled. Image processing device. An image processing system in which an imaging device and an image processing device for remotely controlling an angle of view of the imaging device and performing image processing of an image captured by the imaging device are communicatively connected,
The image processing device,
An angle-of-view control unit that controls the angle of view of the imaging device,
A non-detection area setting unit for setting a non-detection area for the image,
And a calculation unit that calculates a non-detection region after changing the angle of view corresponding to the non-detection region set by the non-detection region setting unit when the angle of view of the imaging device is remotely controlled. Image processing system. An image processing method in an image processing device for performing image processing of an image captured by an imaging device capable of remotely controlling an angle of view,
A non-detection area setting step of setting a non-detection area for the image,
And a calculation step of calculating a non-detection area after changing the angle of view corresponding to the non-detection area set by the non-detection area setting step when the angle of view of the imaging device is remotely controlled. Image processing method.

Images