JP2019125946A - Control device, imaging device, control method, and program - Google Patents

Abstract

To set an imaging range appropriately for tracking and imaging a specific subject according to an imaging scene.SOLUTION: A control device for controlling an imaging range of an imaging device for tracking and imaging a predetermined subject acquires information on a first distance between the predetermined subject and the imaging device, and a position where the predetermined subject appears in an image captured by the imaging device is set on the basis of at least the first distance, and an imaging range is controlled in accordance with the set position.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging apparatus for tracking and imaging a specific subject.

  There is a system for automatically tracking a specific subject and performing imaging using an imaging device such as a camera that can freely change the imaging range by pan / tilt control or the like. In such a system, setting is performed so that the subject of the target of automatic tracking comes to a designated position in the captured image. In the automatic tracking system, it is important to set a designated position where the subject is photographed. Patent Document 1 describes a method of setting a designated position such that a moving direction of a person (object) to be tracked and an imaging region in front of a direction of interest become large.

JP, 2006-229322, A

  In Patent Document 1, a method for monitoring a subject to be tracked is considered. However, depending on an imaging scene, there may be cases in which setting such that the imaging region in the line-of-sight direction or the moving direction becomes large may not be appropriate.

  The present invention has been made in view of such problems, and provides a technique for appropriately setting an imaging range at the time of tracking and imaging a specific subject according to an imaging scene.

  A control device according to an aspect of the present invention is a control device that controls an imaging range of an imaging device that tracks and images a predetermined subject, and the control device controls a first distance between the predetermined object and the imaging device. The position set by the setting means, the setting means for setting the position where the predetermined subject appears in the image captured by the imaging device based on at least the acquisition means for acquiring information, the first distance, and Control means for controlling the imaging range in accordance with

  According to the present invention, the imaging range at the time of tracking and imaging a specific subject can be appropriately set according to the imaging scene.

It is a block diagram showing the example of hardware constitutions of an imaging device. It is a block diagram showing an example of functional composition of an imaging device. FIG. 6 is a schematic view showing an example of a captured image during automatic tracking when a subject to be tracked is photographed at the image center position. It is a figure explaining the example of the scene where imaging is performed. 5 is a flowchart illustrating an example of the flow of processing performed in the imaging device. FIG. 6 is a schematic view showing an example of a captured image during automatic tracking when a subject to be tracked is photographed at a position shifted from the center of the image. It is a figure explaining the example of the scene where imaging is performed. 5 is a flowchart illustrating an example of the flow of processing performed in the imaging device. FIG. 7 is a schematic view showing an example of a scene in which the right end and the left end of an object outside the tracking target are detected. 5 is a flowchart illustrating an example of the flow of processing performed in an imaging device. 5 is a flowchart illustrating an example of the flow of processing performed in an imaging device. 5 is a flowchart illustrating an example of the flow of processing performed in an imaging device.

  Hereinafter, embodiments of the present invention will be described using the attached drawings. In the present embodiment, as an example, a case where an imaging apparatus capable of setting an imaging range by pan, tilt, and zoom tracks and images a predetermined subject (person) will be described. However, the present invention is not limited to this, and a configuration may be used in which a control device capable of controlling the imaging range of the imaging device and existing outside the imaging device controls the imaging device. Note that such a control device may control an imaging range in the imaging device by controlling, for example, a pan head on which the imaging device is installed, or by inputting an instruction signal from the outside to the imaging device. The imaging apparatus may control pan, tilt, and zoom. Also, such control devices may be installed individually for each imaging device, or may be installed one for a plurality of imaging devices. Also, the signal to the imaging device may be transmitted using a dedicated physical signal line, or may be transmitted via a general-purpose network line. That is, the following discussion can be applied to any type of control device as long as control of the imaging range of the imaging device can be performed.

  The imaging apparatus according to the present embodiment, in one example, when tracking and imaging a predetermined subject, according to the distance between the subject and the imaging apparatus itself, the subject is photographed at any position among the images obtained by imaging. Set the water level and control the imaging range accordingly. For example, the distance between an imaging device fixedly installed at a predetermined position and an object (stationary object or quasi-stationary object) without position variation can be specified in advance, and the distance between a subject and the imaging device is an imaging device May correspond to the distance between the object and the object. Here, the situation in which the subject exists at a position sufficiently close to the predetermined object may include the situation in which the subject uses the predetermined object. For example, in a situation where an imaging device is installed in a school, it is assumed that a subject to be tracked is a teacher and a predetermined object is a whiteboard or a blackboard. At this time, when the teacher is approaching the white board etc., it is assumed that the teacher is explaining using the white board. In this case, if imaging is always performed with the teacher at the center, the whiteboard will not be photographed, and for example, it will not be suitable as a recording for a class absentee. For this reason, the imaging device acquires information on the distance between itself and a subject (for example, a teacher), and based on the information, a predetermined object (for example, a whiteboard) falls within the imaging range as necessary. The position of the subject in the captured image can be set so as to fit. According to this, the imaging device can appropriately set the position of the tracking target object in the captured image according to the scene. For example, the imaging device may control the imaging range so that imaging is performed focusing on the subject in a situation where it is sufficient to simply capture the subject, and in a scene where it is important to capture a predetermined object, the object is imaged together with the subject. it can. Note that the imaging device can hold in advance the setting value of the imaging range when the predetermined object is photographed based on the position of the predetermined object, and the imaging range at the time of imaging the object together with the subject , Can be set based on the previously held setting value.

  In addition, in another example, the imaging apparatus may acquire information on the distance between a first subject (for example, a tracking target person) and a second subject (for example, a stationary object or a quasi-stationary object). According to such a configuration, even in a situation where the positional relationship between the imaging device and the second subject is not acquired in advance, the imaging range can be appropriately specified. In addition, the imaging apparatus may set which position of the image obtained by imaging the subject is to be photographed from the relationship between the position of the object other than the subject and the position of the subject. For example, when the subject is positioned to the right of the object when viewed from the imaging device, the imaging apparatus can shift the position of the subject from the center to the right to determine the imaging range so that the object and the subject are captured.

  Furthermore, in another example, the imaging apparatus can determine the position at which the tracking target is to be photographed in the image, based on the positional relationship between the tracking target subject and the subject outside the tracking target, and the setting of the zoom. According to this, it is possible to determine the imaging position of the tracking target subject so that the subject outside the tracking target is appropriately photographed as needed. In addition, after the control such that the subject outside the tracking target is photographed, the imaging apparatus fixes the imaging range so that the subject outside the tracking target can be stably photographed regardless of the movement of the tracking target subject. be able to.

  Hereinafter, some examples of the configuration of the imaging apparatus and the flow of processing will be described.

(Device configuration)
FIG. 1 shows an example of the hardware configuration of the imaging device of the present embodiment. In one example, the imaging apparatus includes a CPU 101, a RAM 102, a ROM 103, an external storage device 104, an input / output circuit 105, and an imaging circuit 106. The CPU 101 is a general-purpose processor, controls the entire imaging apparatus, and executes various processing such as setting processing of an imaging range by an imaging circuit 106 described later. When the CPU 101 executes a program stored in, for example, the ROM 103 or the external storage device 104, various processes are executed. The CPU 101 is an example of a processor included in the imaging apparatus, and may be replaced with a gate array circuit such as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) capable of realizing the same function. . The RAM 102 is used to temporarily save information, such as temporarily saving an image captured by the imaging circuit 106, for example. The RAM 102 can also be used as a work memory when the CPU 101 executes a program. The ROM 103 stores, for example, various information such as programs and parameters to be executed by the CPU 101. The external storage device 104 stores, for example, a program to be executed by the CPU 101 and information on processing results such as a video captured by the imaging circuit 106. The RAM 102, the ROM 103, and the external storage device 104 are an example of a configuration for storing information, and configurations other than these may be used. The input / output circuit 105 performs input and output of information. The input / output circuit 105 performs, for example, acceptance of an operation by a user, display of a captured image on a display (not shown), and the like. Also, the input / output circuit 105 may be a communication interface for connecting to a wired network or a wireless network, and may input / output information via the network. The imaging circuit 106 is a circuit for performing imaging, and includes a lens, an imaging element, and a circuit capable of executing various processes related to imaging. The imaging circuit 106 is configured to be able to execute pan-tilt-zoom control in one example.

  As described above, in the case where a control device that controls the imaging device from the outside is used, the control device can have a configuration that does not include the imaging circuit 106 in the configuration of FIG. 1. For example, the control device transmits a pan-tilt-zoom control signal to the imaging device by executing a program by the CPU 101, or controls a camera platform or the like on which the imaging device is installed. The imaging range can be controlled. Note that the control device may be included in the imaging device, and in this case, the control device may be realized by a configuration other than the imaging circuit 106, for example, and configured to control the imaging range of the imaging circuit 106.

  Subsequently, an example of the functional configuration of the imaging device will be described with reference to FIG. In one example, the imaging apparatus includes an imaging unit 201, an image processing unit 202, a pan / tilt control unit 203, a system control unit 204, a storage unit 205, a communication processing unit 206, an automatic tracking control unit 207, an object recognition unit 208, and distance acquisition. A unit 209 and an object position control unit 210 are included.

  The imaging unit 201 captures an image of a subject by the imaging circuit 106, and converts an image obtained by the imaging into an electrical signal. The image processing unit 202 performs image processing and compression encoding processing on the electric signal converted by the imaging unit 201, and generates image data. The image data is transmitted to the system control unit 204. The pan / tilt control unit 203 is a control unit that controls the operation of a pan drive unit or a tilt drive unit (not shown), and performs rotational drive control in the pan direction or the tilt direction. The system control unit 204 distributes the generated image data to the client device via the communication processing unit 206. The system control unit 204 also analyzes the camera control command received via the communication processing unit 206, and performs processing according to the command. For example, the system control unit 204 performs setting of image quality adjustment to the image processing unit 202, and setting of imaging parameters such as setting of zoom and focus control to the imaging unit 201. For example, the imaging unit 201 performs lens control based on the transmitted instruction. The system control unit 204 also instructs the pan / tilt control unit 203 to rotate in the pan or tilt direction. A storage unit 205 stores setting values such as image quality adjustment parameters and network settings. By storing such setting values in the storage unit 205, the imaging device can be operated using the previously set values even when the imaging device is restarted. The communication processing unit 206 performs processing of network communication by LAN. The automatic tracking control unit 207 specifies the position in the captured image of the subject of the automatic tracking target recognized by the subject recognition unit 208, and the rotation angle so that the subject is photographed at a predetermined position in the captured image Are calculated, and a pan / tilt control instruction is issued to the system control unit 204. The subject recognition unit 208 extracts the shape of a face or a body from the captured image generated by the image processing unit 202, and performs processing of recognizing a subject. The subject recognition unit 208 can recognize, for example, a subject to be tracked in a pattern based on face recognition. However, the subject to be tracked may be recognized by another recognition technique such as color, outline, or moving object recognition. Further, the subject to be tracked is not identified as a person, and may be, for example, an object (for background or foreground). In this case, an object recognition technique that performs object recognition may be used. The distance acquisition unit 209 performs processing for calculating and acquiring the distance between the camera and the subject from the index of a known size included in the captured image generated by the image processing unit 202. For example, the size data of each part of the subject's face or body is given as an index of known size, and the distance between the camera and the subject is calculated from the number of pixels occupied by the index area (each part) of the image data. Also, an indicator of known size such as a two-dimensional barcode may be attached to the subject as a batch, the indicator in the image data may be recognized, and the distance between the camera and the object may be calculated from the number of pixels occupied by the indicator. . The subject position control unit 210 determines, based on the distance information acquired by the distance acquisition unit 209, at which position in the captured image the object to be tracked is to be photographed. Then, the subject position control unit 210 instructs the automatic tracking control unit 207 the position in the captured image in which the determined subject is photographed.

(Flow of processing)
Subsequently, a flow of processing performed by the imaging device (or control device) will be described.

  An image captured by the imaging unit 201 and generated by the image processing unit 202 is subjected to image analysis by the subject recognition unit 208, and it is determined whether, for example, a predetermined subject determined as a target in advance is shown. Then, when it is determined by the subject recognition unit 208 that the predetermined subject is photographed, the automatic tracking control unit 207 sets an imaging range so that the predetermined subject is photographed at a predetermined position in the image. FIG. 3 shows an example where automatic tracking control is performed such that a predetermined subject (subject 301) is at the center position in the captured image 300. A subject area 302 indicates an area recognized by the subject recognition unit 208 that a predetermined subject is photographed. Although FIG. 3 shows an example in which the recognized area is a rectangular area, this area may not be a rectangular area. Also, the recognized area may not be displayed as shown in FIG. 3 and may only be recognized internally by the imaging device. In the example of FIG. 3, since the automatic tracking control is performed so that the subject to be tracked is captured at the central position in the captured image, the subject 301 is captured near the approximate center of the captured image. In the image, in addition to the tracking target subject, a subject other than the tracking target is also captured. In FIG. 3, as an example, the whiteboard 303 is shown as a subject outside the tracking target on the back of the subject 301. Note that FIG. 3 is an image obtained by imaging a scene in which the subject 301 is explaining using the whiteboard 303, and in this case, it is desirable that the whiteboard 303 be imaged as well. However, as a result of automatic tracking control being performed so that the subject 301 to be tracked is captured at the center of the image, a part of the whiteboard 303 is out of the imaging range, and a part of the characters written on the whiteboard 303 is viewed. I can not

  For this reason, the imaging apparatus of the present embodiment executes processing for setting an imaging range that includes the subject outside the tracking target in a scene where it is useful to photograph such a subject outside the tracking target. The following describes some examples of such processing.

(Processing example 1)
The imaging apparatus can determine whether or not to capture the subject 301 at the center of the image according to the distance between the tracking target subject 301 and the imaging apparatus. That is, when the imaging device is fixed, the distance between the whiteboard 303 and the imaging device is generally fixed, and the imaging device can acquire the distance in advance by, for example, user input. Then, when the difference between the distance between the subject 301 to be tracked and the subject device and the distance between the whiteboard 303 and the subject device obtained in advance is small, the subject 301 and the whiteboard 303 are close. It can be determined that it is at a position. Then, when the subject 301 and the whiteboard 303 exist at a close position, it can be estimated that the subject 301 is using the whiteboard 303. For this reason, when it is estimated that the subject 301 uses the whiteboard 303 according to the distance between the subject 301 and the own apparatus, the imaging apparatus according to the present embodiment causes the whole or a predetermined ratio of the whiteboard 303. The position where the subject 301 is photographed is determined so that

  An example of a scene when this processing is performed is shown in FIG. The camera 401 is an imaging apparatus according to the present embodiment, and images a subject 301 to be tracked and a subject (whiteboard 303) other than the tracker. The subject 301 mounts a batch 403 on which a two-dimensional marker having a known size and pattern is printed, as an example. The distance acquisition unit 209 can specify the three-dimensional relative position between the camera 401 and the batch 403 by analyzing the captured image including the two-dimensional marker. In addition, since the calculation method of the relative position using a two-dimensional marker is a well-known technique, it abbreviate | omits about the detailed description here. The distance acquisition unit 209 can acquire DstT 402, which is the distance between the subject 301 and the camera 401, by acquiring depth distance information from this three-dimensional relative position. Although the example using a two-dimensional marker is shown in the present embodiment, the distance between the camera 401 and the subject 301 may be specified by a method other than this. For example, the size data of each part of the subject's face or body is given as an index of a known size, and the distance between the camera 401 and the subject 301 is calculated from the number of pixels in the area occupied by each part in the image data It is also good. In addition, the distance acquisition unit 209 may be realized by, for example, a distance measurement sensor such as a depth sensor, or the distance between the camera 401 and the object 301 may be obtained regardless of image analysis, such as acquiring the distance to the object. It may be specified. Furthermore, instead of the distance acquisition unit 209 inside the camera 401 estimating the distance between the camera 401 and the subject 301, the distance measuring device outside the camera 401 determines the distance between the camera 401 and the subject 301. It may be measured. In this case, the distance acquisition unit 209 can acquire the information of the measurement result input from the distance measurement device to the camera 401. Further, the distance acquisition unit 209 may use a plurality of the above-described distance measurement methods in combination. As described above, the distance acquisition unit 209 acquires the distance between the camera 401 and the subject 301 using at least one of various methods.

  Then, the subject position control unit 210 sets the position at which the subject 301 is photographed in the captured image, depending on whether the distance DstT 402 acquired by the distance acquisition unit 209 is smaller than a predetermined distance (threshold StT). Here, StT is a setting value set by, for example, the user operating a PC (not shown), and is previously set as a setting value in the storage unit 205 of the camera 401 (for example, via the communication processing unit 206 of the camera 401). It is memorized. Also, StT may be determined, for example, according to the distance between the camera 401 and the whiteboard 303. For example, when DstT 402 is equal to or greater than StT, the distance is determined as the distance at which it can be determined that the subject 301 is sufficiently close to the whiteboard 303. Then, when the DstT 402 is smaller than the predetermined distance StT, for example, the subject position control unit 210 determines the position of the subject 301 so that the subject 301 is photographed at the center position. Further, for example, when the DstT 402 is equal to or more than the predetermined distance StT, the subject position control unit 210 shifts the position of the subject 301 to the right so that the whiteboard 303 is captured. At this time, the subject position control unit 210 holds in advance the setting value of the imaging range of the camera 401 where the whiteboard 303 appears, and in the range where the subject 301 appears in the image, A process of shifting the imaging range may be performed. Further, the subject position control unit 210 sets a pattern A which defines the position where the subject appears at the center and a pattern where the position where the subject appears is a predetermined position to the right of the center according to whether DstT 402 is StT or more. It can be decided to use either B or B. In this case, the imaging range is set so that the subject fits in the position where the subject determined for each pattern should be taken. FIG. 5 shows an example of the flow of such processing performed by the subject position control unit 210.

  The subject position control unit 210 first obtains the value of the distance (DstT) between the subject to be tracked and the imaging apparatus (S501). In the example of FIG. 4, the subject position control unit 210 acquires the distance DstT 402 between the subject 301 and the camera 401. Then, the subject position control unit 210 compares the DstT acquired in S501 with the threshold StT (S502). When the distance DstT is smaller than the threshold StT (YES in S502), the subject position control unit 210 determines that the pattern A centering on the position of the subject in the image is used (S503). In addition, when the distance DstT is equal to or larger than the threshold StT (NO in S502), the subject position control unit 210 determines to use the pattern B in which the position of the subject in the image is a predetermined position on the right side of the center. (S504). For example, it is assumed that the subject on the back (white board 303) is fixed at a distance of 4.5 m from the camera, and the distance of 4.0 m from the camera 401 is a threshold value StT. In this case, when the distance DstT is less than 4.0 m, the subject position control unit 210 uses the pattern A (center) for the position of the subject in the image, and when the distance DstT is 4.0 m or more, the pattern B Make a decision to use (rightward).

  The subject position control unit 210 notifies the automatic tracking control unit 207 of the determined subject position (S505). The automatic tracking control unit 207 sets an imaging range so that the tracking target subject appears at the notified position. The automatic tracking control unit 207 calculates, for example, an angle for rotating the camera 401, and instructs the system control unit 204 to perform pan / tilt control. Note that the pattern of the position of the subject in the image may be determined only when the same comparison result is continuously obtained a predetermined number of times in the comparison of S502. According to this, when the subject is present at a position where the difference between the distance DstT and the threshold value StT becomes very small, it is possible to prevent the pattern of the position of the subject in the image from being frequently switched.

  FIG. 6 shows an example of an image 600 obtained by imaging when the use of the pattern B is determined by the process of FIG. 5 and the subject 301 is photographed at a position shifted to the right from the center in the imaged image. A subject area 601 indicates an area recognized by the subject recognition unit 208 as a predetermined subject. As shown in FIG. 6, the left side of the subject 301 is widely photographed by photographing the subject area 601 at a position on the right side of the center of the captured image. As a result, the whiteboard 303 which is a subject other than the automatic tracking target can be included in the imaging range, and the image can be captured so that all the character strings described in the whiteboard 303 can be seen.

(Processing example 2)
In the first processing example, the method of determining the position in the image where the subject is to be photographed is described using the distance between the subject that is the automatic tracking target and the imaging apparatus. In this processing example, the distance index between the subject that is the object of automatic tracking and the subject that is not the object of automatic tracking is specified by acquiring the distance between the subject that is not the object of automatic tracking and the imaging device. Based on the value, the position where the subject is photographed in the image is determined.

  An example of a scene when this processing is performed is shown in FIG. The camera 401 captures an image of a subject 301 to be tracked and a whiteboard 303 which is a subject not to be tracked. As in the case of FIG. 4, a batch 403 on which a two-dimensional marker is printed is attached to the subject 301. The distance acquisition unit 209 specifies the distance DstT 702 between the camera 401 and the subject 301 to be tracked by analyzing the captured image on which the two-dimensional marker appears. Further, in this process, a two-dimensional marker 704 having a pattern different from that of the two-dimensional marker of the batch 403 attached to the subject 301 to be tracked is attached to the subject (whiteboard 303) not to be tracked. The distance acquisition unit 209 uses the two-dimensional marker 704 to specify the distance DstB 703 between the subject (whiteboard 303) that is not the tracking target and the camera 401. In this example, although the case of calculating the distance between the camera and each of the tracking target subject and the non-tracking target based on the information in the image data has been described, the present invention is not limited thereto. For example, a distance measuring sensor such as a depth sensor may be mounted on the camera 401 to acquire the distance to each subject. When the distance acquiring unit 209 acquires the distances (DstT and DstB) between the camera and each of the tracking target subject and the non-tracking target, the distance index corresponding to the distance between these subjects is | DstT-DstB. Acquired by calculating |.

  The subject position control unit 210 determines that the distance between the subject to be tracked and the subject outside the tracking target is longer as the value of the distance index is larger, and as the value of the distance index is smaller, the subject to be tracked and the subject outside the tracking target It is estimated that the distance of is close. Then, based on the estimation result, the subject position control unit 210 sets a position at which the subject to be tracked is photographed in the captured image. An example of the flow of processing at this time will be described with reference to FIG. First, the subject position control unit 210 obtains the distance DstT 702 between the subject (subject 301) to be tracked and the camera 401, and the distance Dst B 703 between the subject (whiteboard 303) not to be tracked and the camera 401. (S801, S802). Next, the subject position control unit 210 calculates a distance index Ld = | DstB−DstT | from the value of the acquired distance (S803). Then, the subject position control unit 210 compares the distance index Ld with the threshold value StD (S804). Here, StD is a setting value set by, for example, the user operating a PC (not shown), and is set in advance as a setting value in the storage unit 205 of the camera 401 (for example, via the communication processing unit 206 of the camera 401). It is memorized. Note that the threshold value StD is, in one example, the distance between the instructor and the whiteboard when the instructor uses the whiteboard, or the like. It can be determined from statistics on distance. When the distance index Ld is less than StD (YES in S804), the subject position control unit 210 determines to use the pattern B for shifting the position of the subject 301 to the right and causing the whiteboard 303 to be captured ( S 805). When the distance index Ld is greater than or equal to StD (NO in S804), the subject position control unit 210 determines to use the pattern A that causes the position of the subject 301 to be captured at the center (S806).

  The subject position control unit 210 notifies the automatic tracking control unit 207 of the determined subject position (S807). The automatic tracking control unit 207 sets an imaging range so that the tracking target subject appears at the notified position. The automatic tracking control unit 207 calculates, for example, an angle for rotating the camera 401, and instructs the system control unit 204 to perform pan / tilt control. Note that the pattern of the position of the subject in the image may be determined only when the same comparison result is continuously obtained a predetermined number of times in the comparison of S804. According to this, when each subject is present at a position where the difference between the distance index Ld and the threshold value StD becomes very small, it is possible to prevent the pattern of the position of the subject in the image from being frequently switched. .

  According to this processing example, even when the position of the subject outside the tracking target is not fixed or when the distance between the subject and the camera is not known in advance, the subject is not captured in the captured image. The position to be photographed can be properly determined.

(Processing example 3)
In the above-described processing example, the positional relationship between the tracking target subject and the subject outside the tracking target in the lateral direction (horizontal direction) in the captured image is a fixed relationship (the tracking target subject is not the tracking target) The case where it is known that there is a relation existing on the right side of the subject of In this processing example, in consideration of the positional relationship in the horizontal direction, it is determined at which position in the image to be imaged the object to be tracked is captured.

  An example of a scene when this processing is performed is shown in FIG. The captured image 900 includes a subject 301 to be tracked, and a subject area 901 indicates an area in which the subject 301 is recognized. In the example of FIG. 9, the horizontal center of the subject area 901 is represented by Tp 904. The center position Tp 904 can be identified based on the recognized area, in which the subject area 901 is recognized by image analysis by the subject recognition unit 208. Further, in FIG. 9, the subject outside the tracking target is the whiteboard 303 on the back of the subject 301. In the present example, the end (left end Le 902 and right end Re 903) of the whiteboard 303 in the horizontal direction is further detected by image analysis by the object recognition unit 208.

  According to the relationship between the positions of the both ends (left end Le 902 and right end Re 903) of the subject other than the tracking target and the center position Tp 904 of the subject 301, the subject position control unit 210 places the subject 301 in the captured image Make settings for An example of the flow of processing at this time will be described with reference to FIG. Note that S1001 to S1005 in FIG. 10 are the same as S801 to S804 and S806 in FIG. The processing of S1001 to S1005 may be replaced with S501 to S503. In this case, when it is determined in S502 that the distance DstT is equal to or greater than the threshold StT, the process proceeds to S1006 described later.

  If it is determined in S1004 that the distance index Ld is less than StD (YES in S1004), then the subject position control unit 210 continues the horizontal direction of the left end and the right end of the subject not to be tracked in the captured image. The position is acquired (S1006). The subject position control unit 210 further obtains the horizontal center position of the subject to be tracked in the captured image (S1007). The subject position control unit 210 acquires, for example, the coordinate value Tp of the center position of the subject to be tracked and the coordinate values (Le and Re) of the left end and the right end of the subject not to be tracked. Then, based on the coordinate values Tp, Le, and Re acquired in S1006 and S1007, the subject position control unit 210 determines whether || Tp-Re |-| Tp-Le || is smaller than a predetermined threshold value StC. It determines (S1008). That is, in S1008, the subject position control unit 210 determines whether the tracking target subject is positioned near the center of the subject outside the tracking target in the horizontal direction. When the subject to be tracked is located in the vicinity of the center of the subject outside the tracking target in the horizontal direction, that is, || Tp-Re |-| Tp-Le || is smaller than StC. In the case (YES in S1008), the process proceeds to S1005. That is, in this case, it is considered that the subject position control unit 210 can capture the entire subject other than the tracking target by copying the subject of the tracking target to the center of the image to be captured. It is decided to use (S1005).

  On the other hand, if || Tp-Re |-| Tp-Le || is greater than or equal to StC (NO in S1008), the subject position control unit 210 compares | Tp-Re | with | Tp-Le | S1009). Based on this comparison, it is determined whether the tracking target subject is closer to the left or right of the non-tracking subject. That is, when | Tp−Le | is larger (YES in S1009), since the distance between the center of the tracking target subject and the left end of the tracking non-tracking subject is large, the tracking target subject is not the tracking target. It is close to the right side of the subject. For this reason, when | Tp−Le | is larger (YES in S1009), the subject position control unit 210 determines to use the pattern B for capturing the tracking target subject to the right in the captured image ( S1010). On the other hand, if | Tp−Re | is larger (NO in S1009), the distance between the center of the object to be tracked and the right end of the object other than the object to be tracked is large. It will be close to the left side of the subject. For this reason, when | Tp−Re | is larger (NO in S1009), the subject position control unit 210 determines to use the pattern C in which the tracking target subject is photographed to the left in the captured image ( S1011). As a result, it becomes possible to sufficiently capture an object outside the tracking target that is biased to the left or right of the tracking target.

  The subject position control unit 210 notifies the automatic tracking control unit 207 of the determined subject position (S1012). The automatic tracking control unit 207 sets an imaging range so that the tracking target subject appears at the notified position. The automatic tracking control unit 207 calculates, for example, an angle for rotating the camera 401, and instructs the system control unit 204 to perform pan / tilt control. Note that the pattern of the position of the subject in the image may be determined only when the same comparison result is continuously made a predetermined number of times in the comparison of S1004, S1008 and S1009.

(Processing example 4)
In the case where the object to be tracked is photographed at a position other than the center by the control of the imaging range as described above, it is important to be able to stably image the object other than the tracking object. On the other hand, when the imaging range is set so that the subject outside the tracking target is imaged, if the imaging range moves finely according to the movement of the tracking target subject, the subject outside the tracking target is stably imaged There are cases where you can not do it. Therefore, in the present processing example, when the automatic tracking control unit 207 receives a notification that the subject to be tracked is photographed at a position shifted from the center of the imaging range, the control of the imaging range is stopped (pan and tilt Do the lock). FIG. 11 shows an example of the flow of this process.

  The automatic tracking control unit 207 first receives, from the subject position control unit 210, a notification of the pattern of the position in the image to be photographed (for example, any of the pattern A and the pattern B described above) to be photographed (S1101) . Then, the automatic tracking control unit 207 confirms whether or not there has been a change in the pattern from the previous notification (S1102), and if there is no change (NO in S1102), returns the process to S1001. On the other hand, when there is a change in pattern from the previous notification (YES in S1102), the automatic tracking control unit 207 determines whether the change in pattern is a change from pattern A to pattern B (S1103). Then, when the change in pattern is a change from pattern A to pattern B (YES in S 1103), automatic tracking control section 207 controls the imaging range by adjusting the position where the subject should be photographed in the image. Later, the system control unit 204 is instructed to lock the control. If the instruction changes the pattern A to the pattern B, the imaging range is locked so that the imaging range does not change after the imaging range is controlled so that the subject 301 is photographed at the position designated by the pattern B. Is done. As a result, after the imaging range is set so that the subject outside the tracking target is imaged, the imaging range is prevented from finely moving according to the movement of the tracking target subject, and the subject outside the tracking target is stably imaged It is possible to If the change in pattern is a change from pattern B to pattern A (NO in S 1103), automatic tracking control unit 207 instructs system control unit 204 to release the lock of the imaging range ( S1105). As a result, when the subject to be tracked does not use the subject not to be tracked, it is possible to shift to a state in which the imaging range can be changed in accordance with the movement of the subject to be tracked.

(Processing example 5)
When the zoom target is set to the wide-angle setting while shooting is performed so that the tracking target subject is captured at the center of the image, the tracking target is not tracked without changing the position of the tracking target subject in the image. In some cases, it is possible to capture the entire subject. In this processing example, a method of determining the position of the tracking target object in the captured image in consideration of the zoom state will be described. FIG. 12 is a diagram showing an example of the flow of this process. This process is executed by the subject position control unit 210. In addition, since S1201 to S1204 and S1208 and S801 to S804 and S806 are the same, the description will be omitted. The processes of S1201 to S1204 and S1208 may be replaced with S501 to S503. In this case, when it is determined in S502 that the distance DstT is equal to or greater than the threshold StT, the process proceeds to S1205 described later.

  When it is determined in S1204 that the distance index Ld is less than StD (YES in S1204), the subject position control unit 210 subsequently acquires the zoom value Zm from the imaging unit 201 (S1205). Next, the subject position control unit 210 checks whether or not the zoom value Zm is smaller than the threshold value Zw (whether it is wide angle) (S1206). Then, when the zoom value Zm is smaller than the threshold Zw (YES in S1206), the subject position control unit 210 determines to use the pattern A as the position of the subject in the image (S1208). On the other hand, when the zoom value Zm is equal to or larger than the threshold Zw (NO in S1206), the subject position control unit 210 determines to use the pattern B as the setting of the position of the subject to follow in the image (S1207). . Then, the subject position control unit 210 notifies the automatic tracking control unit 207 of the determined subject position (S1209). The automatic tracking control unit 207 sets an imaging range so that the tracking target subject appears at the notified position. The automatic tracking control unit 207 calculates, for example, an angle for rotating the camera 401, and instructs the system control unit 204 to perform pan / tilt control. This prevents unnecessary tracking of the subject to be tracked at a position shifted from the center.

  In the above-described process, the horizontal position of the predetermined subject in the captured image is determined based on at least the distance between the camera and the predetermined subject to be tracked, but the present invention is not limited to this. In other words, if it is possible to specify the distance between a predetermined subject (for example, a person) to be tracked and another predetermined subject (for example, a whiteboard) not to be tracked, imaging of the predetermined subject is possible based on the distance. The horizontal position in the captured image may be determined. For example, apart from the camera, for example, a sensor capable of specifying an inter-subject distance in a direction orthogonal to the line-of-sight direction of the camera performing imaging (that is, depth direction when viewed from the camera) is prepared The position of a predetermined subject may be determined based on the distance. At this time, the inter-subject distance is specified more accurately by the inter-subject distance in the depth direction (as viewed from the camera) by the sensor and the inter-subject distance in the horizontal direction obtained by analyzing the image captured by the camera. May be In this case, when the inter-subject distance is small, the imaging range is set such that the predetermined subject other than the tracking target is also imaged. Thus, in a scene where an object outside the tracking object is to be imaged together with a predetermined object to be tracked, not only the predetermined object following the tracking object but an object outside the tracking object can be appropriately imaged.

<Other Embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  201: imaging unit, 202: image processing unit, 203: pan / tilt control unit, 204: system control unit, 205: storage unit, 206: communication processing unit, 207: automatic tracking control unit, 208: subject recognition unit, 209 A distance acquisition unit 210: an object position control unit 301: an object 302: an object area 303: a whiteboard (an object other than a tracking target) 401: a camera

Claims (19)

A control device that controls an imaging range of an imaging device that tracks and images a predetermined subject, the control device comprising:
Acquisition means for acquiring information of a first distance between the predetermined subject and the imaging device;
Setting means for setting the position where the predetermined subject appears in the image captured by the imaging device based at least on the first distance;
Control means for controlling the imaging range in accordance with the position set by the setting means;
A control device characterized by having. When the first distance is smaller than a predetermined threshold value, the setting means performs the setting so that the predetermined subject is photographed in a central area in the horizontal direction in the image, and the first distance is the predetermined distance. Setting is performed so that the predetermined subject is photographed in a predetermined area shifted from the center in the horizontal direction in the image,
The control device according to claim 1, characterized in that: The setting means performs the setting so that the predetermined subject is photographed in a central area in the horizontal direction in the image, when it is determined that the first distance is continuously smaller than a predetermined threshold a predetermined number of times. The setting is performed so that the predetermined subject is photographed in a predetermined area shifted from the center in the horizontal direction in the image when it is determined that the first distance is continuously larger than the predetermined threshold a predetermined number of times Do,
The control device according to claim 1, characterized in that: The acquisition means further acquires a second distance between the imaging device and another predetermined subject which is not a target of the tracking;
The setting means sets the horizontal position of the predetermined subject in the image captured by the imaging device, further based on the second distance.
The control device according to claim 1, characterized in that: The setting means performs the setting so that the predetermined subject is photographed in a central region in the horizontal direction in the image when the difference between the first distance and the second distance is larger than a predetermined threshold. The setting is performed so that the predetermined subject is photographed in a predetermined area shifted from the center in the horizontal direction in the image when the difference is smaller than the predetermined threshold.
The control device according to claim 4, characterized in that: The setting means is configured to set the predetermined central region in the horizontal direction in the image when it is determined that the difference between the first distance and the second distance is continuously larger than a predetermined threshold a predetermined number of times. The setting is performed so that the subject is photographed, and it is determined that the difference is continuously smaller than the predetermined threshold a predetermined number of times, the predetermined subject is shifted to a predetermined area shifted from the center in the horizontal direction in the image. Set the above settings to copy,
The control device according to claim 4, characterized in that: The acquisition unit acquires the second distance by specifying the size of a predetermined index attached to the other predetermined subject in the image captured by the imaging device.
The control device according to any one of claims 4 to 6, characterized in that: The acquisition unit acquires the second distance using a distance measurement sensor.
The control device according to any one of claims 4 to 6, characterized in that: The acquisition means further acquires the position of the predetermined subject in the horizontal direction and the position of the end of the other predetermined subject,
The setting means may further include a horizontal position of the predetermined subject in an image captured by the imaging device based on the position of the predetermined subject and the position of the end of the other predetermined subject. To set
The control device according to any one of claims 4 to 8, characterized in that. The acquisition unit acquires the first distance by specifying the size of a predetermined index attached to the predetermined subject in an image captured by the imaging device.
The control device according to any one of claims 1 to 9, characterized in that. The acquisition means acquires the first distance using a distance measurement sensor.
The control device according to any one of claims 1 to 9, characterized in that. When the control unit changes the setting by the setting unit from setting for photographing the predetermined subject in the center area in the horizontal direction to setting for photographing the predetermined subject in the predetermined area shifted from the center in the horizontal direction, The control of the imaging range is stopped after controlling the imaging range according to the setting after the change.
The control device according to any one of claims 1 to 11, characterized in that: While the control unit is stopping the control of the imaging range, the setting by the setting unit is set to a predetermined area shifted from the center in the horizontal direction. When the setting for photographing the predetermined subject is changed, the control of the imaging range is canceled.
The control device according to claim 12, characterized in that. The acquisition means further acquires the zoom state of the imaging device;
The setting means sets the horizontal position at which the predetermined subject appears in the image captured by the imaging device, further based on the state of the zoom.
The control device according to any one of claims 1 to 13, characterized in that: A control device that controls an imaging range of an imaging device that tracks and images a predetermined first subject,
Acquisition means for acquiring information on a distance between the first subject and a predetermined second subject different from the first subject;
Setting means for setting the position where the first subject appears in an image captured by the imaging device based on the distance;
Control means for controlling the imaging range in accordance with the position set by the setting means;
A control device characterized by having.   An imaging device comprising the control device according to any one of claims 1 to 15. A control method of a control device that controls an imaging range of an imaging device that captures and images a predetermined subject,
An acquiring step of acquiring information of a first distance between the predetermined subject and the imaging device;
A setting step of setting the setting position of the predetermined subject in the image captured by the imaging device based on at least the first distance;
A control step of controlling the imaging range in accordance with the position set in the setting step;
A control method characterized by comprising: A control method of a control device that controls an imaging range of an imaging device that captures and images a predetermined first subject,
An acquiring step of acquiring information on a distance between the first subject and a predetermined second subject different from the first subject;
A setting step of setting the setting position of the first subject in the image captured by the imaging device based on the distance;
A control step of controlling the imaging range in accordance with the position set by the setting means;
A control method characterized by comprising:   The program for functioning a computer as each means which the control apparatus in any one of Claims 1-15 has.

Images