JP2014143619A - Imaging device, remote operation terminal, camera system, control method and program for imaging device, and control method and program for remote operation terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a subject remotely designated by an operator be more accurately followed regardless of a communication time lag.SOLUTION: An imaging device externally controlled by a remote operation terminal comprises: communication means for granting grant information to an image signal generated by imaging means to transmit it to the remote operation terminal, receiving position information on a subject designated when the subject is designated in an image that is being displayed on a display unit of the remote operation terminal, and receiving the grant information from the remote operation terminal when the image signal is displayed on the display unit; holding means for holding the image signal generated by the imaging means on the basis of communication delay time calculated on the basis of the received grant information, for a predetermined period; selection means for selecting an image signal held in the holding means on the basis of communication delay time calculated when receiving the position information; and identification means for identifying a region for following the subject from a present image signal on the basis of the selected image signal and received position information.

Description

  The present invention relates to a system including an imaging apparatus having a tracking function for an arbitrary subject and a remote operation terminal that controls the imaging apparatus from the outside.

  Patent Document 1 discloses a camera system including a camera 2 and a controller 1 that can control the camera 2 from the outside. In this camera system, when the operator operates the controller 1, the focus position can be set (designated) to a desired subject position while observing captured image data transmitted from the camera 2. When the setting is made, the controller 1 transmits information related to the set subject position to the camera 2, so that the camera 2 executes AF processing or the like at the desired subject position based on the received position information. can do.

  Some imaging devices such as digital cameras can automatically follow a selected subject by selecting an arbitrary subject within the imaging screen.

JP 2009-273033 A

  In the above system that controls the camera 2 from the controller 1, it takes time to transmit and receive data by communication, which causes a communication time lag. For example, when an image captured by the camera 2 is displayed on the controller 1, it takes time to communicate captured image data, and thus it takes time to display a captured image on the controller 1. Due to this communication time lag, when the operator confirms the captured image with the controller 1, the image is displayed with a delay from the time when the image was captured. Similarly, when the controller 1 issues a control instruction to the camera 2, information on the position on the camera 2 side is delayed from the time when the desired subject position is designated on the controller 1 side due to the communication time lag. Will be received.

  As described above, since a communication time lag occurs in the above system, for example, when it is desired to perform follow-up control on a moving subject using the system, there is a possibility that the following problems may occur. For example, as shown in FIG. 7, when the operator designates and transmits the position of a desired subject (animal) with the controller 1, there is no possibility that the subject is located at the position designated by the operator when the camera 2 receives it. There is sex. This problem is particularly noticeable when the amount of movement of the subject is large or when the communication time lag is long. If the subject that the operator wants to follow is not already at the specified position, it will follow the subject that happens to be at the specified position at that time, so the focus and exposure adjustments differ from the operator's intention. There is a concern that will go. In such a case, in order to follow the subject more accurately, the controller 1 must pre-read the motion of the subject to be followed and consider the length of the communication time lag when selecting the subject. I must.

  The present invention relates to an imaging apparatus, a remote operation terminal, a camera system, an imaging apparatus control method and program, and a remote operation terminal control capable of causing a subject specified by an operator from a remote location to follow more accurately regardless of a communication time lag. Methods and programs are provided.

  An imaging apparatus according to an aspect of the present invention is an imaging apparatus that is externally controlled by a remote operation terminal, and includes an imaging unit that photoelectrically converts a subject image via an optical system and generates an image signal. When the subject is specified on the image displayed on the display unit of the remote operation terminal while transmitting the image signal and the grant information provided when transmitting the image signal to the remote operation terminal, the remote The position information specifying the subject is received from the operation terminal, and when the image corresponding to the image signal is displayed on the display unit, the addition information given to the image signal is received from the remote operation terminal. Communication means, a calculation means for calculating a communication delay time based on the received grant information, and an image generated by the imaging means based on the calculated communication delay time Holding means for holding a signal for a predetermined period, selection means for selecting an image signal held in the holding means based on the communication delay time calculated when the position information is received, and the selected image And a specifying unit that specifies a first tracking region of the subject from a current image signal based on the received signal and the received position information.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to follow a subject designated by an operator from a remote location more accurately without depending on a communication time lag.

It is a block diagram which shows the structure of the imaging device and remote control terminal in embodiment of this invention. It is a flowchart which shows the process of the imaging device and remote control terminal in embodiment of this invention. It is a flowchart which shows the process at the time of remote operation execution of the imaging device and remote control terminal in embodiment of this invention. It is a flowchart which shows the tracking start process of the imaging device in embodiment of this invention. It is a flowchart which shows the pattern matching process in embodiment of this invention. It is a flowchart which shows the follow-up continuation process of the imaging device in embodiment of this invention. It is the figure which showed the problem at the time of the tracking instruction | indication by a remote control terminal in the system of an imaging device and a remote control terminal. It is a figure showing the tracking subject specific process in embodiment of this invention in the system of an imaging device and a remote control terminal.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed depending on the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus having an image processing function and a remote operation terminal having a function of operating the imaging apparatus from a remote (external) position according to an embodiment of the present invention.

  As shown in FIG. 1, the camera system of the present embodiment includes a camera 10 and a remote operation terminal 20, and includes an imaging side communication unit 169 and a remote side communication unit 241 for communicating information respectively. It is possible to exchange information.

  First, the configuration of the camera (imaging device) 10 will be described. The camera 10 is configured to acquire an image signal from the photographing lens 11, and includes an image sensor (CMOS) 121, a CDS / AGC 122, a timing generator (TG) 123, and an image input controller 124. In addition, a tracking signal processing unit 125 and a subject area detection unit 126 are provided. The zoom lens driving unit 131, the aperture driving unit 132, the focusing driving unit 133, the operation unit 134, and the system control unit 14 are also provided. Further, a bus 15, a recording medium control unit 161, a recording medium 162, a display control unit 163, a display unit 164, an AF detection unit 165, an SDRAM 166, a ROM 167, a flash ROM 168, an imaging side communication unit 169, and a shake detection unit 17 are provided. .

  The system control unit 14 functions as a control unit that performs overall control of the operation of the entire camera, and controls each unit according to a predetermined control program based on an input from the operation unit 134.

  A ROM 167 connected via the bus 15 stores a control program executed by the system control unit 14 and various data necessary for control. The flash ROM 168 connected via the bus 15 stores various setting information related to the operation of the camera 10 such as user setting information. The SDRAM 166 connected via the bus 15 is used as a calculation work area for the system control unit 14 and also as a temporary storage area for image data. As described above, in the present invention, the SDRAM 166 functions as a holding unit for holding the image signal captured by the image sensor 121 for a predetermined period.

  The photographing lens (imaging optical system) 11 includes a fixed first group lens 111, a variable power lens 112, a diaphragm 113, a fixed second group lens 114, and a focus lens 115 in order from the subject side.

  The variable power lens 112 is driven by the variable power lens driving unit 131 and moves back and forth along the optical axis of the imaging optical system. Here, in this embodiment, the front side refers to the subject side, and the rear side refers to the image side. The focus competition lens 115 is driven by the focusing drive unit 133 and moves back and forth along the optical axis of the imaging optical system.

  The system control unit 14 controls the position of the zoom lens 112 and the position of the focus lens 115 by controlling the zoom lens drive unit 131 and the focusing drive unit 133 to perform zooming and focusing of the photographing lens 11.

  The diaphragm 113 is constituted by, for example, an iris diaphragm, and is driven by the diaphragm driving unit 132 to operate. The system control unit 14 controls the aperture driving unit 132 to control the aperture amount (aperture value) of the aperture 113 and control the amount of light incident on the image sensor 121.

  The imaging element (imaging means) 121 is a photoelectric conversion element (photodiode) configured by a CMOS sensor or a CCD sensor. The subject image formed on the light receiving surface of the image sensor 121 by the photographing lens 11 is converted into signal charges corresponding to the amount of incident light by the photodiode. The signal charge accumulated in each photodiode is sequentially read out from the image sensor 121 as a voltage signal (image signal) corresponding to the signal charge based on a drive pulse supplied from the TG 123 according to a command from the system control unit 14. In this embodiment, a CMOS sensor is used as the image sensor, but an image sensor having another configuration such as a CCD sensor can also be used.

  The CDS / AGC 122 is a correlated double sampling circuit (CDS) for removing reset noise (low frequency) included in the image signal output from the image sensor 121, and amplifies the image signal to a certain level. This is an AGC circuit for control. The CDS / AGC 122 performs correlated double sampling processing and amplifies the image signal output from the image sensor 121.

  The image input controller 124 takes in the image signal output from the CDS / AGC 122 and stores it in the SDRAM 166. The image signal stored in the SDRAM 166 is displayed on the display unit 164 by the display control unit 163. In the mode for recording an image signal, the recording medium control unit 161 records the image signal on the recording medium 162. Further, when the remote operation is performed by the remote operation terminal 20, the image-side communication unit 169 transmits the remote operation to the remote communication unit 241 in the remote operation terminal 20. Further, the image signal is held in the SDRAM 166 for the length of the communication delay time described later. It is a characteristic part of the present invention to hold an image signal corresponding to the communication delay time in the SDRAM 166, and details will be described later.

  The follow-up image generation unit 141 in the system control unit 14 takes in the image signal stored in the SDRAM 166 and generates a follow-up image (follow-up image signal) when the operation unit 134 instructs to follow. In the present embodiment, there is a touch panel as one of the operation units 134, and a predetermined area centered on the touched coordinates on the display image is set as a follow-up image. Instead of the touch panel, a system that can set the center coordinates of the area to be tracked by an operation key or the area itself may be used. The generated follow-up image signal is stored in the SDRAM 166. Further, the follow-up image generation unit 141 receives a past image signal based on the coordinate information generated by the remote-side follow-up image generation unit 211 in the remote operation terminal 20 to be described later when there is a follow-up instruction from the remote operation terminal side. From this, it is possible to generate a follow-up image signal. This is a characteristic part of the present invention, and details will be described later.

  The follow-up signal processing unit 125 takes in the follow-up image signal from the SDRAM 166 according to a command from the system control unit 14, and generates a feature amount including a color information histogram and a luminance information histogram from the follow-up image signal.

  The subject region detection unit 126 performs a tracking subject search process on the currently captured image signal based on the above-described feature amount, and determines whether there is a subject corresponding to the tracking subject from a predetermined region in the shooting screen. Is identified. When the tracking subject exists, the tracking region is determined based on the position coordinates where the subject exists, and the result is transmitted to the system control unit 14 to perform AF control and the like. Further, the position coordinates where the subject exists are held in the SDRAM 166, and the area where the tracking subject search process is performed can be limited by using this information when detecting the tracking subject region from the next time onward. When the tracking subject area is specified, the position coordinates held in the SDRAM 166 are updated whenever the position coordinates where the subject exists are updated.

  As described above, in the present invention, the follow-up signal processing unit 125 and the subject region detection unit 126 described above function as a specifying unit that specifies the follow-up region of the follow-up subject.

  When the system control unit 14 receives the tracking area described above, it transmits the tracking area information to the AF detection unit 165, and the AF detection unit 165 focuses the position including the tracking subject in the shooting screen based on the tracking area information. It is set as an area (focus detection area) used for detection. Next, the AF detection unit 165 calculates the degree of focus of the set focus detection area and transmits it to the system control unit 14. The system control unit 14 that has received the result controls the focusing drive unit 133 according to the received degree of focus, and the focusing drive unit 133 controls the focus compensator 115. Further, the system control unit 14 generates a histogram in the area based on the tracking area, and controls the aperture driving unit 132 based on the histogram so that the appropriate exposure set in the flash ROM 168 is obtained. AE control is performed on the tracking area.

  The imaging-side communication unit (first communication unit) 169 communicates with a remote-side communication unit 241 in the remote operation terminal 20 described later to exchange various information. The content of information to be communicated is a feature of the present invention and will be described later.

  Next, the configuration of the remote operation terminal 20 will be described. The remote operation terminal 20 includes a remote system control unit 21 and a remote operation unit 22. Also provided are a bus 23, a remote communication unit 241, a remote display control unit 242, a remote display unit 243, a remote SDRAM 244, a remote ROM 245, and a remote flash ROM 246. The remote operation terminal 20 is configured to be able to exchange information such as captured image signals and camera control information from the imaging side communication unit 169 in the camera 10 by the remote side communication unit 241.

  The remote system control unit 21 functions as a control unit that performs overall control of the operation of the entire remote operation terminal, controls each unit according to a predetermined control program based on an input from the remote operation unit 22, and controls the camera 10. Various control information is generated.

  The remote ROM 245 connected via the bus 23 stores a control program executed by the remote system control unit 21 and various data necessary for control. The remote flash ROM 246 stores various setting information relating to the operation of the remote operation terminal 20 such as user setting information.

  The remote SDRAM 244 is used as a calculation work area of the remote system control unit 21 and also as a temporary storage area for received image signals and the like.

  The remote display control unit 242 displays an image corresponding to the received image signal received from the imaging communication unit 169 by the remote communication unit 241 stored on the remote SDRAM 244 on the remote display unit 243.

  Next, communication between the remote side communication unit 241 and the imaging side communication unit 169 will be described in detail. The imaging-side communication unit 169 continues to transmit an image signal every time the imaging element 121 captures an image without confirming reception by the remote-side communication unit 241. The remote side communication unit (second communication unit) 241 stores the image signal in the remote side SDRAM 244 every time an image signal is received, and displays the image signal on the remote side display unit 243 by the remote side display control unit 242. If a new image signal is received from the imaging-side communication unit 169 before the display on the remote-side display unit 243 is completed, the newly received image signal is ignored and the remote-side display control unit 242 is executing Continue the current display process. In this way, streaming playback is performed on the remote operation terminal 20.

  Further, the remote side system control unit 21 transmits control information and setting change information for the camera 10 by the remote side communication unit 241. The system control unit 14 controls the camera 10 in accordance with the control information received by the imaging-side communication unit 169, and changes various setting information in the flash ROM 168 in accordance with the setting change information. Make a change. For example, when a zooming operation is performed by the remote system control unit 21, the system control unit 14 receives control information (zoom direction, zoom lens drive amount information) related to zooming through communication. Then, the system control unit 14 performs zooming by controlling the variable power lens driving unit 131 based on the control information. For example, when the recording image quality is changed by the remote-side system control unit 21, the system control unit 14 receives setting change information (recording image quality change information) related to the recording image quality by communication, and rewrites the information in the flash ROM 168. Change the recording quality setting.

  In this way, the camera 10 and the remote operation terminal 20 can communicate information, and can confirm the image signal at the remote operation terminal 20, control the camera 10, and the like.

  Further, when the camera 10 periodically transmits an imaging signal (image signal) captured by the imaging device 121 by the imaging-side communication unit 169, the time information of the system control unit 14 is also added to the image signal. Send. In addition to the control information transmitted to the camera 10, the remote operation terminal 20 is a time given to the image signal displayed when the remote display control unit 242 completes image display on the remote display unit 243. Information (granting information) is sent back to the camera 10 by the remote communication unit 241. In this way, the camera 10 transmits the captured image signal and also transmits time information when transmitting the image signal, and the remote operation terminal 20 displays the received image signal each time the remote display control unit 242 performs display. This time information is sent back to the camera 10. When the camera 10 receives the time information from the remote communication unit 241, the system control unit 14 (calculation unit) determines whether the camera 10 and the camera 10 are remote from the difference between the current time information of the camera 10 and the received time information. The communication delay time of the operation terminal 20 is calculated. The calculated communication delay time is held in the SDRAM 166.

  Next, in a characteristic part of the present invention, a flow from when the subject operation is instructed by the remote operation terminal 20 until the imaging apparatus performs the operation will be described. The remote side follow-up image generation unit 211 (generation unit) in the remote-side system control unit 21 generates coordinate information (position information) for which an instruction has been issued when the follow-up instruction is given by the remote-side operation unit 22. Do. In the present embodiment, the remote operation unit 22 is a touch panel, and the touched coordinates in the image displayed on the remote display unit 243 are generated as coordinate information. The remote side operation unit 22 may be a system that can set the coordinates of a subject area to be tracked by an operation key, instead of a touch panel. The generated coordinate information is transmitted to the imaging side communication unit 169 by the remote side communication unit 241 according to an instruction from the remote side system control unit 21. The system control unit 14 holds the coordinate information received from the imaging-side communication unit 169 in the SDRAM 166 and starts the follow-up start process on the assumption that the follow-up instruction has been given by the remote operation terminal 20.

  First, the system control unit 14 (selecting means) selects a past image signal corresponding to the communication delay time from the time of receiving the coordinate information, among the past image signals held in the SDRAM 166. The past image signal corresponding to the communication delay time selected here is the same image signal as when the subject is selected on the remote operation terminal side. The system control unit 14 takes the selected past image signal into the follow-up image generation unit 141 and generates a follow-up image from a predetermined area centered on the received coordinate information. Next, the follow-up signal processing unit 125 takes in the generated follow-up image and generates a feature amount including a color information histogram and a luminance information histogram from the follow-up image signal as described above. The subject region detection unit 126 performs a tracking subject search process based on the generated feature amount for the image signal held in the future one after the selected past image signal, and follows the predetermined region in the shooting screen. It is determined whether there is a subject corresponding to the subject. When the tracking subject exists, the tracking region is determined based on the position coordinates where the tracking subject exists. When the tracking instruction is given by the camera 10, the tracking area is determined for the image signal currently being picked up, so it can be assumed that the tracking subject can be specified in real time.

  However, when a follow-up instruction is given from the remote control terminal 20, there is a communication time lag, and the camera is in a stage where a follow-up subject is specified for a past image signal. Therefore, on the camera side, it is necessary to perform a tracking subject specifying process for one future image signal from a past image signal, and to continue this process until the current image signal is reached. When the system control unit 14 can identify the tracking area for the image signal held in the future, the system control unit 14 performs tracking signal processing on the image signal held in the future further than the image signal held in the future. The tracking subject is specified by the unit 125 and the subject region detection unit 126. In this way, the tracking subject is repeatedly identified, and the tracking subject is sequentially identified from the past image signal selected based on the communication delay time in the time series order. To identify. However, if the communication delay time is extremely short, the tracking subject may be specified for the current image signal based on the selected past image signal without sequentially specifying the tracking subject in chronological order. Good. When a tracking subject is identified for the current imaging signal, the result is transmitted to the system control unit 14, and AF and AE control (adjustment of focus and exposure) are performed.

  In addition, the position coordinates where the tracking subject specified in the current image signal exists is held in the SDRAM 166, and the tracking subject search process is performed by using this information when detecting the tracking subject region from the next time on. The area can be limited. When the tracking subject area is specified, the position coordinates held in the SDRAM 166 are updated whenever the position coordinates where the subject exists are updated.

  According to the above-described procedure, the follow operation is instructed from the remote operation terminal 20 and the follow is started on the camera side (FIG. 8). In performing tracking, the image for which the tracking instruction is given and the image for extracting the feature value from the tracking position must be the same. This is because if the images are not the same, there is a possibility that the subject to be followed is not at the position where the follow instruction has been issued. The camera 10 measures the communication delay time between the camera 10 and the remote operation terminal 20 so that the feature amount can be extracted from the same image as the image for which the follow-up instruction is given by the remote operation terminal 20, and the communication delay in advance. The image signal for the time is held.

  At this time, the image signal is retained for the communication delay time because the communication delay causes the communication delay time when the camera 10 receives the coordinate information when the remote operation terminal 20 instructs to follow. This is because the image at the time point advanced by the amount is displayed. Since this communication delay time changes from moment to moment according to the state of communication, past images are held accordingly. The past images may be held for a time slightly longer than the communication delay time. When a follow-up instruction is given by the remote operation terminal 20, the camera 10 selects a past image for the communication delay time at the time of receiving the coordinate information, and extracts a feature amount from an area centered on the received coordinate information of this image. Extract. Based on this feature quantity, past image signals are sequentially subjected to pattern matching in chronological order, and finally a tracking subject is specified from the image signal currently captured. Even if the tracking subject is found at the time of a certain past image, there is a possibility that the currently captured image signal is far away from the position.

  For example, when pattern matching is performed on the current image signal at the stage where the feature amount has been extracted, if the subject to be followed is greatly separated, there is a possibility that the subject cannot be found if the pattern matching area is narrow. Further, if the pattern matching area is wide, there is a possibility that the subject to be followed is erroneously recognized. By sequentially specifying subjects in chronological order, the tracking subject can be specified more accurately. With such a system, it is possible to improve the communication time lag problem and accurately specify the subject to be followed even when following the instruction from the remote operation terminal 20.

  In the present embodiment, the refresh interval of the past image held by the camera 10 is all the captured images, but may be thinned and held in order to reduce the storage capacity of the SDRAM 166. This refresh interval needs to be set to an interval at which at least a subject to be tracked can be determined when specifying subjects in time series. This is determined based on the pattern matching range when the subject is specified and the maximum amount of movement that follows the subject. In the present embodiment, the image touched by the remote operation terminal 20 is specified from the held past image signal based on the communication delay time. However, an image having the same frame information as the image touched by the remote operation terminal 20 is used. You may make it the process to select. When the refresh interval is increased, there may be no past image with the same communication delay time or frame information. In this case, the closest image is selected.

  Next, the subject tracking function will be described in detail. Here, a case where a follow instruction is given by the camera 10 will be described as an example. When the operation unit 134 instructs the subject to follow, the follow-up image generation unit 141 uses the positional information when the follow-up instruction is given, in this embodiment, a predetermined range centered on the touch coordinates of the touch panel as the follow-up image. Generate. The predetermined range needs to be at least large enough to generate a feature amount to the extent that erroneous tracking does not occur frequently in performing tracking control. The predetermined range may be fixed, or may be a system that can be changed by a user setting or the like. For the generated follow-up image, the follow-up signal processing unit 125 generates a histogram of color information and a histogram of luminance information, which are used as feature amounts of the follow-up image. In the present embodiment, the color and luminance histograms are generated, but at least one of the color and luminance histograms may be generated.

  Further, the subject area detection unit 126 generates an image having the same size as the follow-up image described above from the image signal currently captured, and sets this as a follow-up candidate image. Similar to the follow-up image, the follow-up signal processing unit 125 generates a color and luminance histogram from the follow-up candidate image, and uses this as a feature amount of the follow-up candidate image. The subject area detection unit 126 calculates a difference for each color information and a difference of luminance information from the feature amount of the follow-up image and the feature amount of the follow-up candidate image, and further calculates the sum of these calculated values. Hold this. At the same time, the position information of the tracking candidate image is also held.

  Furthermore, matching is performed by sequentially generating a tracking candidate image from the image signal currently captured, and the difference between the feature amount of the tracking candidate image and the feature amount element of the tracking image and the sum thereof are calculated. I will do it. The value at the time when the sum of the differences of the feature amount elements is the smallest and the position information of the follow-up candidate image at that time are overwritten on the previously stored information and held. If the sum of the differences between the elements of the finally obtained feature values is smaller than a predetermined value, the tracking candidate image is specified as a tracking subject, and a predetermined range is set as a tracking region from the held position information. The predetermined value related to the sum of the difference between feature amounts is determined based on the ratio that can follow the subject that you want to follow and the ratio that does not follow the subject that should not follow when performing the tracking operation. To do. In this way, the tracking area is determined and tracking is started. In addition, when performing a follow-up instruction from the remote operation terminal 20, it is necessary to perform pattern matching sequentially from past images.

  In addition, a method for continuing tracking will be described. As information for continuing the tracking, the SDRAM 166 holds position information in which the tracking candidate image and the tracking area are set. At the time of continuous tracking, each time imaging is performed, based on the tracking image held in the SDRAM 166, the subject region detection unit 126 and the tracking signal processing unit 125, as described above, obtain the captured image signal and the tracking image. Perform matching.

  At this time, the range of the image signal to be matched is limited based on the position information in which the tracking area that is held is set. If matching is performed only for a predetermined range centering on the position where the tracking area is set, it is possible to achieve false tracking by matching the extra range and shorten the calculation time. The predetermined range for matching is the maximum distance that the subject that wants to follow can move within the imaging screen during the time between the follow-up continuation process and the next follow-up continuation process. Set it to a range that can sometimes be covered. If this range is large, it can follow even if the subject moves large, but there will be many false follow-ups. If the range is small, it will not be able to follow if the subject moves large, but there will be less false follow-up. In this way, when matching is performed again to continue tracking, as described above, if the sum of the differences between the elements of the finally obtained feature amount is smaller than a predetermined value, the tracking candidate image is a tracking subject. Specifically, a predetermined range is updated as the tracking area from the position information of the tracking candidate image.

  In addition, the position information of the tracking candidate image at this time is overwritten and updated with the position information held in the SDRAM 166. In the next follow-up continuation process, an appropriate matching range can be set by using the updated position information. In this way, the tracking is continued. At this time, it is not necessary to perform the follow-up continuation process every time an image is taken. The longer the imaging interval for performing the follow-up continuation process, the less follow-up is possible when the subject moves greatly, but the processing time can be reduced.

  A case where the tracking is stopped will be described. When a follow-up stop instruction is issued by the operation unit 134 or the remote-side operation unit 22 when the follow-up is continued, the system control unit 14 stops the follow-up and returns to the non-follow-up operation. For example, the AF area set for the tracking area is returned to the AF area that is normally set when the tracking is stopped. Similarly, if the subject to be followed cannot be specified when the follow-up is continued and a predetermined time has elapsed, the follow-up is stopped. If the predetermined time to wait while the tracking subject cannot be specified is long, there is a high possibility that the tracking subject can be specified and returned again, but the process of returning to the normal state is delayed. The reverse is true when the predetermined time to wait while the tracking subject cannot be identified is short.

  In this way, subject tracking start, continuation, and stop processing are performed.

  Next, operations of the camera 10 and the remote control terminal 20 having the above-described configuration in this embodiment will be described with reference to FIGS.

  FIG. 2 is a flowchart showing camera remote operation processing performed by the system control unit 14 in the camera 10 and the remote-side system control unit 21 in the remote operation terminal 20 in the present embodiment. The flowchart on the left side of FIG. 2 represents the operation of the camera 10 in the camera remote operation mode, and each process is indicated by “SC”. The flowchart on the right side of FIG. 2 represents the operation of the remote operation terminal 20 in the camera remote operation mode, and each process is displayed as “SR”.

  First, the processing on the camera side will be described. Step SC001 represents camera control when the camera remote operation is not being performed (hereinafter referred to as normal camera control), and proceeds to step SC002 after the control is performed. Since remote operation is not performed during normal control of the camera, the details are omitted. However, only the camera performs various camera control, imaging signal display, and follow-up instructions. Step SC002 judges whether camera remote operation mode is performed.

  If there is an instruction to execute the camera remote operation mode in step SC001, the process proceeds to step SC003, and if there is no instruction, the process proceeds to step SC001. In step SC003, it is determined whether or not the camera remote operation mode is cancelled. If there is no cancel, the process proceeds to step SC004, and if there is a cancel, the process proceeds to step SC001. If communication with the remote operation terminal 20 is established according to the camera remote operation mode execution instruction and the remote operation mode is canceled, normal control of the camera is performed in SC001.

  In SC004, communication with the remote control terminal 20 is established. A communication establishment instruction is transmitted to the remote operation terminal 20, and if communication establishment possible information can be received from the remote operation terminal 20, a communication establishment completion information is transmitted to the remote operation terminal 20 assuming that communication has been established. At the same time, the process proceeds to step SC005. If communication cannot be established due to timeout in response to the communication establishment instruction transmitted to the remote operation terminal 20, the process returns to step SC003 to determine again whether or not the camera remote operation mode has been canceled.

  In step SC005, the camera setting information is transmitted to the remote operation terminal 20 in order to share the camera setting at the time of establishing communication with the remote operation terminal. After executing the process of step SC005, the process proceeds to step SC006. Further, although details will be described later, even when there is an instruction to change the setting from the remote operation terminal 20 to the camera 10, the setting information of the camera is transmitted to the remote operation terminal 20 in order to share this setting. . In step SC006, it is determined whether the current camera mode is the shooting mode or the playback mode. If it is the shooting mode, the process proceeds to step SC007, and if it is the playback mode, the process proceeds to step SC008. Step SC007 is a characteristic part of this embodiment and will be described later with reference to FIG.

  In step SC008, processing in the playback mode is performed. At the same time as reproducing the moving image taken on the camera side, the reproduced moving image is displayed on the remote operation terminal 20. In addition, the camera 10 is selected from the remote operation terminal 20 to select a moving image to be played back and execute various commands (playback, stop, fast forward, etc.) during the playback of the moving image. Details are omitted. If the process of step SC007 or step SC008 is executed, the process proceeds to step SC009. In step SC009, it is determined whether or not the power is turned off. If not turned off, the process proceeds to step SC010, and if turned off, the process is terminated. In step SC010, it is determined whether or not there is an instruction to cancel the camera remote operation mode from the remote operation terminal 20. If there is no instruction to cancel the camera remote operation mode, the process proceeds to step SC011. If there is a cancel instruction, the process returns to step SC001 to perform normal camera control without remote control.

  In step SC011, it is determined whether or not there has been an instruction to switch between the photographing mode and the reproduction mode by the remote operation terminal 20. If there is a mode switching instruction, the process proceeds to step SC012, and if there is no mode switching instruction, the process returns to step SC005. In step SC012, the camera setting is changed to the playback mode if the current shooting mode is selected, and the camera setting is changed to the shooting mode if the current playback mode is set, and the process returns to step SC005. In step SC005, the switching between the shooting mode and the reproduction mode and the change of the camera setting information performed in step SC007 are shared with the remote operation terminal, and the operation is performed in the set mode.

  Next, processing on the remote operation terminal side will be described. Step SR001 represents control of the remote operation terminal when the camera remote operation is not performed (hereinafter referred to as normal control of the remote operation terminal). Since remote operation is not performed during normal control of the remote operation terminal, the details are omitted, but other various functions (Internet, music reproduction, moving image reproduction, etc.) provided in the remote operation terminal are executed. In step SR002, it is determined whether or not the camera remote operation mode is executed. In step SR001, if the camera remote operation mode is executed, the process proceeds to step SR003, and if the remote operation mode is not executed, the various processes performed in step SR001 are continued. In step SR003, it is determined whether or not the camera remote operation mode is cancelled. If there is no cancel, the process proceeds to step SR004, and if there is a cancel, the process returns to step SR001.

  In SR004, communication with the camera is established. If the communication establishment instruction from the camera 10 can be received, the communication establishment possible information is transmitted to the camera 10, and if the communication completion information from the camera 10 can be received again, it is determined that the communication is established and the process proceeds to step SR005. move on. If the communication establishment instruction cannot be received from the camera 10 and communication cannot be established due to timeout, the process returns to step SR003 to determine again whether the camera remote operation mode has been canceled. If there is an instruction to cancel the camera remote operation mode, the process returns to SR001 to resume various controls of the remote operation terminal 20. In step SR005, the setting similar to the setting information received on the remote operation terminal side is performed by receiving the setting information from the camera 10 in order to reflect the setting of the camera at the time of establishing the communication to the remote operation terminal.

  For example, by receiving whether the current camera focus mode is AF or MF and setting it, control is performed such that an icon for manually controlling the focus is grayed out during AF. After executing the process of step SR005, the process proceeds to step SR006.

  Although details will be described later, even when a setting change instruction is issued from the remote operation terminal 20 to the camera 10, the setting information of the camera 10 is received again in order to share this setting, and the remote operation terminal 20 Reflect the setting to. In step SR006, it is determined whether the current camera mode is the shooting mode or the playback mode. If the current camera mode is the shooting mode, the process proceeds to step SR007, and if it is the playback mode, the process proceeds to step SR008.

  Step SR007 is a characteristic part of this embodiment and will be described later with reference to FIG. In step SR008, processing in the playback mode is performed. Selection of a moving image to be reproduced from the remote operation terminal 20 and execution of various commands (play, stop, fast forward, etc.) during moving image reproduction are performed on the camera 10 and display of the moving image being reproduced by the camera 10 I do. Details are omitted. If the process of step SR007 or step SR008 is executed, the process proceeds to step SR009.

  In step SR009, it is determined whether the power is turned off. If not turned off, the process proceeds to step SR010, and if turned off, the process is terminated. In step SR010, it is determined whether the remote operation terminal 20 has issued a camera remote operation mode cancel instruction. If the camera remote operation mode cancel instruction has been executed, the process proceeds to step SR011, and if not, step SR012 is executed. Proceed to

  In step SR011, a camera remote operation mode cancel instruction is transmitted to the camera 10, and the process returns to step SR001 to perform normal control of the remote operation terminal. In step SR012, it is determined whether or not the remote operation terminal 20 has issued an instruction to switch between the shooting mode and the playback mode. When the mode switching instruction is executed, the process proceeds to step SR013. When the mode switching instruction is not executed, the process returns to step SR005, and the processing of the remote operation terminal in the camera remote operation mode is continued. In step SR013, a camera mode switching instruction is transmitted to camera 10, and the process returns to step SR005. In step SR005, the switching between the photographing mode and the reproduction mode is shared with the remote operation terminal, and the operation is performed in the set mode.

  Next, the remote control camera control in step SC007 and the remote control terminal control in step SR007 of FIG. 2, which are features of the present invention, will be described with reference to the flowchart of FIG.

  The flowchart on the left side of FIG. 3 shows camera control during remote operation, and the flowchart on the right side represents control of the remote operation terminal during remote operation.

  First, the processing on the camera side will be described. In SC101, the video signal (image signal) and the time when the camera 10 transmits the video signal to the remote operation terminal 20 (hereinafter referred to as camera video transmission time) are transmitted to the remote operation terminal 20, and the process proceeds to step SC102. move on. The camera side transmits the video signal without confirming reception of the remote operation terminal 20. In step SC102, it is determined whether or not the camera setting has been changed by the remote operation terminal 20. If the camera setting has been changed, the process proceeds to step SC103, and if there has been no change, the process proceeds to step SC104. In step SC103, the camera setting changed by the remote operation terminal 20 is reflected, and the process proceeds to step SC104.

  For example, when the recording image quality is changed on the remote operation terminal side, the recording image quality change setting is reflected on the camera side in step SC103. In step SC104, it is determined whether or not a follow-up start instruction is given by the remote operation terminal 20. If a follow-up start instruction is given, the process proceeds to step SC105, and if not, the process proceeds to step SC108. In this embodiment, when tracking subject position information (tracking subject coordinate information) is received from the remote operation terminal 20, it is considered that a tracking start instruction has been issued.

  In step SC105, a tracking start process is performed based on the received tracking subject position information, and the process proceeds to step SC106. The process of step SC105 is a characteristic part of the present invention, and a detailed description thereof will be described later with reference to FIG. In step SC106, it is determined whether or not the follow-up can be performed in step SC105. If the follow-up cannot be performed, the process proceeds to step SC107. If the follow-up is performed, the process proceeds to step SC112, and other processes other than the follow-up process are performed.

  In step SC107, follow-up impossible information is transmitted to remote operation terminal 20 because follow-up cannot be performed. Whether or not the follow-up can be performed is determined by whether or not follow-up impossible information indicating whether or not the follow-up can be performed by 1 bit is issued in step SC105. Next, in step SC108 which proceeds when there is no follow-up start instruction in step SC104, it is determined whether or not follow-up is currently being performed. If follow-up is being performed, the process proceeds to step SC109. If not, the process proceeds to step SC112, and the process proceeds to a process other than the follow-up process. In step SC109, which proceeds when currently following, a follow-up continuation process is performed, and the process proceeds to step SC110. A detailed description of the follow-up continuation process will be described later with reference to FIG. In step SC110, it is determined whether there is follow-up impossible information in step SC109, or whether follow-up stop is designated from the remote operation terminal 20. If applicable, the process proceeds to step SC111. If not, the process proceeds to step SC112, and the process proceeds to a process other than the follow-up process.

  In step SC111, a follow-up stop process is performed, the process proceeds to step SC112, and a process other than the follow-up process is performed. When the tracking cannot be continued, that is, when the tracking subject cannot be specified, or when the user wants to stop the tracking, the tracking is stopped and the normal shooting state is restored. The normal shooting state in the present embodiment is a state in which a frame for adjusting AF or AE is set at the center of the screen instead of the tracking area. Of course, the normal shooting state is not limited to the configuration of the present embodiment, and any other configuration may be used as long as the tracking is not performed.

  Next, in step SC112, which proceeds after processing related to tracking, etc., it is determined whether there has been a camera control instruction from the remote operation terminal 20. If there is a camera control instruction, the process proceeds to step SC113, and if not, step SC113 is performed. Proceed to SC114. In step SC113, the camera is controlled in accordance with the camera control instruction from the remote operation terminal 20, and the process proceeds to step SC114. As an example of the processing performed in step SC112 and step SC113, when an instruction for zooming is given at the remote operation terminal 20, zooming is performed accordingly. In step SC114, camera settings and control are performed by the camera body, and the process proceeds to step SC115. In step SC114, a camera setting change or camera control instruction is received by operating the camera body, and the camera is controlled accordingly.

  Further, camera control performed without following the user's instruction such as AF is also performed here. Details are omitted. In step SC115, it is determined whether the camera video transmission time has been received from remote operation terminal 20. If received, the process proceeds to step SC116, and if not received, the process is terminated. In step SC116, the communication delay time is calculated and the process is terminated.

  Next, processing on the remote operation terminal side will be described. In step SR101, reception of the video signal from the camera 10 is started, and the camera video transmission time is received, and the process proceeds to step SR102. Although the received video signal is displayed in step SR114, which will be described later, the processing from step SR102 to step SR113, which will be described later, is already performed in step SR114 during the remote control terminal control during the previous remote operation. Will be described as being performed. In step SR102, it is determined whether there has been a camera setting change instruction from the user at remote operation terminal 20. If there is a setting change instruction, the process proceeds to step SR103, and if not, the process proceeds to step SR104.

  In step SR103, the camera setting change information is transmitted to the camera 10. Next, in step SR104, it is determined whether or not a follow-up start instruction has been issued at the remote operation terminal 20. If a follow-up start instruction has been issued, the process proceeds to step SR105, and if not, the process proceeds to step SR109. In the present embodiment, a video signal is displayed on the remote operation terminal 20 and a follow-up start instruction is issued when a subject is touched on a touch panel that can be operated by touch.

  In step SR105, the coordinates touched by the user are generated as tracking subject position information, and the process proceeds to step SR106. In step SR106, the tracking subject position information is transmitted to camera 10, and the process proceeds to step SR107. Next, in step SR107, as a result of the tracking start process performed in step SC105 on the camera side, it is determined whether or not follow-up disable information transmitted from the camera 10 is received when pattern matching fails and tracking cannot be performed. To do. If there is follow-up impossible information, the process proceeds to step SR108; otherwise, the process proceeds to step SR109.

  In step SR108, in order to inform the user that tracking cannot be performed on the camera side, a message indicating that tracking cannot be performed is displayed, and the flow proceeds to step SR109. By displaying that the follow-up could not be performed in this way, the user is prompted to perform the follow-up instruction again. In step SR109, it is determined whether or not a follow-up stop instruction has been given by the remote operation terminal 20. If a follow-up stop instruction has been issued, the process proceeds to step SR110. Otherwise, the process proceeds to step SR111. This follow-up stop instruction is an operation performed when the user wants to stop following when currently following. When the user touches the follow stop icon displayed on the remote operation terminal 20, a follow stop instruction is given.

  In step SR110, a follow-up stop instruction is transmitted to camera 10, and the process proceeds to step SR111. In step SR111, it is determined whether or not there has been an instruction for camera control from the remote operation terminal 20. If there is an instruction for camera control, the process proceeds to step SR112, and if not, the process proceeds to step SR113. In step SR112, the camera control information performed at the remote operation terminal 20 is transmitted to the camera side, and the process proceeds to step SR113. For example, when zooming is performed by the remote operation terminal 20, control information is transmitted to the camera 10 so as to perform zooming. In step SR113, it is determined whether reception of the video signal started in step SR101 is completed. If completed, the process proceeds to step SR114. If not completed, the process proceeds to step SR115.

  In step SR114, display processing of the video signal that has been received is performed, and the flow proceeds to step SC116. In step SR116, the camera video transmission time received in step SR101 is sent back to the camera side as it is, and the process ends. In step SR115, various instruction information for the camera 10 performed in steps SR102 to SR112 is initialized (in the state where there is no instruction), the process returns to step SR102, and the process is repeated from step SR102. The camera side transmits a video signal to the remote operation terminal 20 every time it takes an image. However, the remote operation terminal 20 does not always display all of them, and the processing capability of the remote operation terminal 20 and the line status are limited. Dependent. As the processing capability of the remote operation terminal 20 is lower, the video signal is displayed at a slower cycle than the cycle displayed on the camera side. While the video signal started to be received in step SR101 is continuously received, the remote operation terminal 20 can change the camera settings, perform camera control, and instruct to start following the currently displayed image. To.

  That is, until the reception is completed in step SR113, the processing from step SR102 to step SR112 can be performed via the processing in step SR115. When the reception of the video signal is completed, the video signal is displayed on the remote operation terminal 20 in step SR114, and then the video signal is received again in step SR101 after the remote operation terminal control is performed at the time of remote operation. Start. While receiving the video signal again, it receives various processing instructions for the currently displayed video signal. If no video signal is displayed yet at the stage where reception of the video signal is started in step SR101, such as immediately after the start of remote control, steps SR102, SR104, step SR107, step SR109, and step SR111 are performed. All processes are N. By doing so, various steps are skipped without receiving various control instructions. Then, after the video signal is displayed for the first time in step SR114, various control instructions in steps SR102 to SR112 are accepted.

  In the present embodiment, when the camera side periodically transmits a video signal in SC 101, the camera side also transmits a camera video transmission time indicating the transmission time. On the other hand, every time the remote operation terminal 20 completes the reception of the video signal and displays the video signal in step SR114, the camera video transmission time is sent back to the camera side in step SR116. In step SC116, the camera side calculates a communication delay time from the difference between the time when the camera video transmission time is received and the camera video transmission time. This communication delay time represents a round-trip communication time between the camera and the remote operation terminal until the camera 10 transmits an image, the remote operation terminal 20 displays a video signal, and communication with the camera is completed. The communication delay time varies sequentially depending on the communication line status.

  As described above with reference to FIG. 8, in order to instruct the subject to follow, it is necessary to specify the subject from the same image as the image for which the user has issued the subject follow-up instruction and the same coordinates as the coordinates for which the follow-up instruction has been issued. There is. When a follow-up instruction is given from the remote operation terminal 20, the camera side has captured a future image rather than an image already displayed on the remote operation terminal 20. Therefore, a past image is held on the camera side in order to issue a follow-up instruction to an image similar to the image for which the follow-up instruction has been given by the remote operation terminal 20. In order to hold an image for which a follow-up instruction has been given by the remote operation terminal 20, it is necessary to hold at least the length of the communication delay time described above (the round-trip communication time between the camera and the remote operation terminal). The camera 10 always keeps past images in this way. Then, when a follow-up start instruction is issued in step SR104, based on the subject position information communicated in step SR106 described above and the communication delay time calculated in step SC116 when the follow-up start instruction is given, In step SC105, follow-up start processing is performed.

  Next, the follow-up start process in step SC105 in FIG. 3 which is a characteristic part of the present invention will be described with reference to the flowchart in FIG.

  In step SC201, a past image corresponding to the current communication delay time is selected as an image from which a feature amount is extracted from the held past images, and the process proceeds to step SC202. The feature amount to be extracted is a histogram of color information and a histogram of luminance information as described above. In the present embodiment, a past image is selected based on the communication delay time. However, a system may be used in which frame information of an image is held together with the past image, and an image with matching frame information is selected. In step SC202, feature amounts are extracted from a predetermined region centered on the subject position information received from the remote operation terminal 20 for the image selected in step SC201, and the process proceeds to step SC203.

  In step SC203, one of the future images is selected again from the currently selected images among the held past images, and the process proceeds to step SC204. Immediately after the process of step SC202, one future image is selected from the image selected in step SC201. In step SC204, pattern matching is performed based on the feature amount extracted in step SC202 and the image selected in step SC203, and the process proceeds to step SC205. Details of step SC204 will be described later with reference to FIG.

  In step SC205, it is determined whether or not follow-up failure information has been issued in the process of step SC204. If it has not been issued, the process proceeds to step SC206. If it has been issued, it is determined that follow-up is impossible and the process ends. In step SC206, it is determined whether pattern matching has been executed for the image captured at the current time. If it can be executed, the process proceeds to step SC207. If not, the process returns to step SC203. As described above with reference to FIG. 8, the processing from step SC203 to step SC206 is repeated, and a region (second tracking region) where the tracking subject is sequentially present with respect to the held past image is found. Then, pattern matching is finally performed on the image captured at the current time, and the region where the tracking subject is present (first tracking region) is specified. As described above, the tracking subject is sequentially chased and found in time-series order, so that the tracking subject can be identified from the currently captured image even if the tracking subject moves greatly.

  Next, in step SC207, which proceeds when pattern matching can be performed on the image captured at the current time in step SC206, a tracking area is set in the area centered on the position information held last. Proceed to SC208. The position information held at the end corresponds to the position where the subject to be tracked can be found in the image captured at the current time. The camera 10 performs AF and AE processing on this tracking area in step SC114 in FIG. In step SC208, as the information for continuing the tracking, the tracking image in which the feature amount is extracted in step SC202 and the position information in which the tracking area is set in step SC207 are held, and the process is terminated. Based on these pieces of information, follow-up continuation processing is performed at step SC109 in FIG.

  As described above, the image for which the follow-up instruction is given by the remote operation terminal 20 is identified from the past image according to the communication delay time, and the feature amount of the follow-up subject is determined based on the position information of the follow-up subject received from the remote operation terminal 20. Extract. Further, the subject to be tracked is specified from the currently captured image by performing pattern matching on each image in chronological order from the specified past image based on the feature amount. By storing past images corresponding to the communication delay time on the camera side, finding an image similar to the image for which the follow-up instruction has been given by the remote operation terminal 20, and extracting a feature amount based on the position information, so that the communication time lag It is possible to prevent erroneous tracking of the subject and specify the tracking subject more accurately. The camera can easily set a suitable focus and exposure for the subject that the user wants to follow by performing control such as AF and AE on the following area.

  Next, the pattern matching process in step SC204 in FIG. 4 will be described using the flowchart in FIG. In step SC301, a pattern matching range is set for the image selected in step SC203, and the process proceeds to step SC302. The range set here sets an area centered on the position information last held in step SC306, which will be described later, in the previous pattern matching process. This position information is a position at which a subject to be followed exists. When the pattern matching process is the first time, it is based on the position information received from the remote operation terminal 20 used when calculating the feature amount in step SC202 in FIG. In step SC302, feature amounts are extracted from a part of the pattern matching range, and the process proceeds to step SC303.

  The size of the region from which the feature amount is extracted in step SC302 is the same as the region from which the feature amount is extracted in step SC202 in FIG. Further, every time the process returns to step SC302 by SC308 described later, the position where the image signal is cut out sequentially from within a predetermined range in the image signal is changed to perform matching with the image signal. Note that the wider the matching range is, the higher the possibility that a tracking subject can be found, but the false tracking also increases. As a method for changing the cutout position, the cutout position is changed for each line, and when one line ends, the next line is cut out. Of course, the column may be cut out with priority. In addition, the cutout range may or may not overlap the cutout images, and is determined by the required tracking accuracy. When the cutout images overlap, for example, when the cutout image is generated every time one pixel is shifted, the tracking accuracy increases in order to achieve more precise matching, but the amount of calculation for matching increases.

  On the other hand, for example, when the range adjacent to the follow-up image is set as the follow-up image so that the follow-up images do not overlap, the amount of calculation for matching is reduced, but the follow-up accuracy is lowered. Next, in step SC303, the difference between the feature amount extracted in step SC202 of FIG. 4 and the feature amount extracted in step SC302 is calculated, and the process proceeds to step SC304. In step SC304, it is determined whether or not the difference between the feature amounts calculated in step SC303 is smaller than the feature value difference value currently held. If smaller, the procedure proceeds to step SC305. Otherwise, the procedure proceeds to step SC307. As described above, the difference between the feature amount of the subject to be tracked and the feature amount of the pattern matching target image is calculated, and the smaller the difference is, the more the matching is achieved.

  Therefore, in step SC304, it is determined whether or not the difference between the feature amounts is the smallest so far, and the minimum value is held by a process described later. If the feature amount difference has not been held at step SC305, the process always proceeds to step SC305. In step SC305, the calculated minimum value of the feature amount difference is held, and the process proceeds to step SC306.

  In step SC306, the center position coordinates of the region from which the feature amount is extracted in step SC302 are held, and the process proceeds to step SC307. This center position information is the position information that is most matched to the image signal that is currently subjected to pattern matching, and holds the position information that the tracking subject is supposed to be in the end. This position information is used to set a pattern matching range in the process of SC301 in the next pattern matching process. Alternatively, if the selected image that has undergone pattern matching is an image that is currently captured by the camera, it is used to set the tracking region. In the present embodiment, the center position coordinates are used, but of course, the center position coordinates may be used, for example, the upper left coordinates of the region.

  Next, in step SC307, it is determined whether or not the minimum value of the feature amount difference held in step SC305 is larger than a predetermined value and the pattern matching time is over. If the minimum value of the difference between the feature quantities held is smaller than the predetermined value or the time has not exceeded, the process proceeds to step SC308. If the minimum value is larger than the predetermined value and the time has elapsed, the process proceeds to step SC310. Even if pattern matching takes time, if a pattern with a small feature amount difference does not occur, it can be said that there is a high possibility that the subject to be tracked is not already in the matching range. Therefore, a time limit is provided for pattern matching, and when the minimum value of the feature amount difference is larger than a predetermined value and exceeds the time limit, the matching is terminated. This time limit may be varied according to the minimum value of the difference between the feature quantities that are held, and may be terminated earlier as the time is larger.

  Next, in step SC308, it is determined whether or not the feature amount extraction performed in step SC302 has been performed on the entire matching range. If so, the process proceeds to step SC309. If not, the process returns to step SC302, and the pattern matching process is continued for the currently selected image signal in steps SC302 to SC307. In step SC309, the pattern matching is ended, and it is determined whether or not the minimum value of the difference between the feature values finally held is smaller than a predetermined value. If it is smaller than the predetermined value, the process is ended. Proceed to SC310. If the minimum value of the feature amount difference is smaller than a predetermined value, it is determined that a tracking subject has been found, and the tracking start process in FIG. 4 is continued.

  Further, it is determined that the tracking subject cannot be specified in step SC310 that is advanced when the minimum value of the feature amount difference is larger than the predetermined value or when the matching time is over, and the processing is ended. When the follow-up impossible information is issued, the follow-up start process is interrupted at step SC205 in FIG. 4, and at the same time, the fact that the follow-up subject cannot be specified is transmitted to the remote control terminal 20 at step SC107 in FIG.

As described above, pattern matching is performed on the selected past image, and position information indicating that the tracking subject is present is acquired. By sequentially performing pattern matching on the past image in time series order, position information where the tracking subject is present in the currently captured image is acquired. Next, the tracking continuation processing in step SC109 in FIG. This will be described with reference to a flowchart.

  In step SC401, a feature amount is extracted from the follow-up image held in step SC208 in the follow-up start process of FIG. 4, and the process proceeds to step SC402. In Step SC402, similarly, the center position of the range to be matched is limited from Step SC208 in the follow-up start process of FIG. 4 or the position information held in Step SC412 described later, and the process proceeds to Step SC403. By setting the matching range around the position information that was matched last time, it becomes easier to identify the tracking subject.

  In step SC403, an image having the same size as the follow-up image used in step SC401 is cut out from the image signal in the matching range set in step SC402, and a feature amount is extracted, and the process proceeds to step SC404. In step SC404, the difference between the feature amounts extracted in steps SC401 and SC403 is calculated, and the process proceeds to step SC405. In step SC405, it is determined whether or not the difference between the feature amounts calculated so far is smaller than the value of the feature amount difference currently held. If smaller, the procedure proceeds to step SC406. Otherwise, the procedure proceeds to step SC408. In step SC406, it is determined that the difference between the calculated feature values is the smallest, and this value is held, and the process proceeds to step SC407. In step SC407, the position information of the image signal cut out in step SC403 is held, and the process proceeds to step SC408.

  Note that if the minimum value of the feature amount difference has not yet been held in step SC406 at the time of determination in step SC405, the process always proceeds to step SC406. In step SC408, it is determined whether or not the minimum value of the feature amount difference held in step SC406 is greater than a predetermined value and the pattern matching time is over. If the minimum value of the difference between the feature quantities held is smaller than the predetermined value or the time has not exceeded, the process proceeds to step SC409. If the minimum value is larger than the predetermined value and the time has elapsed, the process proceeds to step SC414. Details of the process in step SC408 are the same as those described in step SC307 in FIG.

  In step SC409, it is determined whether or not matching has been performed for all the matching ranges set in step SC402. If so, the process proceeds to step SC410. If not, the process returns to step SC403 to continue the pattern matching process. To do. In step SC410, it is determined whether or not the minimum value of the feature amount difference held in step SC406 is smaller than a predetermined value as a result of pattern matching. If smaller than the predetermined value, the process proceeds to step SC411. Proceed to In step SC411, assuming that the tracking subject has been identified, the tracking area is set in a range centered on the position information of the image signal held in step SC407, and the flow proceeds to step SC412.

  In step SC412, the position information of the tracking area set in step SC411 is held, and the process proceeds to step SC413. The coordinate information held in step SC412 is used in step SC402 in the next follow-up continuation process. In step SC413, the follow-up impossible counter is cleared and the process is terminated. The follow-up impossibility counter will be described in steps SC414 to SC417 described later.

  In this way, by repeatedly performing pattern matching on sequentially captured image signals, the tracking area is sequentially identified and tracking is continued. Next, in step SC414 which proceeds when the minimum value of the feature amount difference is larger than the predetermined value or when the matching time is exceeded, the follow-up impossible counter is added and the process proceeds to step SC415. The follow-up impossible counter is used to count the number of times that the follow-up subject cannot be identified continuously when the follow-up process is performed. In step SC415, it is determined whether or not the follow-up impossible counter is greater than or equal to a predetermined number. If the counter is greater than or equal to the predetermined number, the process proceeds to step SC416, and if not, the process proceeds to step SC417.

  In step SC416, it is determined that the tracking subject cannot be specified, and tracking impossible information is issued and the process is terminated. When follow-up disable information is issued, follow-up stop processing is performed at step SC111 in FIG. In step SC417, it changes to the state of follow-up control temporary stop, and a process is complete | finished. The follow-up control pause state is a state in which camera control is not performed on the follow-up control area, and AF and AE controls are temporarily stopped.

  Here, step SC414 to step SC417 will be described in detail. During the follow-up continuation process, even if the follow-up subject cannot be temporarily specified, there is a possibility that it can be specified again. For example, there may be a case where the subject that has been tracked once frames out and then enters the frame again, or a case where another subject crosses before the subject that has been following. Assuming such a case, even if the subject cannot be tracked temporarily, a waiting time for specifying the subject again is set. This waiting time is a follow-up impossible counter, and the follow-up impossible information is not issued in step SC416 until the follow-up impossible counter reaches a predetermined number of times or more.

  However, if the tracking subject has not been specified, AF and AE control cannot be performed on the currently set tracking area, so these controls are temporarily stopped in step SC417. In this way, even if it becomes impossible to specify the tracking subject temporarily, the tracking can be continued when the tracking subject appears again.

  As described above, the camera 10 transmits not only the captured image signal but also the transmission time information of the image signal to the remote operation terminal 20, and the remote operation terminal 20 displays the transmission time every time the image signal is displayed. The information is sent back to the camera 10 as it is. The camera 10 always measures the communication delay time from the difference between the time information when the transmission time information is received from the remote control terminal 20 and the received transmission time information, and at least the communication delay time is stored in the SDRAM 166. The captured image signal is held. When performing the follow-up control by the follow-up instruction from the remote operation terminal 20, the system control unit 14 receives the coordinate information on which the follow-up instruction is generated generated by the remote-side system control unit 21. Based on the communication delay time when the coordinate information is received, the system control unit 14 displays on the remote display unit 243 when a follow-up instruction is given from the past image held in the SDRAM 166 by the remote operation terminal 20. Select the image that was being used.

  Further, the follow-up image generation unit 141 generates a predetermined area centered on the coordinate information received from the selected image as an image of the follow-up subject (follow-up image). The system control unit 14 performs the tracking control by specifying the tracking subject from the image signal currently captured by the tracking signal processing unit 125 and the subject area detection unit 126 based on the image of the tracking subject. In addition, in order to identify the tracking subject from the image signal currently captured, pattern matching is performed between the past image and the tracking subject image that have been retained sequentially from the past image signal that generated the tracking subject image. I do.

  As described above, the follow-up subject is specified in consideration of the communication delay time, instead of instructing the follow-up from the remote operation terminal 20 only by the position information of the follow-up subject. This improves the problem that the user mistracks the subject other than the subject that the user wants to follow due to the communication delay, and the user must give a follow instruction to the position considering the communication time lag. To do. As a result, the subject that the user wants to follow can be followed more accurately regardless of the communication delay, and the user can issue a follow-up instruction without worrying about the communication delay.

Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included.
(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is supplied to the imaging device and the remote operation terminal. And the computer (or CPU, MPU, etc.) of the imaging device and the remote control terminal reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Moreover, the function of the above-described embodiment is realized by making the program code read by the computer executable. Further, based on the instruction of the program code, an OS (operating system) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments may be realized by the processing. included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  The present invention can be suitably used for a camera system including an imaging device such as a compact digital camera, a single-lens reflex camera, and a video camera and a remote operation terminal such as a mobile phone, a smartphone, and a portable game machine.

DESCRIPTION OF SYMBOLS 10 ... Imaging device 121 ... Imaging element 125 ... Tracking signal processing part 126 ... Subject area detection part 14 ... System control part 141 ... Tracking image generation part 166 ... SDRAM
169 ... Imaging-side communication unit 20 ... Remote operation terminal 21 ... Remote-side system control unit 22 ... Remote-side operation unit 241 ... Remote-side communication unit 243 ... Remote-side display unit 244 ... Remote-side SDRAM

Claims (13)

An imaging device controlled from the outside by a remote operation terminal,
Imaging means for photoelectrically converting a subject image via an optical system to generate an image signal;
When the generated image signal and the grant information given when transmitting the image signal to the remote control terminal are transmitted, and the subject is specified on the image displayed on the display unit of the remote control terminal Receiving the position information specifying the subject from the remote operation terminal and providing the image signal from the remote operation terminal when an image corresponding to the image signal is displayed on the display unit A communication means for receiving information;
Calculating means for calculating a communication delay time based on the received grant information;
Holding means for holding an image signal generated by the imaging means based on the calculated communication delay time for a predetermined period;
Selection means for selecting an image signal held in the holding means based on the communication delay time calculated when the position information is received;
Identification means for identifying a first tracking area of the subject from the current image signal based on the selected image signal and the received position information;
An imaging device comprising:   The imaging apparatus according to claim 1, wherein the assignment information is information relating to a transmission time at which the image signal is transmitted to the remote operation terminal.   The imaging apparatus according to claim 1, wherein the holding unit holds an image signal corresponding to at least a communication delay time.   The selection unit selects an image signal that is closest to a past time point by a communication delay time calculated when the position information is received, from among the image signals held by the holding unit. Item 4. The imaging device according to any one of Items 1 to 3.   The specifying unit specifies the second tracking region of the subject sequentially from the selected image signal in time series among the image signals held by the holding unit, so that the current image The imaging apparatus according to claim 1, wherein the first tracking area is specified from a signal. A remote control terminal for controlling the imaging device from the outside,
A display unit for displaying an image corresponding to the image signal received from the imaging device;
Generating means for generating position information related to a position where the subject is designated when a subject is designated on the image being displayed on the display unit;
In addition to receiving the image signal and the attached information added to the image signal from the imaging device, and when the subject is specified on the image being displayed on the display unit, the position information is transmitted to the imaging device A remote operation terminal comprising: a communication unit configured to transmit the addition information added to the image signal to the imaging device when an image corresponding to the image signal is displayed on the display unit. The attached information is information related to a transmission time at which the image signal is transmitted from the imaging device,
The communication means receives the image signal to which information on the transmission time is given from the imaging device,
When the display unit displays an image corresponding to the image signal, the remote control terminal transmits information about the transmission time given to the image signal from the communication unit to the imaging device. The remote control terminal according to claim 6, wherein: A camera system comprising an imaging device and a remote operation terminal that controls the imaging device from the outside,
An image pickup means provided in the image pickup apparatus for photoelectrically converting a subject image via an optical system to generate an image signal;
A first communication unit that is provided in the imaging apparatus and that transmits the generated image signal and grant information that is given when the image signal is transmitted to the remote operation terminal;
A second communication means provided in the remote control terminal for receiving the image signal and the grant information;
A display unit provided in the remote operation terminal and displaying an image corresponding to the image signal received from the imaging device;
A generating unit that is provided in the remote operation terminal and generates position information about a position where the subject is designated when a subject is designated on an image being displayed on the display unit;
A calculating unit that is provided in the imaging device and calculates a communication delay time based on the provision information transmitted from the second communication unit when the image is displayed on the display unit of the remote operation terminal;
A holding unit that is provided in the imaging apparatus and holds an image signal generated by the imaging unit based on the calculated communication delay time;
Calculated when the subject position information transmitted from the second communication means is received when the subject is specified on the image displayed on the display unit of the remote control terminal provided in the imaging device Selecting means for selecting an image signal held in the holding means based on the communication delay time,
A specifying unit that is provided in the imaging device and specifies a first tracking region of the subject from a current image signal based on the selected image signal and the received position information;
A camera system comprising: The grant information is information related to a transmission time at which the image signal is transmitted from the imaging device to the remote operation terminal;
When the image pickup device transmits the image signal by the first communication unit, the image pickup device transmits the information related to the transmission time,
When the display unit displays an image corresponding to the image signal, the remote control terminal transmits information on the transmission time given to the image signal from the second communication unit to the imaging device. Return,
The calculating means includes
9. The camera system according to claim 8, wherein a communication delay time is calculated from a difference between a time at which the imaging apparatus receives information on the transmission time from the second communication unit and the transmission time. A control method for an image pickup apparatus having an image pickup means for photoelectrically converting a subject image via an optical system to generate an image signal and controlled from the outside by a remote operation terminal,
Transmitting the generated image signal and grant information given when transmitting the image signal to the remote control terminal;
Receiving a position information specifying the subject from the remote operation terminal when a subject is designated on the image being displayed on the display unit of the remote operation terminal;
When the image corresponding to the image signal is displayed on the display unit, receiving the attachment information attached to the image signal from the remote operation terminal;
Calculating a communication delay time based on the received grant information;
Holding the image signal generated by the imaging unit based on the calculated communication delay time in a holding unit for a predetermined period;
Selecting an image signal held in the holding means based on the communication delay time calculated when the position information is received;
Identifying a first tracking region of the subject from a current image signal based on the selected image signal and the received position information;
A method for controlling an imaging apparatus, comprising: An imaging device that has an imaging unit that photoelectrically converts a subject image via an optical system to generate an image signal and is controlled from the outside by a remote operation terminal,
Transmitting the generated image signal and grant information given when transmitting the image signal to the remote control terminal;
Receiving a position information specifying the subject from the remote operation terminal when a subject is designated on the image being displayed on the display unit of the remote operation terminal;
When the image corresponding to the image signal is displayed on the display unit, receiving the attachment information attached to the image signal from the remote operation terminal;
Calculating a communication delay time based on the received grant information;
Holding the image signal generated by the imaging unit based on the calculated communication delay time in a holding unit for a predetermined period;
Selecting an image signal held in the holding means based on the communication delay time calculated when the position information is received;
Identifying a first tracking region of the subject from a current image signal based on the selected image signal and the received position information;
A control program for an imaging apparatus, characterized in that A control method of a remote control terminal for controlling an imaging device from the outside,
Receiving the image signal generated by the imaging device and the attachment information attached to the image signal;
Displaying an image corresponding to the image signal received from the imaging device on a display unit;
When a subject is designated on the image being displayed on the display unit, generating position information relating to the designated position of the subject;
A step of transmitting the position information to the imaging device when a subject is designated on an image being displayed on the display unit;
A method of controlling the remote operation terminal, comprising: transmitting the attachment information added to the image signal to the imaging device when an image corresponding to the image signal is displayed on the display unit. . To the remote control terminal that controls the imaging device from the outside,
Receiving the image signal generated by the imaging device and the attachment information attached to the image signal;
Displaying an image corresponding to the image signal received from the imaging device on a display unit;
When a subject is designated on the image being displayed on the display unit, generating position information relating to the designated position of the subject;
A step of transmitting the position information to the imaging device when a subject is designated on an image being displayed on the display unit;
And a step of transmitting to the imaging device the attached information given to the image signal when an image corresponding to the image signal is displayed on the display unit. program.