JP2012099975A - Video tracking apparatus, video tracking method and video tracking program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video tracking apparatus, a video tracking method and a video tracking program that can efficiently produce an accurate tracking result in tracking an object in video.SOLUTION: In an image processing system 100, one of a plurality of frame images stored in a storage section 40 is displayed as an initial image on a display section 20. When an operation section 4 is used to set a tracking condition in the initial image and command a start of tracking, a tracking operation is executed in the initial image and subsequent frame images. When the operation section 4 is used after a stop or end of the tracking operation to instruct any one of the plurality of frame images to be displayed as a new initial image, the new initial image is displayed on the display section 20. When the operation section 4 is further used to set a tracking condition again in the new initial image and command a start of tracking, a tracking operation is executed in the new initial image and subsequent frame images.

Description

  The present invention relates to a moving image tracking device, a moving image tracking method, and a moving image tracking program for tracking a tracking target in a moving image.

  Various methods have been proposed for displaying a track of a tracking target that moves in a moving image by tracking the tracking target in the moving image.

  For example, according to the moving image data analysis program described in Patent Document 1, when an operator designates an arbitrary portion of an image displayed on a monitor, the locus display means displays the locus of the designated portion on the monitor. Can do.

JP 2010-71687 A

  A moving image is composed of a plurality of frames of images. In tracking of a tracking target in a moving image, the position of the tracking target is identified in each of a plurality of frames constituting the moving image. The probability of successful identification of the tracking target position in each frame image is relatively high. However, the probability that the tracking target position is successfully identified in all the images of a plurality of frames constituting the moving image decreases exponentially.

  Therefore, in order to obtain an accurate trajectory as an accurate tracking result of the tracking target, it is necessary to repeatedly perform tracking of the tracking target until the tracking target is successfully tracked for all the frames of the moving image. As a result, it takes a lot of time to obtain an accurate tracking result, so that the work efficiency is lowered.

  An object of the present invention is to provide a moving image tracking apparatus, a moving image tracking method, and a moving image tracking program capable of obtaining an efficient and accurate tracking result in tracking of a tracking target in a moving image.

  (1) A moving image tracking apparatus according to a first aspect is a moving image tracking apparatus that tracks a tracking target in a moving image composed of a plurality of frames of images, and a plurality of frames of images for displaying a plurality of frames of images. A storage unit for storing data, a display unit for displaying an image based on the image data of each frame stored in the storage unit, setting of tracking conditions for the tracking target in the image displayed on the display unit, and a tracking target The operation unit operated by the user in order to instruct the tracking of the image and the display unit display images of a plurality of frames in order, and in each displayed image, the position of the tracking target is identified by the tracking condition set by the operation unit And a control unit that executes a tracking operation for displaying a symbol at the identified position, and the control unit displays one of the images of the plurality of frames as an initial image. When the tracking condition is set in the initial image by the operation unit and the start of tracking is instructed, the tracking operation is performed on the image of the frame after the initial image, and after the tracking operation is stopped or ended, the operation unit When an instruction to display any one of the images of a plurality of frames as a new initial image is displayed on the display unit, the tracking condition is set in the new initial image by the operation unit. When the resetting is instructed and the start of tracking is instructed, the tracking operation is executed for images of frames after the new initial image.

  In this moving image tracking apparatus, one image of a plurality of frames is displayed on the display unit as an initial image based on the image data stored in the storage unit. When the tracking condition is set in the initial image by the operation unit and the start of tracking is instructed, the tracking operation is executed for the images of the frames after the initial image.

  In the tracking operation, images of a plurality of frames are sequentially displayed on the display unit. In each displayed image, the position of the tracking target is identified under the tracking condition set by the operation unit, and a symbol is displayed at the identified position. A plurality of symbols displayed for a plurality of frames of images indicate a tracking target trajectory that moves in a moving image.

  When the operation unit instructs to display any one of the images of a plurality of frames as a new initial image after the tracking operation is stopped or ended, the new initial image is displayed on the display unit. Further, when the tracking condition is reset in the new initial image and the start of tracking is instructed by the operation unit, the tracking operation is executed for the images of the frames after the new initial image.

  Thereby, when the tracking of the tracking target fails during the execution of the tracking operation, the tracking operation can be restarted from the initial image of an arbitrary frame. Therefore, the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed is set as a new initial image, so that the tracking result of the image that has been successfully tracked is maintained and the image in which tracking has failed. The tracking operation can be resumed.

  Further, since the tracking condition can be reset in the new initial image, the probability that the tracking for each image is successful in the resumed tracking operation is increased. Therefore, the entire execution time of the tracking operation is shortened. As a result, an accurate and accurate tracking result can be obtained.

  (2) The control unit sets the tracking target position identified in the image currently displayed on the display unit and the tracking target position identified in the image of the frame before the image currently displayed on the display unit. Each symbol is displayed, and when the tracking condition is reset in the new initial image and the start of tracking is instructed, the position of the tracking target identified in the image currently displayed on the display unit and the new initial image The position of the tracking target identified in the image of the frame after the image and before the image currently displayed on the display unit, and the position of the tracking target identified in the image of the frame before the new initial image The symbols may be displayed respectively.

  In this case, symbols are respectively displayed at the position of the tracking target identified in the image currently displayed on the display unit and the position of the tracking target identified in the image of the frame before the image currently displayed on the display unit. Is done. Thereby, the user can easily observe the track of the tracking target in the moving image.

  Further, when the tracking operation is resumed, the position of the tracking target identified in the image currently displayed on the display unit and the frame after the new initial image and before the image currently displayed on the display unit Symbols are respectively displayed at the position of the tracking target identified in the previous image and the position of the tracking target identified in the image of the frame before the new initial image. Thereby, even when the tracking operation is resumed, the user can easily observe the track of the tracking target in the moving image.

  (3) The tracking condition may include at least one of a range of the image portion to be tracked in the initial image and a range for searching for the tracking target in the image of each frame.

  In this case, the user restarts the tracking operation under an appropriate condition by resetting at least one of the range of the image portion to be tracked in the initial image and the range to search for the tracking target in the image of each frame. can do. Thereby, even when tracking fails, it is possible to prevent the tracking failure in the previous tracking operation from being repeated in the tracking operation at the time of resumption.

  (4) The control unit has a reverse playback function for displaying images of a plurality of frames on the display unit in a time sequence opposite to that in the tracking operation, and the operation unit performs reverse playback after the tracking operation is stopped or ended. It may be possible to instruct the display of a new initial image by instructing.

  In this case, the user can display an image of a desired frame as a new initial image by instructing reverse reproduction after the tracking operation is stopped or ended. As a result, the user can easily find the image of the frame in which tracking has failed first while viewing the image and symbol of each frame. Therefore, the tracking operation can be easily restarted from the image of the first frame in which the tracking has failed or the image of the frame immediately before the tracking has failed.

  (5) The control unit displays a selection unit for selecting an image to be displayed on the display unit on the display unit, and the operation unit displays a new initial image by operating the selection unit displayed on the display unit. May be instructed.

  In this case, the user can display an image of a desired frame as a new initial image by operating the selection unit after the tracking operation is stopped or ended. Therefore, the user can quickly display the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed as an initial image. Accordingly, the tracking operation can be quickly restarted from the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed.

  (6) The control unit calculates the accuracy of the identification result of the tracking target position for each image during the tracking operation, and stops the tracking operation when the accuracy falls below a predetermined threshold value. May be.

  In this case, the tracking operation is automatically stopped when the accuracy falls below the threshold when the tracking fails. Thereby, the tracking operation can be easily restarted from the image of the first frame in which the tracking has failed or the image of the frame immediately before the tracking has failed.

  (7) A moving image tracking method according to the second invention is a moving image tracking method for tracking a tracking target in a moving image composed of a plurality of frames of images, and a plurality of frames of images for displaying a plurality of frames of images. A step of storing data, a step of displaying one of the images of the plurality of frames on the display unit as an initial image based on the stored image data of the plurality of frames, a tracking condition is set in the initial image and the tracking is performed When the start of the image is instructed, the images of the frames after the initial image are sequentially displayed on the display unit based on the stored image data of the plurality of frames, and the tracking target is set under the tracking condition set in each displayed image. Performing a first tracking operation for identifying a position and displaying a symbol at the identified position; and after stopping or ending the first tracking operation, A step of causing the display unit to display a new initial image based on the stored image data of a plurality of frames when an instruction is given to display any one of several frames of images as a new initial image; When the tracking condition is reset in the new initial image and the start of tracking is instructed, the images of the frames after the new initial image are displayed in order on the display unit based on the stored multiple frames of image data. And a step of performing a second tracking operation for identifying the position of the tracking target under the tracking condition set in each displayed image and displaying a symbol at the identified position.

  In this moving image tracking method, a plurality of frames of image data for displaying a plurality of frames of images are stored. Further, one of the images of the plurality of frames is displayed on the display unit as an initial image based on the stored image data of the plurality of frames. Here, when the tracking condition is set in the initial image and the start of tracking is instructed, as the first tracking operation, the image of the frame after the initial image is displayed on the display unit based on the stored plural frames of image data. The position of the tracking target is identified by the tracking condition set in each displayed image, and a symbol is displayed at the identified position. A plurality of symbols displayed for a plurality of frames of images indicate a tracking target trajectory that moves in a moving image.

  When it is instructed to display any one of the images of a plurality of frames as a new initial image after the stop or end of the first tracking operation, a new initial is generated based on the stored image data of the plurality of frames. An image is displayed on the display unit. Further, when the tracking condition is reset in the new initial image and the start of tracking is instructed, as a second tracking operation, the frames of the frames after the new initial image are based on the stored plural frames of image data. Images are displayed in order on the display unit, and the position of the tracking target is identified under the tracking condition set in each displayed image, and a symbol is displayed at the identified position.

  Thereby, when the tracking of the tracking target fails during the execution of the first tracking operation, the second tracking operation can be executed from the initial image of an arbitrary frame. Therefore, the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed is set as a new initial image, so that the tracking result of the image that has been successfully tracked is maintained and the image in which tracking has failed. A second tracking operation can be performed.

  Further, since the tracking condition can be reset in the new initial image, the probability that the tracking for each image is successful in the second tracking operation is high. Therefore, the overall execution time of the first and second tracking operations is shortened. As a result, an accurate and accurate tracking result can be obtained.

  (8) A moving image tracking program according to a third aspect of the present invention is a moving image tracking program for causing a processing device to execute a tracking process for tracking a tracking target in a moving image composed of a plurality of frames of images, and displays a plurality of frames of images. A process of storing image data of a plurality of frames for performing, a process of displaying one of the images of the plurality of frames on the display unit as an initial image based on the stored image data of the plurality of frames, When the tracking condition is set and the start of tracking is instructed, the images of the frames after the initial image are sequentially displayed on the display unit based on the stored image data of the plurality of frames, and set in each displayed image. The first tracking operation for identifying the position of the tracking target under the tracking condition and displaying the symbol at the identified position; When it is instructed to display one of the images of multiple frames as a new initial image after the operation stops or ends, a new initial image is displayed based on the stored multiple frames of image data. When the tracking condition is reset in the new initial image and the start of tracking is instructed, the images of the frames after the new initial image based on the stored multiple frames of image data Are sequentially displayed on the display unit, and the second tracking operation for identifying the position of the tracking target under the tracking condition set in each displayed image and displaying the symbol at the identified position is performed. It is what is executed by the device.

  In this moving image tracking program, a plurality of frames of image data for displaying a plurality of frames of images are stored. Further, one of the images of the plurality of frames is displayed on the display unit as an initial image based on the stored image data of the plurality of frames. Here, when the tracking condition is set in the initial image and the start of tracking is instructed, as the first tracking operation, the image of the frame after the initial image is displayed on the display unit based on the stored plural frames of image data. The position of the tracking target is identified by the tracking condition set in each displayed image, and a symbol is displayed at the identified position. A plurality of symbols displayed for a plurality of frames of images indicate a tracking target trajectory that moves in a moving image.

  When it is instructed to display any one of the images of a plurality of frames as a new initial image after the stop or end of the first tracking operation, a new initial is generated based on the stored image data of the plurality of frames. An image is displayed on the display unit. Further, when the tracking condition is reset in the new initial image and the start of tracking is instructed, as a second tracking operation, the frames of the frames after the new initial image are based on the stored plural frames of image data. Images are displayed in order on the display unit, and the position of the tracking target is identified under the tracking condition set in each displayed image, and a symbol is displayed at the identified position.

  Thereby, when the tracking of the tracking target fails during the execution of the first tracking operation, the second tracking operation can be executed from the initial image of an arbitrary frame. Therefore, the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed is set as a new initial image, so that the tracking result of the image that has been successfully tracked is maintained and the image in which tracking has failed. A second tracking operation can be performed.

  Further, since the tracking condition can be reset in the new initial image, the probability that the tracking for each image is successful in the second tracking operation is high. Therefore, the overall execution time of the first and second tracking operations is shortened. As a result, an accurate and accurate tracking result can be obtained.

  According to the present invention, it is possible to obtain an accurate and accurate tracking result in tracking of a tracking target in a moving image.

It is a block diagram which shows the structure of an image processing system provided with the moving image tracking apparatus which concerns on one embodiment of this invention. It is a figure which shows the example of a display of the display part at the time of setting of imaging conditions. It is a schematic diagram which shows the memory | storage state of the video recording part of FIG. 1 during video recording processing. It is a schematic diagram which shows the other example of the memory | storage state of the video recording part of FIG. 1 during video recording processing. It is a figure which shows the example of a display of the display part during a video recording process. It is a figure which shows the example of a display of the display part at the time of completion | finish of a video recording process. It is a schematic diagram of the moving image file memorize | stored in a memory | storage part by file memory | storage process. It is a figure which shows the example of a display of the display part at the time of a reproduction | regeneration process. It is a figure which shows the example of a display of a display part at the time of the execution start of a tracking process. It is a figure which shows an example of the moving image displayed on an image display area. It is a figure which shows the example of a display of the display part at the time of tracking operation | movement of tracking object. It is a figure which shows an example of the moving image displayed on the image display area at the time of tracking operation | movement of tracking object. It is a figure which shows the example of a display of a display part at the time of tracking operation | movement completion | finish of a tracking object being completed. It is a figure which shows an example of the moving image displayed on the image display area when the tracking of tracking object fails. It is a flowchart which shows the tracking process of the tracking target in the moving image in a moving image tracking device. It is a flowchart which shows the tracking process of the tracking target in the moving image in a moving image tracking device. It is a flowchart which shows the tracking process of the tracking target in the moving image in a moving image tracking device. It is a figure which shows the transition of the image displayed on an image display area during execution of the tracking process of a tracking object. It is a figure which shows the transition of the image displayed on an image display area during execution of the tracking process of a tracking object. It is a figure which shows the transition of the image displayed on an image display area during execution of the tracking process of a tracking object. It is a figure which shows the data memorize | stored in RAM of a control part during tracking process execution. It is a flowchart which shows the tracking process of the tracking object in the moving image in the moving image tracking apparatus which concerns on other embodiment.

  Hereinafter, a moving image tracking apparatus, a moving image tracking method, and a moving image tracking program according to an embodiment of the present invention will be described with reference to the drawings. As will be described later, the image processing system including the moving image tracking apparatus according to the present embodiment includes a display unit. A moving image is displayed by continuously displaying images of a plurality of frames on the display unit.

(1) Basic Configuration of Image Processing System FIG. 1 is a block diagram showing a configuration of an image processing system 100 including a moving image tracking apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image processing system 100 includes a main body unit 1, a camera 2, a lighting device 3, and an operation unit 4. The main unit 1 includes a control unit 10, a display unit 20, a recording unit 30, a storage unit 40, a video memory 50, a waveform collection unit 60, a bus BS, and a plurality of interfaces (not shown). The control unit 10, the display unit 20, the recording unit 30, the storage unit 40, the video memory 50, and the waveform collection unit 60 are connected to the bus BS. The camera 2, the lighting device 3, and the operation unit 4 are each connected to the bus BS via an interface.

  A PLC (programmable logic controller) 5, a sensor 6 and a waveform collection device 7 can be connected to the main body 1. A sensor 8 can be connected to the waveform collection device 7. Details will be described later.

  The camera 2 includes an image sensor such as a CCD (charge coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor. The exposure time (shutter speed), the shooting frame rate, and the like of the image sensor can be set by the user using the operation unit 4. The shooting frame rate is the number of frames of an image acquired per unit time. Note that the camera 2 of the present embodiment can shoot the subject W at a shooting frame rate of 15 fps (frame per second) or more and 32000 fps or less.

  When the subject W is photographed by the camera 2, an image of the subject W is acquired by the imaging device at the set shooting frame rate, and the acquired image is output as image data. As a result, a plurality of frames of image data corresponding to a plurality of frames of images are provided from the camera 2 to the main body 1. A moving image is constituted by images of a plurality of frames. Hereinafter, image data of a plurality of frames is referred to as moving image data.

  The camera 2 is attached to a support base (not shown) such as a tripod stand together with the illumination device 3. The illumination device 3 includes a light source such as a metal halide lamp or a high power LED (light emitting diode). The illumination device 3 is used for irradiating the subject W with light. When the subject W is sufficiently bright, the lighting device 3 may not be provided.

  The video memory 50 includes a memory such as a RAM (Random Access Memory). A plurality of frames of image data are sequentially supplied from the camera 2, the recording unit 30, or the storage unit 40 to the video memory 50. The given image data of a plurality of frames is sequentially stored in the video memory 50.

  The display unit 20 is configured by, for example, a liquid crystal display panel or an organic EL (electroluminescence) panel. A plurality of frames of images are continuously displayed on the display unit 20 based on the plurality of frames of image data sequentially stored in the video memory 50. Thereby, the moving image is displayed on the display unit 20.

  The operation unit 4 is configured by a pointing device such as a mouse or a trackball. The operation unit 4 may be configured with a keyboard together with or instead of the pointing device, or may be configured with a remote controller or a touch panel. The operation unit 4 may be configured integrally with the main body unit 1.

  The recording unit 30 includes a memory such as a RAM. The recording unit 30 includes a moving image data storage area for storing moving image data composed of a plurality of image data and a waveform data storage area for storing waveform data composed of a plurality of evaluation values. Details of the waveform data composed of a plurality of evaluation values will be described later.

  Recording multiple frames of image data (moving image data) from the camera 2 in the moving image data storage area of the recording unit 30 is called recording. When the user operates the operation unit 4, a moving image of the subject W photographed by the camera 2 is recorded. In the present embodiment, a plurality of evaluation values (waveform data) given from the waveform collection unit 60 are simultaneously stored in the waveform data storage area of the recording unit 30 during the recording process.

  The storage unit 40 is composed of a mass storage device such as a hard disk. When the user operates the operation unit 4, a moving image file including moving image data and waveform data stored in the recording unit 30 is generated, and the generated moving image file is stored in the storage unit 40. The storage unit 40 stores a moving image tracking program according to the present embodiment. Details of the moving image tracking program will be described later.

  The waveform collection unit 60 collects waveform data composed of a plurality of evaluation values, and gives the collected waveform data to the recording unit 30. The evaluation value is, for example, an average luminance value of each frame image of the subject W photographed by the camera 2 (an average value of luminances of a plurality of pixels constituting each frame image). In this case, the waveform collection unit 60 sequentially calculates the average luminance value of the image of each frame as the evaluation value based on the image data of each frame sequentially given from the camera 2 to the recording unit 30. The plurality of average brightness values calculated sequentially are stored as waveform data in the waveform data storage area of the recording unit 30. In this case, a plurality of evaluation values are obtained in synchronization with a plurality of frames of image data. Accordingly, the number of evaluation values per unit time is equal to the number of image data of a plurality of frames per unit time. The evaluation value may be the sum of the luminance values of the images of each frame, or may be the difference between the sum of the luminance values of the images of each frame and the sum of the luminance values of the images of the subsequent frames. It may be a variance of luminance values.

  The waveform collection unit 60 may acquire the value of the output signal of the sensor 6 during the acquisition of the image data of a plurality of frames as the evaluation value. In this case, the plurality of evaluation values are obtained at the same period or different periods from the image data of a plurality of frames. Therefore, the number of evaluation values per unit time does not necessarily match the number of image data of a plurality of frames per unit time. When a plurality of evaluation values are obtained with a shorter cycle than image data of a plurality of frames, a plurality of evaluation values correspond to the image data of each frame. When a plurality of evaluation values are obtained with a period longer than that of a plurality of frames of image data, each evaluation value corresponds to a plurality of frames of image data.

  When the sensor 6 is a temperature sensor, a distance sensor, a pressure sensor, an acceleration sensor, a displacement sensor, or the like, a measured value of the measurement object such as temperature, distance, pressure, acceleration, or displacement is used as an evaluation value as a waveform collection unit. The plurality of acquired evaluation values are sequentially acquired by 60, and stored in the waveform data storage area of the recording unit 30 as waveform data.

  Further, when a microphone is used in place of the sensor 6, audio signals are sequentially acquired as evaluation values by the waveform collection unit 60, and the acquired plurality of audio signals are stored as waveform data in the waveform data storage area of the recording unit 30. Remembered.

  Thus, the plurality of evaluation values constituting the waveform data are values obtained during the acquisition of the image data of a plurality of frames.

  In FIG. 1, the waveform collection device 7 connected to the main body 1 has the same configuration as the waveform collection unit 60. Therefore, when the waveform collection device 7 is connected to the main body unit 1, the value of the output signal of the sensor 8 is sequentially acquired as an evaluation value by the waveform collection device 7, and the plurality of acquired evaluation values are stored in the recording unit 30. It is stored as waveform data in the waveform data storage area.

  The control unit 10 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM. The CPU of the control unit 10 controls the operation of other components in the camera 2, the lighting device 3, and the main body unit 1 based on the operation of the operation unit 4 by the user. In addition, the CPU of the control unit 10 executes the moving image tracking program stored in the storage unit 40 on the RAM. A system program is stored in the ROM.

(2) Basic Operation of Image Processing System The basic operation of the image processing system 100 in FIG. 1 will be described.

(2-1) Setting of shooting conditions The shooting conditions of the camera 2 are set when the user operates the operation unit 4. FIG. 2 is a diagram illustrating a display example of the display unit 20 when setting photographing conditions. As shown in FIG. 2, when setting the shooting conditions, an image display area Sa, a waveform display area Sb, a condition setting area Sc, and a recording button Se are displayed on the screen of the display unit 20.

  In the image display area Sa, an image based on the image data stored in the video memory 50 is displayed. A waveform graph based on the waveform data collected by the waveform collection unit 60 is displayed in the waveform display area Sb. The horizontal axis of the waveform graph represents time, and the vertical axis represents the evaluation value. For example, when the evaluation value is the average luminance value of the image displayed in the image display area Sa, a waveform graph is displayed in the waveform display area Sb with the average luminance value of the image as the vertical axis and the time as the horizontal axis. Is done.

  In the condition setting area Sc, basic shooting conditions of the camera 2 are displayed. As shooting conditions, for example, a plurality of input frames Sc1, Sc2, and Sc3 for setting a shutter speed, a shooting frame rate, and a resolution are displayed. The user inputs the shutter speed, shooting frame rate, and resolution values into the input frames Sc1, Sc2, and Sc3 using the operation unit 4, respectively. Based on the values input to the input frames Sc1, Sc2, and Sc3, the shutter speed, shooting frame rate, and resolution of the camera 2 are set by the control unit 10.

  In the present embodiment, for example, when the user operates the operation unit 4, a trigger signal serving as a reference for the end timing of a recording process described later is given to the control unit 10. In the condition setting area Sc, input frames Sc4 and Sc5 for trigger setting of the control unit 10 based on the trigger signal are further displayed.

  The user inputs values of a total recording time and a recording end time, which will be described later, into the input frames Sc4 and Sc5 using the operation unit 4, respectively. Based on the values respectively input to the input frames Sc4 and Sc5, the trigger setting is performed by the control unit 10. Details of the trigger setting will be described.

  In the present embodiment, the trigger setting determines a storage time (hereinafter referred to as total recording time) of image data stored in the recording unit 30 by one recording process, and a trigger signal is given to the control unit 10. A time (hereinafter referred to as “recording end time”) from the point in time until the end of the recording process is determined.

  For example, when the trigger is set with the total recording time set to 30 seconds and the recording end time set to 0 seconds, the recording process ends when a trigger signal is given to the control unit 10 after the recording is started. To do. Thereby, the moving image data and waveform data acquired from 30 seconds before the time when the trigger signal is given to the control unit 10 until the time when the trigger signal is given are stored in the recording unit 30.

  On the other hand, when the trigger setting is performed with the total recording time being 30 seconds and the recording end time being 30 seconds, 30 seconds have elapsed since the trigger signal was given to the control unit 10 after the recording was started. At this point, the recording process ends. Thereby, the video recording unit 30 stores the moving image data and waveform data acquired until 30 seconds have elapsed since the trigger signal was given to the control unit 10.

  In addition, when the trigger setting is performed with the total recording time set to 30 seconds and the recording end time set to 15 seconds, 15 seconds have passed since the trigger signal was given to the control unit 10 after the recording was started. At this point, the recording process ends. As a result, the video recording unit 30 stores the moving image data and waveform data acquired from 15 seconds before the time when the trigger signal is given to the control unit 10 until 30 seconds have passed.

  Instead of setting the total recording time as the trigger setting, the total number of frames of image data stored in the recording unit 30 by the recording process may be set. Further, instead of setting the recording end time as the trigger setting, the number of frames of image data stored in the recording unit 30 from the time when the trigger signal is given may be set.

  A trigger signal may be given from the PLC 5 to the control unit 10. In this case, for example, a trigger signal can be given to the control unit 10 in a cycle preset in the PLC 5 or in response to a specific operation timing of an external device connected to the PLC 5. Further, output signals of various sensors may be given to the control unit 10 as trigger signals.

  The recording button Se is operated by the user using the operation unit 4 to start a recording process described later.

(2-2) Recording Process FIG. 3 is a schematic diagram showing a storage state of the recording unit 30 in FIG. 1 during the recording process. As described above, the recording unit 30 includes the moving image data storage area 30A for storing moving image data (image data of a plurality of frames) and the waveform data storage area 30B for storing waveform data (a plurality of evaluation values). Is set.

  In the main body 1, as shown in FIG. 3A, the recording process is started by the control unit 10 when the recording button Se in FIG. 2 is operated. In the recording process, the image data v1 corresponding to the first frame image given from the camera 2 is stored in the moving image data storage area 30A of the recording unit 30. In addition, the first evaluation value i1 given from the waveform collecting unit 60 is stored in the waveform data storage area 30B of the recording unit 30.

  Subsequently, as shown in FIG. 3B, image data v2, v3, and v4 corresponding to the second, third, and fourth frame images given from the camera 2 are stored in the moving image data storage area 30A of the recording unit 30. The image data v1, v2, v3, and v4 are sequentially shifted in the moving image data storage area 30A while being sequentially stored. At the same time, the second, third, and fourth evaluation values i2, i3, i4 given from the waveform collection unit 60 are sequentially stored in the waveform data storage area 30B of the recording unit 30, and the evaluation values i1, i2 , I3, i4 are sequentially shifted in the waveform data storage area 30B.

  If the amount of image data stored in the moving image data storage area 30A during the recording process exceeds the storage capacity of the moving image data storage area 30A, the image data of the latest frame (current frame) is newly added to the moving image data storage area 30A. The image data of the oldest frame stored in the moving image data storage area 30A is deleted every time.

  Similarly, when the amount of evaluation value stored in the waveform data storage area 30B during the recording process exceeds the storage capacity of the waveform data storage area 30B, the latest evaluation value is newly stored in the waveform data storage area 30B. Every time, the past evaluation value stored in the waveform data storage area 30B is deleted.

  In the example of FIG. 3C, in a state where 12 pieces of image data v1 to v12 are stored in the moving image data storage area 30A, two pieces of image data v13 and v14 are sequentially stored. Are sequentially deleted. Further, in the state where 12 evaluation values i1 to i12 are stored in the waveform data storage area 30B, the evaluation values i13 and i14 of two images are sequentially stored, so that the past evaluation values i1 and i1 are stored. i2 is deleted sequentially. Thereby, the image data v3 to v14 are stored in the moving image data storage area 30A, and the evaluation values i3 to i14 are stored in the waveform data storage area 30B.

  As described above, in the moving image data storage area 30A and the waveform data storage area 30B during the recording process, the moving image data and the waveform data are sequentially stored in a so-called ring buffer format.

  In the example of FIG. 3C, the evaluation values i3 to i14 stored in the waveform data storage area 30B have a one-to-one correspondence with the image data v3 to v14 stored in the moving image data storage area 30A. As described above, when a plurality of evaluation values are obtained at a different period from the image data of a plurality of frames, the evaluation value stored in the waveform data storage area 30B is paired with the image data stored in the moving picture data storage area 30A. It does not correspond to one.

  FIG. 4 is a schematic diagram showing another example of the storage state of the recording unit 30 of FIG. 1 during the recording process. In the example of FIG. 4, a plurality of evaluation values are obtained with a shorter cycle than the image data of a plurality of frames. Therefore, the evaluation values i7 to i42 stored in the waveform data storage area 30B do not correspond one-to-one with the image data v3 to v14 stored in the moving image data storage area 30A. Three consecutive evaluation values correspond to one frame of image data. For example, the evaluation values i7, i8, i9 correspond to the image data v3, and the evaluation values i10, i11, i12 correspond to the image data v4.

  When a plurality of evaluation values are obtained with a longer cycle than that of image data of a plurality of frames, each evaluation value corresponds to image data of a plurality of frames.

  FIG. 5 is a diagram illustrating a display example of the display unit 20 during the recording process. As shown in FIG. 5, a trigger button Sf is displayed on the display unit 20 instead of the recording button Se of FIG.

  When the trigger button Sf is operated by the user using the operation unit 4, a trigger signal is given to the control unit 10. Thereby, as described above, the recording process ends at the timing based on the trigger setting.

(2-3) File Storage Process FIG. 6 is a diagram showing a display example of the display unit 20 at the end of the recording process. As shown in FIG. 6, at the end of recording, a file button Sj and a playback button Sg are displayed on the display unit 20 instead of the trigger button Sf of FIG.

  When the user operates the file button Sj using the operation unit 4, the control unit 10 performs file storage processing. In the file storage process, a moving image file including moving image data and waveform data stored in the recording unit 30 is generated, and the generated moving image file is stored in the storage unit 40 of FIG.

  FIG. 7 is a schematic diagram of a moving image file stored in the storage unit 40 by the file storage process.

  For example, the file storage process is performed in a state where the image data v1 to v12 are stored in the moving image data storage area 30A of the recording unit 30 in FIG. 3 and the evaluation values i1 to i12 are stored in the waveform data storage area 30B. In this case, as shown in FIG. 7, a moving image file FA including the image data v1 to v12, the evaluation values i1 to i12, and the file information fi is generated, and the generated moving image file FA is stored in the storage unit 40.

  The file information fi includes, for example, information indicating a file name and a data storage format. By repeating the above recording process and file storage process, a plurality of moving image files FA, FB, FC, and FD are stored in the storage unit 40 of FIG.

(2-4) Reproduction Process Next, the reproduction process will be described. The reproduction process is any of a plurality of frames of image data sequentially given from the camera 2, a plurality of frames of image data stored in the recording unit 30, or a plurality of frames of image data in a moving image file stored in the storage unit 40 It refers to displaying a moving image on the display unit 20 based on continuous image data of a plurality of frames.

  First, image reproduction processing based on image data in a moving image file stored in the storage unit 40 will be described.

  Icons corresponding to the plurality of moving image files stored in the storage unit 40 are displayed on the display unit 20 of FIG. In this case, the user selects one of the plurality of icons displayed on the display unit 20 using the operation unit 4, thereby selecting the selected movie file from the plurality of movie files stored in the storage unit 40. Movie data can be selected as a playback target.

  FIG. 8 is a diagram illustrating a display example of the display unit 20 during the reproduction process. As shown in FIG. 7, for example, when one moving image file is selected by the user using the operation unit 4, the display unit 20 in FIG. A display area Sb, a condition setting area Sc, and a playback button Sg are displayed, and a tracking button Sy is displayed instead of the file button Sj in FIG.

  When the user operates the playback button Sg using the operation unit 4, the control unit 10 starts the playback process. In the reproduction process, the control unit 10 causes the video memory 50 to sequentially store a plurality of frames of image data in the selected moving image file in the storage unit 40. Thereby, an image based on the image data sequentially stored in the video memory 50 is displayed in the image display area Sa. In this way, a moving image is reproduced based on the selected moving image file. Further, the control unit 10 stores the waveform data in the selected moving image file in the storage unit 40 in the video memory 50. Thereby, a waveform graph based on the waveform data is displayed in the waveform display area Sb.

  In this case, the control unit 10 currently has an evaluation value corresponding to the frame of the image currently displayed in the image display area Sa (hereinafter referred to as a corresponding evaluation value) on the waveform graph of the waveform display area Sb. The display position bar L is displayed. When the correspondence evaluation value changes due to a change in the frame of the image displayed in the image display area Sa, the control unit 10 moves the current display position bar L to the left and right on the time axis.

  Next, the reproduction process before the recording process will be described. Prior to the recording process, the control unit 10 sequentially stores image data of a plurality of frames provided from the camera 2 in the video memory 50. Thereby, an image based on the image data of a plurality of frames is displayed in the image display area Sa of FIG. In this way, the moving image of the subject photographed by the camera 2 is reproduced. Further, the control unit 10 causes the video memory 50 to store the waveform data given from the waveform collection unit 60. Thereby, a waveform graph based on the waveform data is displayed in the waveform display area Sb.

  Further, an image reproduction process based on image data stored in the recording unit 30 at the end of the recording process will be described. After the recording process, when the user operates the playback button Sg of FIG. 6 using the operation unit 4, the control unit 10 sequentially stores the image data of a plurality of frames in the recording unit 30 in the video memory 50. Thereby, an image based on the image data sequentially stored in the video memory 50 is displayed in the image display area Sa. In this manner, a moving image is reproduced based on the moving image data stored in the recording unit 30. Further, the control unit 10 stores the waveform data stored in the recording unit 30 in the video memory 50. Thereby, a waveform graph based on the waveform data is displayed in the waveform display area Sb.

(3) Video Tracking Device The video tracking device includes the operation unit 4, the control unit 10, the display unit 20, the storage unit 40, and the video memory 50 shown in FIG. The moving image tracking device tracks a designated tracking target in the moving image displayed on the display unit 20, and performs a tracking process for displaying the tracking result on the display unit 20. The storage unit 40 stores a moving image tracking program for performing tracking processing.

  When the user operates the tracking button Sy of FIG. 8 using the operation unit 4, a tracking mode for executing the tracking process is activated.

  FIG. 9 is a diagram illustrating a display example of the display unit 20 at the start of execution of the tracking process. As shown in FIG. 9, when the execution of the tracking process is started, an image display area Ta, a playback button Tc, a pause button Td, a reverse playback button Te, a seek bar Tf, and a tracking start button Th are mainly displayed on the screen of the display unit 20. Is displayed.

  In the image display area Ta, a moving image based on a moving image file selected by the user using the operation unit 4 is displayed.

  Further, a start point display cursor Tl and an end point display cursor Tm are displayed on the seek bar Tf. The start point display cursor Tl and the end point display cursor Tm are used for designating a range of images that can be displayed in the display area Ta among the images of a plurality of frames constituting the moving image. The positions of the start point display cursor Tl and the end point display cursor Tm correspond to the frames in the moving image file. In the image display area Ta, an image of a frame corresponding to the position of the end point display cursor Tm can be displayed as a moving image from an image of the frame corresponding to the position of the start point display cursor Tl.

  FIG. 10 is a diagram illustrating an example of a moving image displayed in the image display area Ta. Here, the moving image file selected by the user includes a plurality of frames of image data v1 to v20. In FIG. 10, the time axis extends from the left to the right of each stage and is connected from the right end of each stage to the left end of the lower stage. The time axis of FIG. 12 and FIG. 14 to be described later is the same as that of FIG.

  First, as shown in order from left to right in the first row of FIG. 10, a plurality of images V1 to V5 based on a plurality of frames of image data v1 to v5 are sequentially displayed in the image display area Ta. Next, as shown in order from left to right in the second row of FIG. 10, a plurality of images V6 to V10 based on a plurality of frames of image data v6 to v10 are sequentially displayed in the image display area Ta. Next, as shown in order from left to right in the third row in FIG. 10, a plurality of images V11 to V15 based on the image data v11 to v15 of a plurality of frames are sequentially displayed in the image display area Ta. Furthermore, as shown in order from left to right in the fourth row of FIG. 10, a plurality of images V16 to V20 based on a plurality of frames of image data v16 to v20 are displayed in order in the image display area Ta.

  Images V <b> 1 to V <b> 20 in FIG. 10 show a rotating cylindrical object M photographed by the camera 2 in FIG. 1. Two protrusions M1 and M2 are formed on one surface side of the object M photographed by the camera 2.

  When the user operates the playback button Tc of FIG. 9 using the operation unit 4 of FIG. 1, a plurality of frames of images V1 to V20 are displayed in the image display area Ta as shown in FIG. In this way, the moving image is reproduced.

  In addition, when the user operates the reverse playback button Te in FIG. 9, images of a plurality of frames are sequentially displayed in the image display area Ta in the reverse direction to the playback on the time axis from the currently displayed image (hereinafter, Called reverse playback).

  When the user operates the pause button Td during the reproduction or reverse reproduction of the moving image, the reproduction or reverse reproduction of the moving image is paused while the image is displayed in the image display area Ta. In addition, when the user continues to operate the playback button Tc, the moving image is played back at high speed until the operation is released. By continuing to operate the reverse playback button Te by the user, the moving image is reversely played back at high speed until the operation is released.

  The seek bar Tf has a slider Tg that can move in the horizontal direction. The slider Tg indicates the position on the time axis of the image currently displayed in the image display area Ta in the moving image. As the playback of the moving image proceeds, the slider Tg moves in one direction on the seek bar Tf (in this example, from the left to the right of the screen of the display unit 20). Similarly, the slider Tg moves in the other direction (in this example, from the right to the left of the screen of the display unit 20) on the seek bar Tf as the reverse playback of the moving image proceeds. The user can display an image of a desired frame in the moving image in the image display area Ta by moving the slider Tg of the seek bar Tf using the operation unit 4. Thereby, a moving image can be reproduced or reversely reproduced from desired images V1 to V20.

  When the user designates a tracking target in the image displayed in the image display area Ta by an operation described later and operates the tracking start button Th, the tracking operation of the tracking target is started. FIG. 11 is a diagram illustrating a display example of the display unit 20 during the tracking operation of the tracking target. As shown in FIG. 11, a tracking stop button Ti is displayed on the screen of the display unit 20 instead of the tracking start button Th of FIG.

  FIG. 12 is a diagram illustrating an example of a moving image displayed in the image display area Ta during the tracking operation of the tracking target. Images V1 to V20 are sequentially displayed in the image display area Ta. In the example of FIG. 12, one protrusion M1 of the object M is designated as a tracking target. As shown in FIG. 12, a symbol S indicating the tracking result in the current frame image and a symbol S indicating the tracking result in the previous frame image are displayed on each image displayed in the image display area Ta. In this example, the symbol S indicating the tracking result has a triangular shape.

  Note that an image V3 of the third frame is shown in the image display area Ta of FIG. In this state, when the user operates the tracking stop button Ti, an image during execution of the tracking operation (the image V3 of the third frame in the example of FIG. 11) and an image of the previous frame (the first and second frames in the example of FIG. 11). In the state where the symbol S indicating the tracking result of the images V1, V2) is displayed, the tracking operation of the tracking target is temporarily stopped.

  FIG. 13 is a diagram illustrating a display example of the display unit 20 at the end of the tracking operation of the tracking target. When the tracking operation of the tracking target in the image V20 of the final frame of the moving image file is completed, a tracking restart button Tj is displayed on the screen of the display unit 20 instead of the tracking start button Th of FIG. 9, as shown in FIG. In addition, an end button Tk is mainly displayed. In the image display area Ta, the image V20 of the last frame is displayed, and a plurality of symbols S indicating the tracking results in the images V1 to V20 of all frames are displayed on the image V20. A trajectory of a tracking target in which a plurality of symbols S move in a moving image is shown.

  In this state, the user can again execute the tracking operation of the tracking target in order from the image of the desired frame of the moving image file by operating the reverse playback button Te and the tracking restart button Tj.

  Further, when the user operates the tracking stop button Ti in FIG. 11 while the tracking operation is being executed, the tracking restart button Tj is displayed on the display unit 20 as in the case of the end of the tracking operation in FIG. An end button Tk is further displayed. In this case, the user can continue the tracking operation of the tracking target by operating the tracking restart button Tj.

  When the user operates the end button Tk, the control unit 10 in FIG. 1 uses the position data indicating the positions of the plurality of symbols S displayed in the image display area Ta in FIG. 13 as tracking results, and the storage unit 40 in FIG. And the tracking process is terminated.

  FIG. 14 is a diagram illustrating an example of a moving image displayed in the image display area Ta when tracking of a tracking target fails. Images V1 to V20 are sequentially displayed in the image display area Ta.

  In the example of FIG. 14, in the images V1 to V3 of the first to third frames, the protrusion M1 that is the tracking target is accurately tracked. Thereby, the symbol S indicating the tracking result is displayed on the protrusion M1. In the image V4 of the fourth frame, a portion different from the protrusion M1 to be tracked is tracked. Thereby, on the image V4, the symbol S indicating the tracking result is displayed at a position different from the protrusion M1 to be tracked. As a result, in the subsequent images V5 to V20 from the fifth frame to the last frame, a portion different from the tracking target protrusion M1 is tracked, and the symbol S indicating the tracking result is at a position different from the tracking target protrusion M1. It is displayed.

  In such a case, as will be described later, the user can reversely play back the moving image, and can resume the tracking operation from the image V4 of the frame in which tracking of the tracking target has failed.

(4) Tracking Target Tracking Processing FIGS. 15, 16, and 17 are flowcharts showing tracking target tracking processing in a moving image in the moving image tracking apparatus. 18, 19, and 20 are diagrams illustrating transition of an image displayed in the image display area Ta during execution of the tracking process of the tracking target. FIG. 21 is a diagram illustrating data stored in the RAM 10a of the control unit 10 during execution of the tracking process. The CPU of the control unit 10 executes the tracking process according to the tracking process program stored in the storage unit 40. Hereinafter, tracking processing of a tracking target by the CPU of the control unit 10 (hereinafter, abbreviated as the control unit 10) will be described with reference to FIGS.

  In the display state of FIG. 9, the user uses the operation unit 4 to select one of a plurality of moving image files stored in the storage unit 40 of FIG. 1. Thereby, the control part 10 reads the image data of the several frames in the selected moving image file (step S1). In this case, the read image data of a plurality of frames is stored in the RAM 10a of FIG.

  The user can designate a range of an arbitrary image among images based on image data of a plurality of frames in the moving image file by moving the start point display cursor Tl and the end point display cursor Tm using the operation unit 4.

  A tracking target may appear from an intermediate image among a plurality of frames constituting a moving image, or the tracking target may disappear from the intermediate image. In such a case, the user can set the start point display cursor Tl to a position where an image where the tracking target appears is displayed and set the end point display cursor Tm to a position where the image where the tracking target disappears is displayed. . As a result, it is possible to perform tracking only on an image having a tracking target. In this example, since the tracking target appears in all the images V1 to V20, the start point display cursor Tl is set at a position corresponding to the image V1 of the first frame, as indicated by a dotted arrow in FIG. The end point display cursor Tm is set at a position corresponding to the image V20 of the last frame. Therefore, the image data v1 to v20 of the last frame are read from the first frame.

  Next, the control unit 10 uses the video memory 50 of FIG. 1 for an image designated by the user (hereinafter referred to as an initial image) among the images V1 to V20 based on the image data v1 to v20 stored in the RAM 10a. Through the image display area Ta (step S2).

  In the initial state, the image V1 based on the image data v1 of the first frame stored in the RAM 10a is displayed in the image display area Ta as an initial image. The user can designate the initial image by moving the slider Tg on the seek bar Tf or by operating the playback button Tc. As will be described later, when the tracking process is resumed, an image designated by the user is displayed as an initial image in the image display area Ta.

  In this example, as shown in FIG. 18A, an image V1 based on the image data v1 of the first frame is displayed as an initial image.

  Next, the control unit 10 determines whether an arbitrary position in the image display area Ta is designated by the user using the operation unit 4 (step S3). When the position in the image display area Ta is not designated by the user, the control unit 10 returns to the process of step S2. As shown in FIG. 18B, when the position in the image display area Ta is designated by the user with an arrow pointer, the control unit 10 displays the symbol S and the template T on the image based on the designated position. And the position of the search range R is determined (step S4).

  In this example, as shown in FIG. 18C, the symbol S has a triangular shape, and the template T and the search range R have a rectangular shape. The size of the search range R is larger than the size of the template T. The position of the symbol S matches the position specified by the user. The centers of the template T and the search range R coincide with the position of the symbol S. The search range R is located so as to surround the template T. As will be described later, an image in the template T is set as a tracking target image. The search range R indicates a range in which a tracking target is searched for in the next frame image. Symbol S indicates the position where the tracking target exists.

  Subsequently, the control unit 10 stores the position data s1 indicating the position of the determined symbol S, the position data td indicating the position of the template T, and the position data rd indicating the position of the search range R in the RAM 10a shown in FIG. (Step S5). Further, the control unit 10 displays the template T, the symbol S, and the search range R in the image display area Ta based on the position data s1, td, and rd stored in the RAM 10a (step S6). In this example, as shown in FIG. 18D, the symbol S is displayed on one protrusion M1 of the object M that is the tracking target. Further, the template T is displayed so as to surround the symbol S, and the search range R is displayed so as to surround the template T.

  Note that the shape of the symbol S, the size of the template T, and the size of the search range R are preset by the user using the operation unit 4.

  The control unit 10 can move the symbol S within the image display area Ta based on the operation of the operation unit 4 by the user. In this case, as the symbol S moves, the template T and the search range R also move. Moreover, the control part 10 can change the size of the template T or the search range R based on operation of the operation part 4 by a user. By adjusting the position of the symbol S and the size of the template T and the search range R, the user can appropriately set the range of the tracking target image and the range of searching for the tracking target in the image of the next frame.

  Furthermore, the control unit 10 performs a similar process based on the operation of the operation unit 4 by the user, and performs another symbol different from the symbol S, the template T, and the search range R at any position in the image display region Ta Templates and search ranges can be displayed. Thereby, the user can set a plurality of tracking targets in the initial image.

  Next, the control unit 10 determines whether or not the execution of the tracking operation is instructed (step S7). Here, if the control unit 10 detects that the user has operated the tracking start button Th in FIG. 9 or the tracking restart button Tj in FIG. 13 using the operation unit 4, the control unit 10 determines that the execution of the tracking operation has been instructed. . When the execution of the tracking operation is not instructed, the control unit 10 returns to the process of step S2.

  When the execution of the tracking operation is instructed, the control unit 10 stores the image in the template T in the RAM 10a as the reference image data Re (step S8). In this example, at this point, the control unit 10 erases the display of the template T and the search range R from the image display area Ta.

  Subsequently, the control unit 10 displays the image of the next frame and the symbol S corresponding to the image of the previous frame in the image display area Ta (step S9). In this example, as illustrated in FIG. 18E, the control unit 10 displays the symbol S corresponding to the image V1 of the first frame on the image V2 of the second frame.

  Here, the control unit 10 identifies the position of the tracking target within the search range R of the image of the current frame based on the reference image data Re stored in the RAM 10a (step S10). In this example, the protrusion M1 of the image V2 of the second frame shown in FIG. In this case, the position of the protrusion M1 is identified by pattern matching between the data corresponding to the search range R in the image data v2 and the reference image data Re.

  Subsequently, the control unit 10 determines the position of the identified tracking target as the position of the new symbol S (step S11). Further, the control unit 10 stores the position data s2 indicating the position of the new symbol S in the RAM 10a (step S12).

  Next, the control unit 10 re-determines the position of the search range R based on the determined position of the new symbol S (step S13). In this case, the position of the search range R is redetermined so that the center of the search range R matches the position of the new symbol S. The control unit 10 updates the position data rd stored in the RAM 10a based on the re-determined position of the search range R (step S14). In this example, as shown in FIG. 18 (f), the search range R moves around the new symbol S on the image V2 of the second frame.

  Next, the control unit 10 displays a new symbol S in the image display area Ta based on the new position data s2 stored in the RAM 10a (step S15). In this example, as shown in FIG. 19A, the symbol S corresponding to the image V1 of the first frame and the symbol S corresponding to the image V2 of the second frame are displayed on the image V2 of the second frame.

  Next, the control unit 10 determines whether or not the last frame image V20 is displayed among the plurality of frames of images V1 to V20 designated in the image display area Ta (step S16).

  When the image V20 of the last frame is not displayed in the image display area Ta, the control unit 10 determines whether or not a pause is instructed (step S17). For example, when the user notices that the tracking has failed during the execution of the tracking operation, the user operates the tracking stop button Ti in FIG. When it is detected that the tracking stop button Ti has been operated, the control unit 10 determines that a temporary stop has been instructed.

  When the pause is not instructed, the control unit 10 returns to step S9 and repeatedly executes the processes of steps S9 to S16. In this example, as shown in FIG. 19B, the image V3 of the third frame is displayed in the image display area Ta, and the symbols S corresponding to the first and second frames are displayed on the image V3. . The position of the tracking target is identified within the search range R in the image V3 of the third frame. Thereby, as shown in FIG. 19C, the identified position of the tracking target is determined as the position of the new symbol S, and the position data s3 indicating the position of the new symbol S is stored in the RAM 10a. Further, the position of the search range R is redetermined based on the position of the new symbol S, and the position data rd stored in the RAM 10a is updated based on the redetermined position of the search range R. Next, as shown in FIG. 19D, a new symbol S is displayed on the image V3 of the third frame based on the position data s3 stored in the RAM 10a.

  When the image V20 of the last frame is displayed in the image display area Ta in step S16 or when a pause is instructed in step S17, the control unit 10 determines whether or not the end of the tracking operation is instructed. (Step S18). For example, as shown in FIG. 19E, when the user determines that accurate tracking has been performed up to the image V20 of the final frame displayed in the image display area Ta, the operation unit 4 is used to end FIG. The button Tk is operated. When detecting that the end button Tk has been operated, the control unit 10 determines that the end of the tracking operation has been instructed. In this case, the control unit 10 stores the position data s1 to s20 of the symbol S stored in the RAM 10a as the tracking result in the storage unit 40 of FIG. 1 (step S19), and ends the tracking process.

  On the other hand, when the end of the tracking operation is not instructed in step S18, the control unit 10 returns to the process of step S2. For example, as shown in FIG. 20A, after the tracking operation is performed on all the images V1 to V20, or as shown in FIG. 20B, the tracking has failed. When noticing, the user does not operate the end button Tk in FIG.

  In this example, as shown in FIG. 20C, tracking of the tracking target has failed in the image V4 of the fourth frame. In this case, as shown in FIG. 20D, when the user operates the reverse playback button Te or the slider Tg of the seek bar Tf in FIG. The image V3 of the previous frame can be designated as the initial image.

  Based on the operation of the operation unit 4 by the user, the control unit 10 displays the image V3 based on the third frame image data v3 as an initial image in the image display area Ta (step S2). The user designates the position of the image V3 of the third frame. Thereby, the control part 10 repeats the process of step S2-S18. The tracking operation of the tracking target is performed again on the image of the frame subsequent to the image V3 of the third frame.

  As a result, as shown in FIGS. 20E and 20F, the symbol S indicating the tracking result is displayed on the protrusion M1 of the object M to be tracked in the images V4 to V20 from the fourth frame to the last frame. Is done. The track of the tracking target in the moving image is displayed by a plurality of symbols S indicating the tracking result.

(5) Effect In the moving image tracking apparatus according to the present embodiment, one image among the images of a plurality of frames stored in the storage unit 40 is displayed on the display unit 20 as an initial image. When the template T and the search range R are set in the initial image by the operation unit 4 and the playback button Tc is operated, the tracking operation is performed on the images of the frames after the initial image.

  In the tracking operation, images of a plurality of frames are displayed in order on the display unit 20. In each displayed image, the position of the tracking target is identified within the search range R set by the operation unit 4, and the symbol S is displayed at the identified position.

  When the operation unit 4 instructs the display unit 20 to display any one of a plurality of frames as a new initial image after the tracking operation is stopped or ended, the new initial image is displayed on the display unit 20. Further, when the template T or the search range R is reset in the new initial image by the operation unit 4 and the tracking resumption button Tj is operated, the tracking operation is executed for the images of the frames after the new initial image.

  When tracking of a tracking target fails during execution of the tracking operation, an image of a desired frame is displayed as a new initial image by operating the reverse playback button Te or seek bar Tf after the tracking operation is stopped or ended. Can do.

  When operating the reverse playback button Te, the user can easily find the image of the frame in which the tracking has failed first while viewing the image and the symbol S of each frame. Thereby, the tracking operation can be easily restarted from the image of the first frame in which the tracking has failed or the image of the frame immediately before the tracking has failed.

  Further, when operating the seek bar Tf, the user can quickly display an image of the first frame in which tracking has failed or an image of the frame immediately before the tracking has failed as an initial image. Accordingly, the tracking operation can be quickly restarted from the image of the first frame in which tracking has failed or the image of the frame immediately before the tracking has failed.

  As a result, it is possible to resume the tracking operation for an image that has failed to be tracked while maintaining the tracking result of the image that has been successfully tracked.

  Further, since the template T or the search range R can be reset in the new initial image, it is possible to prevent the tracking failure in the previous tracking operation from being repeated in the tracking operation at the time of restart. Thereby, the probability that the tracking for each image is successful in the resumed tracking operation is increased. Therefore, the entire execution time of the tracking operation is shortened. As a result, an accurate and accurate tracking result can be obtained.

(6) Other Embodiments (6-1) FIG. 22 is a flowchart showing tracking processing of a tracking target in a moving image in a moving image tracking apparatus according to another embodiment. In the present embodiment, as shown in FIG. 22, in the tracking process of the tracking target, the accuracy of the identified tracking target position is calculated after step S10 of FIG. 16 (step S10a). The accuracy is, for example, the degree of coincidence between the image data at the identified position and the reference image data.

  The degree of coincidence is, for example, the number of coincidence between the values of a plurality of pixels constituting the image portion at the identified position and the values of the plurality of pixels constituting the template T. In this case, the greater the number of matches, the higher the accuracy. Further, the degree of coincidence may be the sum of the difference values between the values of the plurality of pixels constituting the image portion at the identified position and the values of the plurality of pixels constituting the template T. In this case, the accuracy increases as the sum of the difference values decreases. Further, the degree of coincidence may be a correlation value between a value of a plurality of pixels constituting the image portion at the identified position and a value of a plurality of pixels constituting the template T. In this case, the closer the correlation value is to 1, the higher the accuracy.

  Thereafter, the control unit 10 compares the calculated accuracy with a predetermined threshold value (step S10b). When the accuracy is equal to or less than the threshold value, the control unit 10 temporarily stops the tracking operation (step S10c), and proceeds to the process of step S18. On the other hand, when the accuracy exceeds the threshold value in step S10b, the control unit 10 continues the processing from step S11.

  In this case, the tracking operation is automatically stopped when the accuracy falls below the threshold when the tracking fails. Thereby, tracking is continued when the tracking result for the tracking target of the image of each frame is accurate, and tracking is stopped when the tracking result is not accurate. Therefore, it is possible to prompt the user to confirm the tracking result or to correct the setting of the template and the search range. As a result, the tracking operation can be easily restarted from the image of the first frame in which the tracking has failed or the image of the frame immediately before the tracking has failed.

  In FIG. 22, the processing of steps S10a to S10c is executed between steps S10 and S11, but is not limited to this. The processes of steps S10a to S10c are executed between steps S10 and S11, between steps S11 and S12, between steps S12 and S13, between steps S13 and S14, between steps S14 and S15, and between steps S15 and S16. Also good.

  (6-2) In step S19, the control unit 10 stores the tracking result and ends the tracking process, but is not limited thereto. When the user does not select storage of the tracking result, the control unit 10 may end the tracking process without storing the tracking result.

(7) Correspondence between each component of claim and each part of embodiment The following describes an example of the correspondence between each component of the claim and each part of the embodiment. It is not limited.

  Images V1 to V20 are examples of images, image data v1 to v20 are examples of image data, image processing system 100 is an example of a moving image tracking device, storage unit 40 is an example of a storage unit, and display unit 20 is an example of a display unit, the operation unit 4 is an example of an operation unit, and the control unit 10 is an example of a control unit or a processing device. The template T is an example of the tracking condition and the range of the image portion to be tracked in the initial image, the search range R is an example of the tracking condition and the range of searching for the tracking target in the image of each frame, and the symbol S is It is an example of a symbol. The function of the reverse playback button Te is an example of the reverse playback function, and the seek bar Tf is an example of the selection unit.

  As each constituent element in the claims, various other elements having configurations or functions described in the claims can be used.

  The present invention can be effectively used for various image processing systems, image processing methods, and image processing programs.

DESCRIPTION OF SYMBOLS 1 Main body part 2 Camera 3 Illumination device 4 Operation part 5 PLC
6, 8 Sensor 7 Waveform collection device 10 Control unit 10a RAM
DESCRIPTION OF SYMBOLS 20 Display part 30 Recording part 30A Movie data storage area 30B Waveform data storage area 40 Storage part 50 Video memory 60 Waveform collection part 100 Image processing system BS bus FA-FD Movie file fi File information i1-i42 Evaluation value L Current display position bar M Object M1, M2 Protrusion rd, s1 to s20, td Position data R Search range Re Reference image data S Symbol Sa, Ta Image display area Sb Waveform display area Sc Condition setting area Sc1 to Sc5 Input frame Se Record button Sf Trigger button Sg , Tc Playback button Sj File button Sy Tracking button T Template Td Pause button Te Reverse playback button Tf Seek bar Tg Slider Th Tracking start button Ti Tracking stop button Tj Tracking restart button Tk End button Tl Start point display Sol Tm end point display cursor v1~v20 image data V1~V20 image W subject

Claims (8)

A video tracking device that tracks a tracking target in a video composed of images of a plurality of frames,
A storage unit for storing a plurality of frames of image data for displaying the plurality of frames of images;
A display unit that displays an image based on image data of each frame stored in the storage unit;
An operation unit operated by a user to set a tracking condition of a tracking target in an image displayed on the display unit and to give an instruction regarding tracking of the tracking target;
A tracking operation in which images of a plurality of frames are sequentially displayed on the display unit, a position of a tracking target is identified in the displayed image according to the tracking condition set by the operation unit, and a symbol is displayed at the identified position. A control unit to execute,
The control unit causes the display unit to display one image among the images of the plurality of frames as an initial image, and when a tracking condition is set in the initial image and the start of tracking is instructed by the operation unit. The tracking operation is performed on frames after the initial image, and after the tracking operation is stopped or ended, any one of the images of the plurality of frames is displayed as a new initial image by the operation unit. Is displayed on the display unit, the tracking condition is reset in the new initial image and the start of tracking is instructed by the operation unit, the new initial image is displayed. A moving image tracking apparatus that performs the tracking operation on an image of a frame after the initial image. The control unit includes a tracking target position identified in an image currently displayed on the display unit, and a tracking target position identified in an image of a frame before the image currently displayed on the display unit. Each of the symbols is displayed, and when the tracking condition is reset in the new initial image and the start of tracking is instructed, the position of the tracking target identified in the image currently displayed on the display unit; The position of the tracking target identified in the image of the frame after the new initial image and before the image currently displayed on the display unit, and the image of the frame prior to the new initial image The moving image tracking apparatus according to claim 1, wherein the symbol is displayed at each of the tracking target positions. The moving image according to claim 1, wherein the tracking condition includes at least one of a range of an image portion to be tracked in the initial image and a range to search for a tracking target in an image of each frame. Tracking device. The control unit has a reverse playback function of displaying images of a plurality of frames on the display unit in a time order opposite to that in the tracking operation,
The said operation part can instruct | indicate the display of the said new initial image by instruct | indicating the reverse reproduction after the tracking operation | movement is stopped or complete | finished. Video tracking device. The control unit causes the display unit to display a selection unit for selecting an image to be displayed on the display unit,
The moving image tracking according to claim 1, wherein the operation unit can instruct display of the new initial image by operating the selection unit displayed on the display unit. apparatus. The control unit evaluates the accuracy of the identification result of the tracking target position for each image during execution of the tracking operation, and stops the tracking operation when the accuracy falls below a predetermined threshold value. The moving image tracking device according to any one of claims 1 to 5, wherein A video tracking method for tracking a tracking target in a video composed of images of a plurality of frames,
Storing a plurality of frames of image data for displaying the plurality of frames of images;
Displaying one image of a plurality of frames of images as an initial image on a display unit based on the stored plurality of frames of image data;
When the tracking condition is set in the initial image and the start of tracking is instructed, the images of the frames after the initial image are sequentially displayed on the display unit based on the stored image data of the plurality of frames. Performing a first tracking operation for identifying the position of the tracking target under the tracking condition set in each of the displayed images and displaying a symbol at the identified position;
When it is instructed to display any one of the images of the plurality of frames as a new initial image after stopping or ending the first tracking operation, based on the stored image data of the plurality of frames Displaying the new initial image on the display unit,
When the tracking condition is reset in the new initial image and the start of tracking is instructed, the display unit displays images of frames after the new initial image based on the stored image data of the plurality of frames. And sequentially executing a second tracking operation for identifying the position of the tracking target under the tracking condition set in each of the displayed images and displaying a symbol at the identified position. Video tracking method. There is a moving image tracking program for causing a processing device to execute a tracking process for tracking a tracking target in a moving image composed of a plurality of frames of images,
Storing a plurality of frames of image data for displaying the plurality of frames of images;
A process of displaying one image among a plurality of frames of images as an initial image on the display unit based on the stored plurality of frames of image data;
When the tracking condition is set in the initial image and the start of tracking is instructed, the images of the frames after the initial image are sequentially displayed on the display unit based on the stored image data of the plurality of frames. A process of performing a first tracking operation for identifying a position of a tracking target under a tracking condition set in each of the displayed images and displaying a symbol at the identified position;
When it is instructed to display any one of the images of the plurality of frames as a new initial image after stopping or ending the first tracking operation, based on the stored image data of the plurality of frames Processing to display the new initial image on the display unit,
When the tracking condition is reset in the new initial image and the start of tracking is instructed, the display unit displays images of frames after the new initial image based on the stored image data of the plurality of frames. A second tracking operation for displaying the symbols in order and identifying the position of the tracking target under the tracking condition set in each of the displayed images and displaying a symbol at the identified position;
A moving picture tracking program which is executed by the processing apparatus.

Images