JP4993678B2 - Interactive moving image monitoring method, interactive moving image monitoring apparatus, and interactive moving image monitoring program - Google Patents

Description

  The present invention relates to an interactive moving image monitoring method, an interactive moving image monitoring apparatus, and an interactive moving image monitoring program. More specifically, the present invention relates to an interactive moving image monitoring method, a moving image monitoring apparatus, and a moving image monitoring program suitable for a long-time moving image monitoring system.

  In recent years, image surveillance technology using IT equipment such as digital video cameras and computers, which have been rapidly developed, has been put into practical use. The technology that becomes the center of the image monitoring technology is an image recognition technology that recognizes a captured image by a computer and determines, for example, whether there is an intruder or detects a failure of a device.

  For example, in electric power companies, in order to reduce maintenance and management costs, image monitoring technology is being put to practical use for detecting damaged parts of power-related equipment. In addition, image monitoring technology for detecting intruders in power-related facilities is being put to practical use.

  Conventionally, supervised machine learning technologies such as SVM (Support Vector Machine) and Boosting (Boosting) exist as technologies for realizing high-accuracy image recognition, and they are already used in various fields. . In these supervised machine learning, cases are learned based on cases that should be recognized (referred to as positive examples) and those that should not be recognized (referred to as negative examples) that humans have taught to computers. Appropriate recognition is also made for. For example, Patent Document 1 describes a technique for determining a human face using SVM.

  Furthermore, if a failure is detected in a power-related facility or the like, a failure rarely occurs, and it is not easy to select a location where an abnormality is shown from the monitoring video. For this kind of case teaching problem, Interactive Machine Learning (IML) that uses human operation to solve the problem by using a direct operation type user interface is proposed. Has been. IML, for example, visualizes the analysis results of cases (hereinafter also referred to as data) with good visibility using colors and symbols, so that the user can improve the quality of case teaching (hereinafter also referred to as labeling) In addition, it is possible to easily grasp the place where the label needs to be corrected.

For example, Non-Patent Document 1 proposes a system in which a machine can be learned by interactive processing instead of batch processing by an interface similar to drawing software, and still image recognition can be realized. Non-Patent Document 2 proposes a system that simultaneously analyzes a plurality of sensor information such as sound, image, RFID, and the like to determine the presence or absence of a person.
JP 20064003 A Jerry Alan Fails and Dan R. Olsen Jr. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent user interfaces, pp. 39-45, 2003. Anind K. Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu.a CAPpella: programming by demonstration of contextaware applications.

  The case teaching required for image recognition gives a task of selecting data to be taught from a huge amount of data (selection of case) and information (positive or negative label) as to whether or not the data is to be recognized. Although it consists of work (labeling), both the selection and labeling of this case requires human effort and is required to be reduced.

  However, although supervised machine learning as described in Patent Document 1 has high recognition accuracy, the teaching of the learning system until it is possible to recognize with high accuracy requires a great deal of time and effort for teaching cases by humans. There is a problem that it takes a long time.

  For example, in the case of an outdoor surveillance video, various phenomena that change the color and shape of appearance (such as rust, daylight, morning, evening, rain, cloudy, snow, artificial lighting, etc.) (Scratches, peeling, discharges, intruders, animals and plants, etc.) must be recognized. I can't say that. Therefore, in order to construct a practical image monitoring system, it is necessary to reduce the trouble of teaching examples.

  In addition, the recognition processing in the supervised machine learning is performed by batch processing, and there is a problem that processing takes time. In addition, since it is a batch process, interactive execution in which the user checks the learning status or confirms the recognition accuracy as needed during the process is impossible. Therefore, if the recognition result is not satisfactory, after batch processing is completed, it is necessary to teach the case again and perform batch processing again, and it takes time to obtain a satisfactory recognition result. There was a problem that it took.

  In order to solve this problem, the technique described in Non-Patent Document 1 attempts to solve the problem by allowing the user to recognize the correctness by immediately visualizing and displaying the recognition result. For this reason, it can be applied to still images, but in tracking an object in a moving image, the amount of information handled is tens of thousands of times that of a still image, so processing takes time and immediate feedback can be returned. There is no problem. That is, when applied to a moving image, the teaching result is not immediately reflected even in interactive processing, and the same problem as in batch processing occurs.

  As described above, the processing time increases as the amount of information increases, so in moving image monitoring that handles an enormous amount of information, processing can only be performed by batch processing, and it is difficult to realize an interactive machine learning system. It was thought to be.

  In the technique described in Non-Patent Document 2, although continuous images are targeted, a person is monitored by combining sound information and wireless ID tag information in addition to image information, and only image information is displayed. It is not possible to monitor moving images using In addition, when a non-human subject is monitored or an intruder is monitored, sound information, wireless IC tag information, or the like may not be obtained. In the technique described in Non-Patent Document 2, monitoring is not possible. Impossible to do.

  Furthermore, in the technique using supervised machine learning described in Patent Document 1 and IML as shown in Non-Patent Documents 1 and 2, in the process of adding a case teaching by a user, the added case teaching is There is a problem that the user cannot recognize the situation because there is no way of knowing how it affects the overall identification accuracy. In other words, there is a problem that the user does not understand the effect of the case teaching currently being performed.

  Therefore, in the monitoring of moving images, it is possible to select the best case teaching method while grasping the case teaching situation immediately in the process of teaching the case by the user, and compared with the conventional batch processing. Realization of an interactive processing system that is quick and has high recognition accuracy is desired.

  Therefore, the present invention monitors and tracks an object in moving image data based on a minimum case taught by a user, and further makes it possible to visualize and display the monitoring and tracking result in an easy-to-understand manner. An object of the present invention is to provide an image monitoring method, an interactive moving image monitoring apparatus, and an interactive moving image monitoring program.

  In order to achieve such an object, the interactive moving image monitoring method according to claim 1 is a method of monitoring and tracking a monitoring object in a moving image, and is a process to be subjected to image processing among frame images of the moving image. One or more image clips are provided in the target area, and label registration processing for registering labeling data as to whether or not the monitoring target is photographed in the image clip, and the color of the pixel included in the image clip for the image clip A high-dimensional space in which the image feature amount is calculated based on the information value, and the image feature amount of the image clip for which labeling data is not registered is centered on each element of the image feature amount of the image clip for which labeling data is registered In the above, for images in a predetermined range, an image in which no labeling data is registered for tagging according to the labeling data. Tagged processing performed with respect to the lip, and to perform a visualization process of displaying together with a frame image of the moving image based on the results of the tagging process to the importance of the tag.

  The interactive moving image monitoring apparatus according to claim 6 is an apparatus that monitors and tracks a monitoring object in a moving image, and reads out a frame image of the moving image and performs processing on a processing target area. One or more image clips are set in the label, and label registration means for registering in the database labeling data designated in advance as to whether or not the monitoring object is photographed in the image clip, and the image clip includes the image clip The image feature amount is calculated based on the color information value of the pixel to be recorded, and the image feature amount of the image clip whose labeling data is not registered in the database is the element of the image feature amount of the image clip whose labeling data is registered. In a high-dimensional space with the axis as the axis, if it is within a predetermined range, the labeling data will be tagged according to the labeling data. Tagging means for storing in association with unrecorded image clips, and visualization means for displaying the result of tagging processing on the output device together with the frame image of the moving image based on the importance of the tag. .

  The interactive moving image monitoring program according to claim 11 sets one or more image clips in a processing target area to be subjected to image processing among frame images of the moving image, and stores them in the main storage device. Read the target area setting process and labeling data registered in advance in the storage device as to whether or not the monitoring object is photographed in the image clip, and use the color information value of the pixel included in the image clip for the image clip. The image feature amount is obtained based on the image feature amount of the image clip in which the labeling data is not registered, in a high-dimensional space with the element of the image feature amount of the image clip in which the labeling data is registered as an axis, If it is within a predetermined range, tag the image clip for which no labeling data is registered according to the labeling data, In addition, the monitoring target in the moving image is monitored by causing the computer to execute image recognition and analysis processing for displaying the tagging result on the output device together with the frame image of the moving image based on the importance of the tag. To track.

  Therefore, first, for each frame image of the moving image in which the monitoring target is photographed, a portion to be subjected to image processing is set as a processing target area, and at least one image clip is set in the processing target area. Divide into areas. A label selected by the user for each image clip, specifically, a label (target label) attached to an image clip in which the monitoring object is photographed, or the monitoring object is not photographed. Any labeling data of a label (non-target label) attached to the image clip is registered in the database. Further, for each image clip of each frame, an image feature amount is calculated based on color information values such as RGB and HSV of pixels in the image clip. In addition, in a high-dimensional space around each element of the image feature amount for an image clip for which labeling data has already been registered in the database, the image feature value that is included in a predetermined range from there is still For an image clip for which labeling data is not registered, a target tag is assigned if the reference label is a target label, and a non-target tag is assigned if the label is a non-target label. Further, the tag information of all frames that have been labeled and tagged are displayed together on the same screen as the displayed frame image based on the importance of the tag. In addition, in this specification, a tag means the label which a computer estimates based on the attached label. The image feature amount is information such as a color or a pattern that characterizes an image calculated from image data targeted by the computer.

  According to a second aspect of the present invention, in the interactive moving image monitoring method according to the first aspect, a local sensitive hash algorithm is used when labeling data is registered and searched. The invention described in claim 7 uses the local sensitive hash algorithm in the interactive moving image monitoring apparatus according to claim 2 when registering and searching the labeling data.

  Therefore, a locality-sensitive hash (LSH), which is an approximate nearest neighbor (ANN) search algorithm using a hash function, is used instead of a tree-type search algorithm.

  According to a third aspect of the present invention, in the interactive moving image monitoring method according to the first or second aspect of the invention, the tagging process is performed by calculating an image feature amount of an image clip labeled that the monitoring target is photographed. In a high-dimensional space with each element as an axis, the target tag and the image feature value of an image clip labeled that the monitoring target object is not photographed are used as axes for those in a predetermined range. In a high-dimensional space, a non-target tag is tagged for a certain range, and an unknown tag is tagged for an image clip that is not in any range.

  According to an eighth aspect of the present invention, in the interactive moving image monitoring apparatus according to the sixth or seventh aspect, the tagging means includes an image feature amount of an image clip labeled that the monitored object is photographed. In a high-dimensional space with each element as an axis, the target tag and the image feature value of an image clip labeled that the monitoring target object is not photographed are used as axes for objects in a predetermined range. In the high-dimensional space, a non-target tag is tagged for a certain range, and an unknown tag is tagged for an image clip that is not in any range.

  Therefore, data that cannot be determined from the labeling data (image feature amount of an image clip), that is, in a high-dimensional space with each element of the image feature amount of any label registered as a positive example or a negative example as a constant. Data that is not within the distance is set as an unknown tag.

  According to a fourth aspect of the present invention, in the interactive moving image monitoring method according to any one of the first to third aspects, the visualization process compresses the result of the tagging process for all frame images of the moving image. Is displayed on one screen. The invention according to claim 9 is the interactive moving image monitoring apparatus according to any one of claims 6 to 8, wherein the visualization means compresses the tagging results for all frame images of the moving image. Displayed on a single screen.

  Therefore, the tagging results for all the frames of the target moving image are compressed and displayed on one screen so as to be within the number of display pixels of the output device.

  According to a fifth aspect of the present invention, in the interactive moving image monitoring method according to the fourth aspect, the importance of the tag is the highest for the unknown tag and the lowest for the non-target tag. According to a tenth aspect of the present invention, in the interactive moving image monitoring apparatus according to the ninth aspect, the importance of a tag is highest for an unknown tag and lowest for a non-target tag.

  Therefore, when compressing and displaying the tagging result, the unknown tag is displayed with the highest priority, and then the target tag and the non-target tag are displayed in this order.

  As described above, according to the interactive moving image monitoring method according to claim 1, the interactive moving image monitoring apparatus according to claim 6, and the interactive moving image monitoring program according to claim 11, the moving image It is possible to analyze the object to be monitored while confirming the effect of the case teaching by interactive processing and improving the monitoring and tracking accuracy.

  Also, by looking at the tagging result visualized and displayed at the same time as the playback of the moving image, the temporal and spatial changes of the tag can be captured instantly. In other words, it is possible to immediately grasp the monitoring and tracking accuracy of the object under the current teaching status (labeling status). What kind of labeling should be performed for further monitoring and improvement of tracking accuracy? It is possible to judge whether it is good.

  In this way, by placing tagging under user control, the amount of information processing can be minimized, and an interactive monitoring system for moving images, which has been impossible in the past, can be realized. .

  In addition, because the amount of information processing is reduced, the labeling by the user is immediately feedback processed, so how the label newly taught by the user affects the monitoring status of the entire image each time. Can be confirmed immediately. As a result, the user can determine what kind of labeling should be added, and how to perform labeling can reduce labeling, that is, improve monitoring and tracking accuracy in a short time. You can judge whether you can.

  In addition, according to the interactive moving image monitoring method according to claim 2 and the interactive moving image monitoring apparatus according to claim 7, the tree structure is not complicated by the addition of data. Compared to the above, high-speed data registration and data search processing can be realized.

  Further, according to the interactive moving image monitoring method according to claim 3 and the interactive moving image monitoring apparatus according to claim 8, a target or non-target tag cannot be assigned from the labeling data, that is, a monitoring target. By adding an unknown tag to data that cannot be determined whether it is an object or a non-monitoring object and displaying it visually, the user is presented with what points should be used to provide case teaching. It is possible to monitor and track a highly accurate object with the minimum number of teachings.

  According to the interactive moving image monitoring method according to claim 4 and the interactive moving image monitoring apparatus according to claim 9, the number of frames of the target moving image is the number of pixels that can be output by the output device (for example, The tagging result is compressed and visualized on a single screen even if it exceeds (horizontal axis direction), allowing the user to continue teaching examples while confirming the labeling effect. Become.

  Further, according to the interactive moving image monitoring method according to claim 5 and the interactive moving image monitoring apparatus according to claim 10, even if the tagging result is displayed in a compressed manner, the tag Labeling can be performed around unknown tags that have not been estimated.

  Hereinafter, the configuration of the present invention will be described in detail based on the embodiment shown in FIGS.

  FIG. 1 shows an example of the configuration of the interactive image monitoring apparatus 1 of the present invention. An interactive image monitoring apparatus 1 according to the present invention includes an output device 2 such as a display, an input device 3 such as a keyboard and a mouse, a central processing arithmetic device (CPU) 4 that performs arithmetic processing, data being processed, parameters, and the like. Is stored, a hard disk as an auxiliary storage device 6 in which various data such as calculation results are recorded, an input interface 7 to which a captured moving image is input, and the like. . Hereinafter, the main storage device 5 and the auxiliary storage device 6 are collectively referred to simply as a storage device. The above hardware resources are electrically connected through the bus 8, for example.

  The input interface 7 has a function of converting a signal input from an imaging means 9 such as a video camera or read from a storage medium 10 such as a DVD or a video tape on which video is recorded into data that can be processed by a computer. Each frame image constituting the video has a function of recording the video data 14 as the video data 14 in the auxiliary storage device 6. As such an input interface 7, for example, an existing NTSC-RGB converter, a frame grabber, or an image capturing board for a personal computer may be used. The output device 2 displays a user interface screen and the like. The interactive image monitoring program of the present invention is recorded in the auxiliary storage device 6, and the computer functions as the interactive image monitoring device 1 when the program is read and executed by the CPU 4.

  The interactive image monitoring apparatus 1 includes a label registration unit 11 that executes label registration processing, a tagging unit 12 that executes tagging processing, and a visualization unit 13 that executes visualization processing. The labeling means 11, the tagging means 12, and the visualization means 13 can be configured by causing software executed by the CPU 4 to be executed by a computer.

  Data necessary for the execution is loaded into the RAM 5. The auxiliary storage device 6 stores video data 14, stores data taught by the user and labels thereof, an LSH database 15 that enables registration and retrieval of data, an image clip identification number, and a frame number I. An ICP (Image Clip Profile) database 16 that stores coordinate positions xy, labels, tags, and the like in the grid is constructed. In the RAM 5, a data map 17 of a visualization area that holds a label or a tag in a horizontal pixel × vertical pixel memory area of the visualization area 37 is formed. It should be noted that the visualization area data map 17 is all set to the unknown tag 53 at the time of initialization. Note that the auxiliary storage device 6 is not necessarily a device inside the computer, and it goes without saying that an external hard disk or an external storage device accessible via a network may be used. The configuration of the interactive image monitoring apparatus 1 described above is an example and is not limited to this.

  The interactive image monitoring method of the present invention is a method for monitoring and tracking a monitoring object in a moving image, wherein one or more image clips are included in a processing target area to be subjected to image processing among each frame image of the moving image. Providing a label registration process for registering labeling data as to whether or not the monitoring object is photographed in the image clip, and obtaining an image feature amount based on the color information value of the pixel included in the image clip. The image feature amount of an image clip for which no labeling data is registered is within a predetermined range in a high-dimensional space with the elements of the image feature amount of the image clip for which labeling data is registered as an axis. A tagging process for performing tagging according to the labeling data for an image clip for which no labeling data is registered; And it performs the visualization processing for displaying together with a frame image of the moving image based on the result of the grayed assigning process to the importance of the tag. Hereinafter, the interactive image monitoring method and the interactive image monitoring program in this embodiment will be described.

  In the interactive image monitoring method of the present invention, the speed of data search is improved by using LSH as an information search method compared to the tree-type search method. Therefore, simply using LSH makes feedback slow and interactive processing is impossible.

  Therefore, in the interactive image monitoring method of the present invention, the problem of feedback delay is solved by using a direct operation type graphical interface that can be easily and quickly labeled by the user, and placing the tagging process under user control. In addition, interactive monitoring with moving images that can immediately display the result of image recognition is realized.

  Direct operation refers to a user interface technique used in a window system, and specifically refers to file movement with a mouse and screen scrolling with a slider. This is characterized in that image symbols such as file icons and slider controls that visually represent resources in the computer can be manipulated as if they were moving in direct contact. In addition, the direct operation works effectively with regard to the visual recognition ability that the image data can be grasped instantaneously, the apparent movement that recognizes that the frame advance image within 50 msec is moving, and the phenomenon that occurred within 200 msec. This is because humans have the ability to detect change sensitively and feel a causal relationship. In direct operation, changes are immediately fed back so that the user can sensuously understand the effects of their labeling, and easily identify and specify cases where the calculator is wrong and needs to be corrected.

For example, as shown in the visual perception characteristics of Table 1, the processing content that can be processed by the user varies depending on the reaction time of the system.

  In the conventional IML, the range that can be directly operated is 50 ms or less. However, there are cases where only a slow response that takes more time than continuous operation can be returned. Therefore, the present invention improves user operability by allowing the user to selectively use three modes of “frame advance”, “normal playback”, and “batch processing” as moving image playback modes. Thus, the processing time is shortened to realize an interactive image monitoring system.

  The “frame advance” mode is a mode in which the user can label while checking images one frame at a time. Newly designated labeled data is instantly registered by LSH. The “normal playback” mode is a mode in which the user can check the tagging status while reproducing an image that has already been tagged at a TV rate (29.97 fps). In the “batch processing” mode, batch processing is performed while tagging processing is performed on all images. In the present embodiment, in the tagging process, the progress is always visualized and displayed, and the operability is improved by allowing the process to be paused and processed at any time even during batch processing.

  Next, description will be made with reference to a flowchart showing the entire processing performed by the interactive image monitoring program of the present invention shown in FIG.

  First, initial setting (S1) is performed. In the initial setting (S1), the target frame number i is initialized (i = 1), and the playback mode is set to “frame advance”. Further, if the LSH database 15 and the ICP database 16 already exist, they are read from the auxiliary storage device 6, and if they do not exist, a new database is created. The visualization area data map 17 is initialized.

  Next, target area setting (S2) is performed. In the target area setting (S2), first, an image of the first frame of the read moving image is displayed on the output device 2. If the monitoring object 40 (hereinafter also referred to as an object or target) is not photographed in the first frame image, the frame is moved by the fast-forward function until the user starts photographing the object 40. Just advance the statue. The video to be recognized and analyzed may be read from the video data 14 stored in advance in the auxiliary storage device 6 or may be directly captured from the imaging unit 9 and the recording medium 10. In the present embodiment, the video data 14 is, for example, an image with a TV rate (29.97 fps), but the frame rate is not particularly limited.

  FIG. 2 shows an example of a user interface screen of the interactive image monitoring program of the present invention. A video display area 22 displays a moving image on the window 21. An area surrounded by the outermost side of the grid 23 displayed on the screen indicates the processing target area 24 specified by the user. All subsequent image processing such as labeling and tagging is performed on the processing target area 24. By presuming such mask processing, the amount of calculation can be reduced and the mixing of noise can be restricted, so that high-speed and high-precision processing is possible. For example, in the case of DV-NTSC, the number of pixels of the video is 720 × 480 pixels, but the processing target area 24 can be limited by performing mask processing.

  In the case where it is desired to monitor only whether or not the entire screen has changed, for example, the processing target area 24 may not be specified, and the entire image displayed in the video display area 22 may be the processing target.

  The processing target area 24 is a grid 23 that delimits the processing target area 24 by selecting the processing target area 24 by a mouse operation or the like on the frame image displayed in the video display area 22 in which the target object 40 is captured. It is specified by specifying the size of. The setting of the grid 23 is not essential. For example, when it is desired to check only whether or not the object 40 exists in the processing target area 24, it is not necessary to set the grid 23.

  In the present embodiment, an image unit surrounded by the grid 23 in the frame image is referred to as an “image clip” and is a unit for labeling or tagging.

  In the present embodiment, the grid 23 is rectangular, but the grid 23 is not necessarily rectangular. Further, preprocessing such as background difference and various filter processes may be performed to cut out the image clip and display it in the processing target area 24. For example, if background difference is performed as preprocessing, only image clips that have moved between frames can be targeted for subsequent processing. Further, if the filtering process is performed as a pre-process, the color feature of the monitoring target object is registered in advance, so that only the image clip having the color feature can be set as a target for subsequent processing.

  Buttons arranged on the right side of the window 21 are user interfaces for designating video reproduction, batch processing, and the like. It should be noted that the user assistance function described below may be implemented as necessary and is not necessarily essential. Of course, the present invention is not limited to the functions described below, and other user assistance functions may be provided.

  The “<1S” button 25a returns to the image frame one second before. Similarly, the “<10f” button 25b returns to the image 10 frames before. When the “frame advance” button 26 is clicked, the next frame is displayed. The “inspect this frame” button 27 performs a tagging process on the frame image currently displayed in the window 21 and displays the result.

  For example, tagging for analyzing video requires tagging 750 data per second and 45,000 data per minute even in the later-described experiment of 25 data per frame (number of image clips). Therefore, if the entire image is tagged every time it is labeled, the processing takes time and interactive processing is impossible. However, since the number of data to be tagged is small (up to 25) if it is limited to the display frame currently being viewed by the user, when the “inspect this frame” button 27 is clicked, the frame is immediately displayed. Perform tagging and return feedback.

  The check for “stop when not specified” 30 is a function in the normal playback mode. In the normal playback mode, the tagging process is not performed, and video playback is performed while displaying the latest tagging result so far overlaid on the window 21. Here, if the “Stop at unspecified target” 30 is unchecked, playback will continue regardless of the type of tag, but if checked, it will automatically pause if an unknown tag appears. . If the check for “stop when not specified” 30 is checked, it is possible to save the trouble of searching for an unknown tag when labeling an unknown tag preferentially.

  The “normal playback” button 31 is for playing back the moving image after the frame image currently displayed in the window 21 in the normal playback mode. The “<< 10s” button 32a returns to the image 10 seconds before. Similarly, the “10s >>” button 32b displays an image after 10 seconds, and the “30s >>” button 32c displays an image after 30 seconds. The “batch inspection” button 33 performs batch processing on the frames after the image currently displayed in the window 21. The “clear tag map” button 34 initializes (clears) the visualization area 37 under the window 21. The “neighbor distance threshold value” button 35 is used to designate a neighborhood distance threshold value r used in the tagging process. The neighborhood distance threshold r is displayed in the field 35b when determined by the slider 35a. Further, by inputting the maximum value of the range of values that can be set with the slider 35a in the field 35c, the range of values that can be set with the slider 35a can be changed. The “InitializeDB” button 36 is used to initialize the LSH database 15 and the ICP database 16 for the image.

  In addition, the pattern displayed under the window 21 is a visualization region 37, that is, a region indicating a result of performing tagging processing on a processing target video. The visualization compression direction 38 can select whether visualization compression described later is performed in the horizontal axis direction or the vertical axis direction.

  The image slider 39 below the window 21 can be operated to operate an image frame at an arbitrary point of the moving image by scrolling. That is, when the image slider 39 is at the leftmost end, the first frame is displayed, and when it is at the rightmost end, the final frame is displayed.

  When the process up to the target area setting (S2) is completed, the image recognition and analysis process (S3 to S12) proceeds according to the designated reproduction mode. In principle, the image recognition and analysis processing is performed for each frame number i.

  In the image recognition and analysis process, first, video setting (S3) is performed. In the video setting (S3), the i-th frame image is read into the memory. At the same time, the grid 23 set in the target area setting (S2) is displayed in the video display area 22.

  In the present embodiment, a case where labeling is performed on a frame image displayed in the video display area 22 will be described. Note that the labeling need not necessarily be performed on the frame image displayed first, and is suitable for labeling with the above-described image skip function (image slider 39, “<< 10s” button 32a, etc.) (Similar to one image clip) may be selected.

  First, labeling that is the basis of image recognition and analysis processing will be described.

  Labeling is performed by the user. Specifically, labeling is performed by selecting an image clip in which the object 40 is photographed and pressing a “register specified label” button 29. There is no particular rule for labeling, and any number of labels may be attached to any image clip in any frame.

  In this embodiment, each time an image clip is clicked, the color of the frame (grid 23) of the image clip changes repeatedly from red → gray → red → gray → red. Here, when the “register specified label” button 29 is clicked in the state of a red frame, the target label is registered as a positive example (target) for the image clip. Conversely, when the “register specified label” button 29 is clicked in a gray frame state, a non-target label is registered as a negative example (non-target) for the image clip.

  In other words, in the case of an image clip in which the object 40 is captured, the user may click the “Register designated label” button 29 in a red frame. If the target object 40 is an image clip that has not been shot, the “register specified label” button 29 may be clicked with the frame in gray. Note that the interactive image monitoring method of the present invention performs the tagging process based on the minimum labeled data specified by the user, and performs the tagging process on the unlabeled image clip. Therefore, it is not necessary to register labels for all the image clips in the frame, and it is only necessary to do so.

  Further, in the present embodiment, when the object 40 is not shown in any image clip on the displayed frame screen, the “register specified label” button 29 is clicked without selecting any image clip. Thus, for all image clips in the frame, a non-target label can be labeled.

  The labeling method by the user is not particularly limited. For example, two registration buttons “register as positive example” and “register as negative example” are provided, and an image clip is selected and any registration button is selected. Label registration may be performed by selecting.

  In this way, the designated label is registered in the LSH database 15 and becomes reference data for the tagging process. Hereinafter, label registration by LSH will be described.

  In conventional IML search, a kind of decision tree (DT) is used to achieve high-speed feedback. However, the decision tree is a method for searching for a good branch point by looking at the entire case data, and therefore, when used for interactive labeling, there is a problem that the tree becomes unbalanced and the speed decreases. To maintain a high registration speed, the tree must be reconstructed, which takes time.

  On the other hand, the nearest neighbor search (NN; Nearest Neighbor) that directly uses the similarity between data is suitable for sequential label addition because there is no need to search for a branch point. However, there is a drawback that the search time generally increases in proportion to the number of cases. In recent years, approximate nearest neighbor (ANN) has been proposed as a technique for shortening the search time. ANN is not perfect, but it enables NNs with a high probability to search for search time while maintaining high search accuracy. In the conventional approximate nearest neighbor search, tree-type search techniques such as kd-tree are often used. However, the tree-type search method has a problem that it takes time to construct a tree as the search target data increases, and a quick search cannot be performed.

  Locality-sensitive hashing (LSH) has been proposed as a highly versatile method for realizing this ANN at high speed. LSH has been experimentally shown to be 40 times faster than typical high-dimensional data kd-trees, and is one of the representative methods of nearest neighbor search.

  In the interactive image monitoring method of the present invention, a label specified by a user is stored using LSH and used for data recognition (tagging). Thus, when the user labels an arbitrary portion of the video, the label information is immediately reflected in the database. Since video data can be recognized on the spot using all labels specified by the user, quick feedback is possible. In this embodiment, LSH (p-LSH) using a p-stable distribution is used in order to realize ANN at a general “Euclidean distance” in image similarity determination, but other LSHs are used. It may be used.

  The following is a brief description of p-LSH. twentieth annual symposium on Computational geometry, pp.253-262, 2004).

  In p-LSH, first, data to be handled (in this specification, an image feature amount) is a d-dimensional real vector v, and this d-dimensional data is mapped to k dimensions (where k <d).

  For this purpose, d independent values according to the p stable distribution (2 stable distributions in the case of Euclidean distance, ie, normal distribution) are prepared, and k d-dimensional vectors a each of which are elements are created.

Further, a k-dimensional integer vector g having h _{a, b} (v) as elements is generated using the function expressed by Equation 1. As a result, the d-dimensional vector v is mapped to a k-dimensional integer vector.
Here, b is a real number parameter in the range of [0, ω].

Here, when there is a vector v1, v2, difference post-mapping (a · v1-a · v2 ) are distributed in ‖v1-v2‖ _p × X. Incidentally, ‖V‖ _p is p- norm, X is p stable distribution. As a result, if v1 and v2 are within r, the same g can be obtained with high probability.

p-LSH prepares L sets of k pieces of a, generates L pieces of g by inner product calculation with each, and stores them in separate tables (buckets). That is, L k-dimensional integer vectors are generated from a certain vector v, and each is stored in a bucket. That is, it can be said that L mapping spaces are prepared and v is mapped to each space. As a result, even if the probability that each neighborhood is found is p (c), if all L buckets are searched, it can be found with a probability of 1- (1-p (c)) ^L. The side search can be obtained with high accuracy.

  Here, if the number of L is increased, the search accuracy is improved, but the search takes time. Also, the value of the dimension number k of the mapping space similarly affects the accuracy and time. If k is increased, the search time decreases, but the calculation time of the inner product increases, and the search accuracy decreases for the same r. Thus, the values of L and k should be selected taking into account the required accuracy and time limitations. This is the end of the description of LSH.

  As described above, the user needs to label some data first, but the interactive image monitoring method of the present invention minimizes the burden on the user, and further, by interactive processing, Minimize the time required for labeling by allowing users to label while considering how the current labeling is reflected in the analysis results, that is, considering effective labeling It becomes possible.

  The processes after S4 are different depending on the selected playback mode. Next, tagging processing will be described.

  FIG. 3 is a schematic diagram showing the tagging process. Here, it is assumed that the rectangular frame 41 indicates a space having the image feature amount as an axis. Data is identified by this feature amount. If it is close in this space, it means that the feature amount is similar, in other words, that the image is similar. In the present embodiment, the image feature amount is given as one d-dimensional vector for each image clip of each frame. Hereinafter, tagging by the tagging process is performed based on this image feature amount.

  FIG. 4 shows an example of an image feature amount designation interface screen. In the present embodiment, it is possible to selectively set a reference as an image feature amount in advance. As the feature amount 47, either general reduction (scaling) or a histogram can be selected.

The color system 48 can be selected from RGB, HSV, grayscale, CIE Yxy, CIE L ^* a ^* b ^* , and the reduction method 49 can be a nearest neighbor method, a bilinear interpolation method, a bilinear interpolation method, Either a cubic interpolation method or an averaging algorithm can be selected. In addition, since all algorithms are well-known algorithms, description is abbreviate | omitted. Further, the dimension number d of the feature data is input to the data dimension 50. Increasing the number of dimensions makes it possible to determine similarity considering fine features, but the tagging processing time increases, and decreasing the number of dimensions makes the similarity determination rough, but the processing time is Shorter. For this reason, the number of dimensions may be set in accordance with constraints such as required accuracy and processing time restrictions. The reference that can be used as the image feature amount is not limited to the above-described example.

  In the following, an image feature amount calculation method will be described by taking as an example the case of using RGB three-dimensional histogram features as the color specification. Assuming that the color quantization number of each channel is n, the three-dimensional histogram is a histogram having a value of n × n × n. For example, when n = 4, a 64-dimensional vector of 4 × 4 × 4 = 64 is obtained.

  Next, the maximum possible color value of each channel is set to maxR, maxG, maxB (0 to 255), and the color value of a certain pixel in the image clip is set to (R, G, B). R '= (maxR + 1) / n, g' = (maxG + 1) / n, b '= (maxR + 1) / n, r = floor (R / r'), g = floor ( If G / g ′) and b = floor (B / b ′), r, g and b are integers from 0 to n.

  The above calculation is performed for all the pixels in the image clip, and the number of pixels is counted for each different (r, g, b). k = r × n × n + g × n + b, the kth element is the total number of pixels of (r, g, b), and the total number of pixels is arranged as a vector expressing the histogram. Note that the total number of pixels is 0 when there is no corresponding color value pixel in the image clip.

  The calculation of the image feature amount will be described by taking as an example the case where a 5 × 5 dimensional scaling feature by the RGB bilinear interpolation method is used for the color specification. The scaling feature is to divide an image into grid-like blocks, calculate the representative value of each block by the method specified by the reduction method 49, and use the representative value as each element of the vector. The bilinear interpolation method is a method in which a pixel at the time of reduction is calculated by linear interpolation from the values of four pixels surrounding the coordinates in the image before reduction.

Specifically, the image I ₁ of 32 × 32 pixels of the image before reduction is, to convert into 5 × 5-dimensional scaling feature I _2, since the reduction of the 5/32, in the I ₂ (1 , 1) coordinate values i1 (1, 1) is, i2 (32 / 5,32 / 5 with I _2), that is, i2 (6.4, 6.4). Since the coordinates of I ₂ (6.4, 6.4) have no value, i2 (6, 6), i2 (6, 7), i2 (7, 6), i2 (7, 7) are linearly complemented, and i1 ( The value is 1,1). Similarly, image feature values are calculated for each of RGB values for all pixels.

  FIG. 3A is a schematic diagram when labeling is performed by the user. Data labeled that the white circle 42 should be recognized (hereinafter, target label data 42) and the black circle 43 should not be recognized. Is labeled (hereinafter, non-target label data 43). In FIG. 3, for the sake of simplicity, two dimensions are used in the vertical and horizontal directions. In this embodiment, several tens to several hundreds of high-dimensional spaces are used.

  FIG. 3B is a schematic diagram showing a state in which tagging is performed by the tagging process of the present invention in accordance with the labeling. In the tagging process, tags are attached to data within the proximity distance threshold r from the target label data 42 and the non-target label data 43 that are labeled data. That is, target tag data 44 is tagged to data within a certain distance r from the target label data 42, and non-target tag data 45 is tagged to data within a certain distance r from the non-target label data 43. . The neighborhood distance threshold r is an arbitrary parameter determined based on the reference value shown in the “inspection distance” 28 (see FIG. 2).

  An example of a method for setting the neighborhood distance threshold r will be described. When the “inspect this frame” button 27 is pressed and the tagging process is performed, the image clip of the displayed frame image is compared with the already-registered labeled data. The comparison target is registered labeled data that is within the neighborhood distance threshold r with respect to the d-dimensional vector representing the image feature amount of the image clip. Here, if the distance to the nearest data in the comparison target is D, after calculating the distance D to the nearest data for all image clips in the screen, the maximum value is set to maxD. In the “inspection distance” 28, maxD is displayed. If there is no data within the neighborhood distance threshold r, D = 9999.

  The numerical value displayed in the “inspection distance” 28 can be a reference value for setting the neighborhood distance threshold r. For example, if there is an image that is clearly similar to the data that has already been registered in the screen and the tag estimation has failed, it means that the neighborhood distance threshold r is set too small. In such a case, the threshold r is increased to deal with it. However, if the threshold r is increased too much, it is erroneously estimated that even similar data is similar. Therefore, a value slightly larger than the inspection distance is set as the neighborhood distance threshold r in a state where the similar image can be correctly estimated.

  An object of the interactive image monitoring method of the present invention is to enable rapid processing by minimizing the amount of labeling by the user. Therefore, the labeled data is only a part of the video as a whole. For this reason, when the target tag data 44 and the non-target tag data 45 are forcibly estimated by increasing the value of r from a small amount of labeled data 42 and 43, the estimation accuracy deteriorates. Not only will there be many misjudgments, but the user will not be able to determine how labeling affects the analysis results.

  Therefore, in the interactive image monitoring method of the present invention, as shown in FIG. 3 (b), the unknown tag data 46 is data that cannot be the target tag data 44 or the non-target tag data 45.

  In the present embodiment, as described above, the grid 23 of the image clip with the target label is displayed in red, and the grid 23 of the image clip with the non-target label is displayed in gray. Here, when the tagging process is executed, any one of the target tag 51, the non-target tag 52, and the unknown tag 53 is tagged for the image clip that is not labeled. Is displayed in orange, the image clip grid 23 with a non-target tag 52 in blue, and the image clip grid 23 with an unknown tag 53 in white. .

  Further, when the user clicks the “batch inspection” button 33 after performing some labeling, the tagging process is continuously started for the frame images after the frame currently displayed on the screen, Unless stop is instructed in the middle, tagging processing is performed up to the final frame of the moving image (batch processing). The tagging speed is affected by the number of dimensions of the feature data, the various parameters of LSH, the number of image clips divided in the screen, and the like.

  In the interactive moving image monitoring method of the present invention, even when the batch processing is in progress, the batch is generated by clicking the “frame advance” button 26 or the “normal playback” button 31 in response to an instruction from the user during the batch processing. The process is stopped, and the user can add a new label or modify a label that has already been added according to the analysis state in the visualization region 37 displayed in parallel at the same time. Further, when the tagging process is started again after the label is added / modified, tagging reflecting the newly added label is performed for the subsequent frame images.

  The status of this tagging process is visualized and displayed on the visualization area 37 as shown in FIG.

  FIG. 5A shows an example of a frame image of a certain moving image, and FIG. 5B shows an example of an enlarged view of the visualization region 37 displayed when the automatic tagging process is performed on the video.

  The horizontal axis of the visualization region 37 is a time axis, the left end represents information of the first frame, moves to the right along with the transition of time in the video, and the rightmost end represents the last frame. The pattern represents the tag of the image clip in each frame, and is the same as the frame color in the grid 23 described above. In the present embodiment, the orange frame (light gray in the figure) represents the target tag 51, the blue frame (dark gray in the figure) represents the non-target tag 52, and the white frame represents the unknown tag 53.

  This visualization result represents how the tag in the screen changes with time. The white area indicating the unknown tag 53 means a portion where neither the target 51 nor the non-target 52 can be tagged in the image information labeled by the user so far.

  For example, in FIG. 5B, it can be seen that the uppermost stage is all blue (dark gray in the figure), and no target (object 40) appears at the upper stage of the screen. In the middle row, an orange (light gray in the figure) band appears, and it can be seen that the target sometimes appears in the middle row.

  Here, unlike the tagging of still images, the tag for the entire video is not an amount that can be displayed on one screen even though the bitmap display has a high resolution. That is, in the case of a long-time video, the total number of frames is much larger than the number of pixels in the application visualization area, so one pixel must display information of a plurality of frames. That is, one point on the visualization region 37 is a display region where a large number of spatial and temporal tags overlap.

  Even if the resolution is high, if the display becomes dense, it will be beyond the human eyesight limit. Of course, providing a zoom function or making the visualization map scrollable can alleviate the problem, but the requirement to view a wide area at once and zooming to enlarge the part are not compatible.

  Therefore, in order to display a list of tagging results within a limited range of the number of pixels, it is necessary to display information on a plurality of tags at the same place. In this embodiment, importance is assigned to tags, and important information is preferentially displayed.

In this embodiment, the importance of the tag is set to the following importance.
Importance / High: Unknown Tag Importance / Medium: Target Tag Importance / Low: Non-target Tag “Unknown Tag” of importance / high indicates what points should be used to teach cases. In order to enable tracking of an object with high accuracy with a minimum number of teachings, it should be displayed most preferentially. In addition, the “target tag” of importance / medium represents unknown data that is very similar to the small number of target labels specified by the user, so it helps to check the progress of labeling work when correctly guessing. If it appears in a place that is not expected by the user, it is important in that it indicates the possibility of a false guess. Furthermore, “non-target tags” are data that closely resembles a large amount of non-targets specified by the user, and unless the user labels, there is little need to see, so the least important .

  Moreover, the above importance ranking is optimal for the following reasons. For example, giving priority to “target tag” over “unknown tag” leads to overlooking a portion that has not yet been tagged. Similarly, if “non-target tag” is given priority over “target tag”, an erroneous estimation of tagging is overlooked. Therefore, by performing the visualization process based on the above-described importance, it is possible to reduce the possibility of overlooking information necessary for the labeling operation.

  The importance level of the tag is not limited to the above example. For example, the importance may be displayed more finely according to the unknown degree of the unknown tag. Here, the degree of unknown is set based on the distance to the nearest target label data 42 and non-target label data 43, the number of target label data 42 and non-target label data 43 within the neighborhood distance threshold r, and the like. be able to. For example, the degree of unknown may be set in descending order of the distance to the nearest target label data 42 and non-target label data 43, and may be displayed preferentially in descending order of the degree of unknown. In this case, the unknown tag may be displayed with finer color coding depending on the degree of unknown.

  Next, the visualization process will be described with reference to FIG. FIG. 6A shows an image image of the i-th frame when the total number of frames of moving images to be handled is F.

The horizontal grid number is Xg, the vertical grid number is Yg, and the grid position of the image grip to be visualized is (xg, yg). In this case, the corresponding region R on the visualization region 37 corresponding to the image grip can be obtained by Expression 2. The upper left coordinates of the region R are indicated by (xv, yv).
<Equation 2>
xv = Xv × (i / F)
w = Xv / F However, if w <1, w = 1. w indicates the width of the region R.

Also, as shown in FIG. 2, in the interface of the interactive image monitoring program of this embodiment, the “visualization compression method” radio button 38 is used to determine whether the projection direction is in the horizontal axis direction or in the vertical axis direction. Can be selected. This is because, as described above, the number of tags attached to a moving image is enormous, and it is impossible to display all the tags on the screen. When projecting in the horizontal axis direction, the vertical width h of the corresponding region R can be obtained by Formula 3 and when projecting in the vertical axis direction, Formula 4. Xv represents the number of horizontal pixels in the visualization region 37, and Yv represents the number of vertical pixels in the visualization region 37.
<Equation 3>
yv = Yv × (yg / Yg)
h = Yv / Yg However, if h <1, h = 1.
<Equation 4>
yv = Yv × (yg / Xg)
h = Yv / Xg However, if h <1, h = 1.

  Furthermore, the case where it projects in a horizontal-axis direction is demonstrated using FIG. As described above, the number of horizontal pixels in the visualization region 37 is limited. When the number of frames exceeds the number of pixels, it is necessary to display several frames of image information in one pixel column. Here, it will be described how k + 1 frame images (j-th row) of frame images (i to i + k) are compressed into one pixel column and displayed.

  First, paying attention to the j-th row of the frame image (i to i + k), the importance of the image clip in the j-th row is compared. In the present embodiment, the above-mentioned items having the highest importance are represented. Here, since the unknown tag 53 exists, the corresponding region of the visualization region 37 is white indicating the unknown tag 53.

  Further, when there is no unknown tag 53 in the corresponding line and there is even one target tag 51, when there is only a non-target tag 52 in orange indicating the target tag 51, the non-target tag 52 is set. The visualization region 37 is mapped to the blue color shown. In addition, when projecting in the vertical axis direction, the processing may be performed with the j-th row as the j-th column.

  In this way, important information is gathered and displayed in the limited visualization region 37, so that user labeling is supported, and thus high-precision image monitoring with less teaching is supported.

  If the visualization result of the tag in the visualization region 37 is, for example, the target object 40 that does not move in the moving image, an orange line indicating the target tag appears in a straight line in the visualization region (Example 1, FIG. 15). reference).

  On the other hand, if the object 40 moving in the moving image is monitored, the object 40 can be tracked by the locus of the orange line appearing in the visualization region (see Example 4, FIG. 22). . In this case, the direction of projection may be selected depending on whether the movement of the object 40 often moves in the horizontal direction of the screen or the vertical direction of the screen.

  In addition, it is rarely known in advance that the location and the generation time to be detected in the image monitoring, and the video to be detected is a very small part of the whole. For example, in the case of an image taken for examining the nighttime discharge of the lion, most of the images are dark nighttime lion images, and it is difficult to teach the discharge image in advance (see Example 3). ). That is, there are cases where it is difficult to teach a target.

  Even if it is difficult to teach a target in this way, according to the interactive video monitoring method of the present invention, if a non-target is taught by a user, an unknown tag is applied to a different target. In addition, since it can be extracted from the video, it is not always necessary to teach the target in advance.

  That is, the user can narrow down the target by checking the image with the unknown tag, and can reliably teach the target without continuing to watch the entire video carefully.

  A schematic diagram of the tagging method in this case is shown in FIG. The user first labels non-targets that can be easily taught in the video. For example, in the case of a discharge image, an image in a normal state where no discharge is generated is taught (FIG. 8A).

  When the tagging process is performed in that state, a non-target tag 52 is automatically attached to an image similar to a taught non-target (an image without discharge), and an unknown tag is attached to all other images (see FIG. 8 (b)). In other words, since the non-target tag 52 is attached if it remains in a completely dark state, there is a possibility that some kind of phenomenon has occurred in the data to which the unknown tag 53 is attached.

  Therefore, the unknown tag data 46 is confirmed by an image, and if there is no problem, a non-target label is attached, and if the target (in this case, discharge) is reflected, a target label is attached (FIG. 8C).

  Then, when tagging processing is performed, further narrowed tagging is performed. Thereafter, the process is repeated until the tag estimation is sufficient (FIG. 8D).

  The tagging method described above is a more efficient method as the proportion of the normal state in the entire video is larger and the occurrence frequency of the phenomenon to be detected is lower, and is a method suitable for monitoring video.

  In addition, as in the above-described example, the neighborhood distance threshold r is set for a moving image that is difficult to teach a target at the start of processing, for example, as follows.

  An image clip with no label specified and no target is set to the minimum distance that the computer can tag estimate as a non-target. At this time, if the distance r is too small, the tag estimation result is only unknown (white). Conversely, if the distance is too large, the target that should be detected is missed.

  Therefore, by setting the minimum distance at which all image clips with no target clearly become blue (non-target tags), that is, the distance at which the image clips change from blue to white (unknown tags) if the distance is further reduced, accurate tags are set. Estimation can be performed.

  Hereinafter, image recognition and analysis processing performed by the interactive moving image monitoring program of the present invention will be described with reference to flowcharts shown in FIGS.

  The contents of the image recognition and analysis processing differ depending on which playback mode is selected.

  When the reproduction mode is the “batch process” mode (S4; Yes), the batch process (S5) is performed.

  The batch process (S5) will be described with reference to the flowchart of FIG.

  First, the leftmost and uppermost image clips in the grid are processed (S501).

  Next, the image clip to be processed is converted into a d-dimensional real vector v (image feature amount) by a method designated in advance by the user (feature amount 47, color system 48, reduction method 49, data dimension number 50). (S502) A question using v as a key is made to the LSH database 15 to search for data similar to v (S503).

  When the distance between the search result data (image feature amount) and v given as a question is larger than the pre-designated neighborhood distance threshold r, the tag of the image clip is set as an “unknown tag” and smaller than r If the label of the nearest data is “target label”, the tag of the image clip is set as “target tag”, and if the label is “non-target label”, it is set as “non-target tag” (S504). .

  The tag of the image clip is registered in the ICP database (S505). A color frame corresponding to the tag of the image clip is displayed at the position of the image clip of the video image (S506).

  The control unit of the visualization area 37 is notified of the frame number, the position in the grid, and the tag, and the visualization result is updated (S507).

  FIG. 11 shows a flowchart detailing the processing of S507. The corresponding region R on the visualization region is calculated from the frame number and the position in the grid (S507-1).

  Next, the importance of the tag Tregist registered in the data map corresponding to the corresponding region R on the visualization region is compared with the importance of the tag Tnew of the image clip (S507-2).

  Next, when the playback mode is “batch processing” and after changing from another mode to “batch processing”, it is the first time to access the region R (S507-3; Yes). The process moves to S507-5. In other cases (S507-3; No), the process proceeds to S507-4. When the tag importance of Tnew is greater than the tag importance of Tregist (S507-4; Yes), the process proceeds to S507-5. On the other hand, when the tag importance of Tnew is the same as or smaller than the tag importance of Tregist, the process of S507 ends.

  In S507-5, Tnew is registered in the data map 17 of the visualization area, and the corresponding area R of the visualization area 37 is displayed in a color corresponding to Tnew, and the process of S507 ends.

  Returning to the flowchart of FIG. It is determined whether or not an unprocessed image clip exists in the grid. If it exists (S508; Yes), the unprocessed image clip is set as a processing target (S509), and the process returns to S502. When the processing is completed for all the image clips (S508; No), the batch processing ends, and the process proceeds to S8.

  Next, when the reproduction mode is “normal reproduction” (S6; Yes), normal reproduction processing (S7) is performed. The normal reproduction process (S7) will be described using the flowchart of FIG.

  The leftmost and uppermost image clip in the grid is set as the processing target (S701), and the registered label or tag is searched from the ICP database 16 using the frame number of the processing target image clip and the position in the grid as keys (S701). S702).

  A color frame corresponding to the tag or label of the image clip is displayed as the color of the frame of the image clip (S703).

  The control unit of the visualization area is notified of the frame number, the position in the grid, and the tag, and processing for updating the visualization result is performed (S704). Note that the processing in S704 is the same as the processing in S507 described above (see FIG. 11), and thus the description thereof is omitted.

  It is determined whether or not an unprocessed image clip exists in the grid. If it exists (S705; Yes), the unprocessed image clip is set as a processing target (S706), and the process returns to S702. When the processing is completed for all the image clips (S705; No), the normal reproduction processing is terminated, and the process proceeds to S8.

  Next, when the playback mode is “frame advance” (S8; Yes), a frame advance process (S9) is performed. The frame advance process (S9) will be described with reference to the flowchart of FIG.

  First, when the position (xg, xy) in the grid is selected (S901; Yes), the tag or label of the image clip is obtained by searching the ICP database, and the obtained tag or label is Tp. (S902).

  If Tp is one of an unknown tag, non-target tag, or non-target label (S903; Yes), the label of the image clip is changed to “target”, registered in the ICP database 16, and the grid frame color is targeted (S904), and the process returns to S901.

  On the other hand, if Tp is a target tag or target label (S905), the label of the image clip is changed to “non-target”, registered in the ICP database 16, and the color of the grid frame is changed to a color corresponding to the non-target. Change (S906) and return to S901.

  When the position (xg, xy) in the grid is not selected (S901; No), it is determined whether or not the “inspect this frame” button 27 has been clicked (S907). When the “inspect this frame” button 27 is clicked (S907; Yes), the batch processing (S5) is performed, and the process returns to S901.

  If the “inspect this frame” button 27 has not been clicked (S907; No), it is determined whether or not the “register specified label” button 29 has been clicked (S908), and if it has been clicked (S908; Yes). All the labels updated during the “frame advance processing” are registered in the LSH database 15 (S909), the batch processing (S5) is performed, and the process returns to S901.

  If the “register specified label” button 29 has not been clicked (S908; No), it is determined whether or not the “frame advance” button 26 has been clicked (S910). If it has been clicked (S910; Yes), The frame advance process (S9) ends.

  If the “frame advance” button 26 has not been clicked (S910; No), it is determined whether the “batch inspection” button 33 has been clicked (S911). If it has been clicked (S911; Yes), the playback mode is selected. Is changed to “batch processing” (S912), and the frame advance processing (S9) ends.

  If the “batch inspection” button 33 has not been clicked (S911; No), it is determined whether or not the “normal playback” button 31 has been clicked (S913). If it has been clicked (S913; Yes), the playback mode. Is changed to “normal playback” (S914), and the frame advance processing (S9) is ended. On the other hand, when it is not clicked (S913; No), the process returns to S901.

  Returning to the flowchart of FIG. When any one of S5, S7, and S9 is completed, the target frame number i is updated to i + 1 (S10).

  At this time, if movement to the frame number i ′ is instructed by the image slider 39, the “<< 10s” button 32a, etc., the target frame number i is changed to i ′ (S11).

  Finally, it is determined whether or not the system termination is instructed (S12). This completes the processing executed by the interactive moving image monitoring program of the present invention.

  The above-described embodiment is a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the gist of the present invention. Moreover, the above arithmetic expression is an example, and various modifications can be made without departing from the scope of the present invention.

  For example, in the present embodiment, LSH is used as an information search method, but other methods such as other learning devices such as a classification tree and SVM may be used. In that case, instead of the LSH database 15, a corresponding database may be configured and processed.

  Further, the visualization area 37 is not necessarily compressed and displayed on one screen. In this case, the visualization area 37 may be scrolled by a scroll bar. Alternatively, the state before compression may be displayed in a zoomed-in display by compressing and selecting the corresponding part.

Example 1
Using the interactive moving image monitoring program of the present invention, an application experiment to an artificial test video simulating lighting fluctuation was conducted.

  In this experiment, as shown in FIG. 14, a moving image (experimental video 1) obtained by photographing a logo 40 printed on one sheet of paper was used. The target 40 is a logo 40. Since the image was taken with the paper fixed in the center, the position of the logo 40 does not change. Although there is no particular difficulty in detecting the logo 40, the purpose of this experiment was to determine whether or not the interactive moving image monitoring program of the present invention can cope with illumination fluctuations.

The illumination in this experiment was performed under the following three conditions.
(1) Artificial lighting with fluorescent lamps (2) State in which the indoor blind is turned off and the light is turned off (3) State in which the indoor blind is opened and the light is turned off Shooting was performed while varying the above. Experimental video 1 was 47 seconds (1,424 frames).

  In this experiment, logo recognition was performed by the three subjects using the interactive video monitoring program of the present invention using the video.

  The subject watched the experimental video 1, selected a specific frame, labeled the minimum image clip, and then executed the interactive video monitoring program of the present invention. As a result, all other unlabeled image clips are tagged, and the monitoring result is displayed in the visualization area 37.

  Further, the test subject repeatedly performed the process of re-labeling the necessary image clip of the necessary frame by viewing the monitoring result and executing the program again. In this experiment, the process was terminated when the subject determined that accurate monitoring was possible, and the experiment was terminated.

In this experiment, the index shown in Table 2 was used to determine the learning level of the computer, that is, the image monitoring accuracy.

  The degree of learning can be determined by what kind of tagging is performed on a label that should be attached (hereinafter, correct label). For example, the number of target tags attached to correct labels.

  In Table 2, a large amount of TP and TN is desired, and a large amount of FN and FP means that the tag estimation accuracy is low. In addition, FU is an overlooked part that makes it unclear where it should be targeted, and its purpose is to reduce it.

  In this experiment, the following parameters were used for the image feature amount and LSH. For the image feature amount, the color system is RGB, and a 5 × 5 dimensional scaling feature by a bilinear interpolation method is used. The LSH parameters were L = 20, k = 10, and ω = 0.4. This parameter setting is a setting value for correctly searching data within the distance r with a probability of 90% and erroneously detecting data outside the distance r with a probability of 5%. Also, the neighborhood distance threshold r = 0.15.

The experimental results are shown in FIG. (A) is a recall (recall) indicating whether or not a target can be reliably detected as a target in the video, and (b) is a non-notice indicating whether or not a non-target can be reliably detected as a non-target. It is a graph which shows how the target detection rate (TNR) changed according to the registration number of labels. The recall (recall) is expressed by Equation 5 and the non-target detection rate (TNR) is expressed by Equation 6, meaning that the closer to 1, the higher the accuracy.
<Equation 5>
recall = TP / (TP + FN + FU)
<Equation 6>
TNR = TN / (TN + FP + TU)

  In addition, for comparison with the subject, selection of a frame was performed at random, and a labeling operation (hereinafter referred to as random selection) was performed in which the labeling in the frame was accurately performed. In the moving image in this experiment, the logo 40 as the object does not move at the center of the screen, and therefore, if a frame in which the logo 40 appears is given, a random selection can be automatically made. By comparing this random selection with the recall rate (recall) and non-target detection rate (TNR) of the subject, it can be confirmed whether or not the subject was able to select the frame efficiently. That is, if the result of the random selection is equal to the result of the subject, it can be said that the subject has randomly selected the frame. If the result of the subject is better than the result of the random selection, the subject efficiently selects the frame. It can be said that.

As shown in FIG. 15, it can be seen that the results of the three subjects approached 1 at an early stage as compared with the random selection. Since FP and FN are almost 0 in both examples, the precision (precision) shown in Equation 7 and the false detection rate (FPR) shown in Equation 8 were accuracy 1 and error detection rate 0. .
<Equation 7>
precision = TP / (TP + FP)
<Equation 8>
FPR = FP / (FP + FU + TN)

  Also, FIG. 16 shows how the visualization region 37 changes as the number of label registrations increases. FIG. 16 shows how the visualization region 37 changes as the number of registered labels increases from above.

  In the initial stage with a small number of registered labels, there are many white areas (unknown tags) 61, indicating that the object cannot be monitored. However, as the labeling progresses, the number of unknown tags decreases, and orange (in the figure) In this case, a gray line 62 appears. In this experiment, since the logo 40 of the object is at the center of the screen, if the orange line 62 appears in the center of the visualization region 37 in this experiment projected in the horizontal axis direction, the logo 40 has been successfully tracked. It shows that.

  From this experiment, it was confirmed that it was possible to cope with changes in the color of the object due to illumination fluctuations.

(Example 2)
Similarly, the experiment was performed using the image shown in FIG. Unless otherwise specified, the experiment is performed under the same conditions as in Example 1.

  In this experiment, the can 63 in the image was rotated and the logo 40 attached to the side surface was tracked. That is, it is a moving image (experimental video 2) in which the logo 40 repeatedly appears and disappears as the can 63 rotates. Experimental video 2 was 120 seconds (3,596 frames).

  FIG. 17 shows the result of having three subjects execute the interactive image monitoring program of the present invention in the same manner as in the first embodiment.

  Even in the experimental video 2, FP and FN are extremely small values, and accuracy ≈ 1 and false detection rate ≈ 0. Since there was no illumination fluctuation in Test Video 2, there was no background fluctuation, and the background could be excluded by first registering the background as a non-target label for several frames.

  Moreover, in the experimental video 2, the can 63 repeatedly rotated four times, and the position thereof is also the same, so that the remaining video can be appropriately tagged by appropriately labeling for one rotation. . Therefore, compared with the experiment video 1, a high reproduction rate can be achieved with a small number of label registrations.

(Example 3)
In this experiment, the leakage current of the insulator was monitored. The video used in the experiment (Experiment Video 3) is a part of a long-term image taken over several months of the DC insulators installed at the test site as an exposure test to prevent discharge noise from DC transmission lines. .

  The total playback time of the experimental video 3 is 48 minutes and 42 seconds, the total number of frames is 87,575 frames, and it is a night video in which discharge can be confirmed. Incidentally, FIG. 18 shows an example of the daytime shooting of Choshiren. The target in the video is the insulator discharge phenomenon, and it is necessary to accurately detect the time and frequency of the discharge from the video.

  In the experimental video 3, there is no discharge during most of the video, and a monotonous screen without change continues. In addition, the discharge time for one time is extremely short (within 33 msec). For this reason, for example, it is considered that the inspector maintains a caution and continues to look while searching for a discharge point, and it is considered that there are many oversights.

  As shown in FIG. 19, a grid 23 having two horizontal columns and 20 vertical columns is set for the rightmost column of lions. Since it is night, the screen is black.

  In this experiment, the image feature amount is gray scale (luminance is a real number of [0,1]), and 4 × 4 dimensional scaling feature by averaging. The LSH parameters were the same as those in Examples 1 and 2, and the neighborhood distance threshold r = 0.24.

  In this experiment, first, non-target labels were attached to all 40 image clips of the first frame image in which no discharge was reflected. And the visualization area | region 37 when a tagging process is performed only in this labeling state is shown to Fig.20 (a). FIG. 20 shows a visualization region 37 obtained by projecting in the horizontal direction of FIG. In addition, tagging of the 48-minute video of the experimental video 3 took 32 minutes with a Pentium (registered trademark) 4 3.6 GHz computer.

  In this experiment, since the number of horizontal pixels in the visualization region 37 is 720, 122 frames and 244 pieces of tag information are aggregated in one horizontal pixel (= 87,575 × 2/720).

  In FIG. 20A, the blue color representing the non-target tag (dark gray in the figure) occupies most, and a white region representing the unknown tag is seen in some places. That is, most of the images were images without discharge. The white area indicates that something that is not similar to the non-target may have been shot. The user does not need to see the entire video and only needs to inspect the image with the white unknown tag. Since no target label is registered, no target tag shown in orange is seen.

  When some of the unknown tags were confirmed by video, it was found that any of the unknown tag collections indicated by reference numeral 65 in FIG. 20A was video tape noise. The noise was caused by scratches on the videotape and the head of the video deck.

  On the other hand, it was found that the portion indicated by a series 66 of unknown tags that appeared from the middle in the lowermost stage and continued to the vicinity of the end of the video captured the discharge phenomenon.

  Therefore, while confirming the video, an operation was performed to attach a non-target label to the video noise and a target label to the discharge part. FIG. 20B shows a tagging state after attaching 297 labels, and FIG. 20C shows a tagging state after labeling up to 266 frames and 414 labels.

  It can be seen that as the number of labels increases, the noise disappears and the target tag sequence 67 of orange (light gray in the figure) increases at the bottom.

  As described above, when the video image is analyzed by the interactive moving image monitoring program of the present invention, the user can pay attention to a representative discharge pattern in the monitoring image only by teaching a small number of non-target information. It was confirmed that the place to be able to be found properly.

  Furthermore, it was confirmed that selection of case images and label teaching work can be performed easily and reliably. According to the interactive video monitoring program of the present invention, not only the discharge image of the insulator but also the occurrence frequency is low and the work effort of the case teaching is greatly reduced even in the long-time monitoring in which it is difficult to find the teaching example to the computer. can do.

Example 4
In this experiment, radio-controlled cars were monitored and tracked. In this experiment, as shown in FIG. 21, an image (experimental image 4) in which a radio controlled car 40 travels across the floor from side to side is used as an object to be monitored. The total playback time of the experimental video 4 is 67 seconds and the total number of frames is 2,008 frames.

  In this experiment, the image feature amount is RGB 3 × 3 × 3 = 27-dimensional histogram feature, and the neighborhood distance threshold is r = 0.487844.

  The grid 23 has a fine grid pattern of 21 × 13 in the horizontal direction, the visualization compression direction 38 is set in the horizontal axis direction, and the interactive moving image monitoring program of the present invention is executed. The result is shown in FIG. In addition, only 26 labels including target tags and non-target tags were registered.

  In the visualization region 37, a line 71 having an orange inclination indicating a target on a blue back 70 indicating a non-target is displayed. An orange line 71 indicates the tracking result of the automobile 40. There are almost no unknown tags, and the tracking result is displayed in accordance with the movement of the automobile 40, indicating that the tracking is successful.

  Here, when the orange line 71 extends from the lower left to the upper right, it indicates that the automobile 40 has moved from the right to the left of the screen. When the orange line 71 extends from the upper left to the lower right, from the left of the screen. Indicates that it has moved to the right. Further, the inclination of the line 71 changes depending on the speed of the automobile 40 and the traveling course. Specifically, when the screen horizontal speed component of the automobile 40 is slow, the inclination increases, and when it is high, the inclination decreases.

  From this experiment, it has been confirmed that the movement of the monitoring object from the tagging status displayed in the visualization region 37, that is, the tracking of the monitoring object can be realized by teaching by a very small number of users.

It is a schematic block diagram which shows an example of the interactive image monitoring apparatus of this invention. It is an example of the interface screen of the interactive moving image monitoring program of this invention. Moreover, it is an example of the frame image of the experiment video 2. It is a conceptual diagram of tagging processing, (a) shows target label data and non-target label data in a two-dimensional image feature amount space, (b) shows target label data and non-target labels shown in (a). A mode that target tag data, non-target tag data, and unknown tag data are tagged based on data is shown. It is an example of the interface screen of the image feature designation | designated of the interactive moving image monitoring program of this invention. (A) shows the frame image of the moving image which becomes object, (b) shows the visualization area | region displayed after performing the visualization process about the said moving image. It is a figure for demonstrating the method of calculating | requiring the applicable area | region R in a visualization area | region from a frame number and the position in a grid. It is a figure for demonstrating the processing method in the case of projecting and displaying a visualization area | region to a horizontal axis direction. It is another example of the conceptual diagram of tagging processing, (a) shows non-target label data in a two-dimensional image feature amount space, (b) is based on the non-target label data shown in (a). The non-target tag data and the unknown tag data are shown to be tagged, (c) shows the target label data being further labeled, and (d) is the target label data and non-target shown in (c). A mode that target tag data, non-target tag data, and unknown tag data are tagged based on label data is shown. It is a flowchart which shows the whole process which the interactive moving image monitoring program of this invention performs. It is a flowchart which shows the detail of the batch process which the interactive moving image monitoring program of this invention performs. It is a flowchart which shows the detail of the process of S507 and S704. It is a flowchart which shows the detail of the normal reproduction | regeneration processing which the interactive moving image monitoring program of this invention performs. It is a flowchart which shows the detail of the frame advance process which the interactive moving image monitoring program of this invention performs. It is an example of the frame image of the experiment image | video 1. It is a graph which shows the experimental result in Example 1, (a) is a graph which shows the relationship between the number of registration labels, and recall. (B) is a graph showing the relationship between the number of registered labels and the non-target detection rate. It is a figure which shows the mode of a change of the visualization area | region accompanying the increase in the label registration number in Example 1. FIG. It is a graph which shows the experimental result in Example 2, (a) is a graph which shows the relationship between the number of registration labels, and recall. (B) is a graph showing the relationship between the number of registered labels and the non-target detection rate. It is an example of the daytime photographed image of Choshiren. It is an example of the image which set the grid of 2 squares by 20 squares with respect to the insulator series. For the experiment video 3, when (a) only 40 non-target labels, (b) 297 labels of target or non-target labels, (c) and 414 labels are further tagged. , Showing the visualization area. It is an example of the frame image of the experiment image | video 4. Experimental video 4 shows a window with grid settings and a tagged visualization area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Interactive type | formula video analysis apparatus 11 Labeling means 12 Tagging means 13 Visualization means 14 Video data 24 Process target area 51 Target tag 52 Non-target tag 53 Unknown tag

Claims (11)

  In the method for monitoring and tracking a monitoring object in a moving image, one or more image clips are provided in a processing target area to be subjected to image processing in the frame image of the moving image, and the monitoring target is included in the image clip. Label registration processing for registering labeling data as to whether or not an object has been photographed, and obtaining an image feature amount based on a color information value of a pixel included in the image clip, and registering the labeling data The image feature amount of the image clip that has not been performed is in a predetermined fixed range in a high-dimensional space around each element of the image feature amount of the image clip in which the labeling data is registered For, the tagging according to the labeling data is performed on the image clip in which the labeling data is not registered. And grayed with processing, interactive video image monitoring method and performing the visualization process of displaying together with a frame image of the moving image on the basis of the results of the tagging process to the importance of the tag.   2. The interactive moving image monitoring method according to claim 1, wherein a local sensitive hash algorithm is used when registering and searching the labeling data.   The tagging process is performed in a predetermined range in a high-dimensional space around each element of the image feature amount of the image clip labeled that the monitoring object is photographed. Is for a target tag, a thing in a predetermined range in a high-dimensional space around each element of the image feature quantity of the image clip labeled that the monitoring object is not photographed. 3. The interactive moving image monitoring method according to claim 1, wherein an unknown tag is tagged to the target tag and the image clip that is not in any range.   The interactive video according to any one of claims 1 to 3, wherein the visualization process compresses and displays the result of the tagging process for all frame images of the moving image on one screen. Image monitoring method.   5. The interactive moving image monitoring method according to claim 4, wherein importance of the tag is highest for the unknown tag and lowest for the non-target tag.   An apparatus for monitoring and tracking a monitoring object in a moving image, wherein a frame image of the moving image is read out, one or more image clips are set in a processing target area to be subjected to image processing, and the image clip includes Label registration means for registering in the database labeling data designated in advance as to whether or not the monitoring object is photographed, and an image feature amount based on color information values of pixels included in the image clip for the image clip The image feature quantity of the image clip for which the labeling data is not registered in the database is high with respect to each element of the image feature quantity of the image clip for which the labeling data is registered. In the dimensional space, if it is within a predetermined range, tagging according to the labeling data is performed. Tagging means for storing in association with the unregistered image clip, and visualization means for displaying the result of the tagging process on the output device together with the frame image of the moving image based on the importance of the tag. An interactive moving image monitoring apparatus characterized by that.   7. The interactive moving image monitoring apparatus according to claim 6, wherein a local sensitive hash algorithm is used when registering and searching the labeling data.   The tagging means is in a predetermined fixed range in a high-dimensional space around each element of the image feature quantity of the image clip labeled that the monitoring object is photographed. Is for a target tag, a thing in a predetermined range in a high-dimensional space around each element of the image feature quantity of the image clip labeled that the monitoring object is not photographed. The interactive moving image monitoring apparatus according to claim 6 or 7, wherein an unknown tag is tagged for the target tag and the image clip that is not in any range.   9. The interactive moving image monitoring apparatus according to claim 6, wherein the visualizing unit compresses and displays a tagging result for all frame images of the moving image on one screen. .   The interactive moving image monitoring apparatus according to claim 9, wherein importance of the tag is highest for the unknown tag and lowest for the non-target tag.   Among the frame images of the moving image, one or more image clips are set in a processing target area to be subjected to image processing and stored in the main storage device, and the image registered in advance in the storage device Reads labeling data as to whether or not a monitoring object is photographed in the clip, obtains an image feature amount based on a color information value of a pixel included in the image clip, and registers the labeling data The image feature amount of the image clip that has not been set is within a predetermined range in a high-dimensional space that has each element of the image feature amount of the image clip in which the labeling data is registered as an axis. For example, the image clip in which the labeling data is not registered is tagged according to the labeling data, and the tag Dialogue for monitoring and tracking a monitoring object in a moving image by causing a computer to execute image recognition and analysis processing for displaying the result of the detection together with the frame image of the moving image on the output device based on the importance of the tag Type video surveillance program.