JP2016039496A - Image processing apparatus, image processing system, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of displaying a desired video quickly, and to provide an image processing system, image processing method and program.SOLUTION: A difference computing unit 308 receives a frame acquired by a frame acquisition unit 307, and computes the difference of pixel values in the area of analysis of an object frame, and the area of analysis of a comparison object frame, as the feature amount of a frame. A graph generation unit 309 generates a difference graph, i.e., a graph of difference value, from the information of difference computed by the difference computing unit 308, for each frame acquired by the frame acquisition unit 307, and displays the graph at a graph display part.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

  2. Description of the Related Art In recent years, image processing systems have been developed that analyze video data obtained by recording with an imaging device such as a video camera to detect a target object, or detect or analyze the movement or abnormality of an object on a video. It is remarkable. As such an image processing system, in past video data, a video to be displayed is divided into a plurality of blocks, the difference between the pixels of the corresponding block is calculated for each frame constituting the video data, and the difference is predetermined. There has been proposed a system for generating a detection signal on the assumption that there is a movement when the threshold value is exceeded (see Patent Document 1).

  However, since the technique described in Patent Document 1 does not display a graph for the difference between the pixels of the frames constituting the video data, for example, the time when the difference exceeds the threshold and the states before and after that There was a problem that could not be grasped. Also, even if it can detect that any of the calculated differences exceeds a predetermined threshold, the time when the difference exceeds the threshold cannot be specified, so the video location around that time or around that time is searched. There was also a problem that it was difficult.

  The present invention has been made in view of the above, and an object of the present invention is to provide an image processing apparatus, an image processing system, an image processing method, and a program capable of quickly displaying a desired video.

  In order to solve the above-described problems and achieve the object, the present invention provides a display unit, a reproduction unit that reproduces and displays video data captured by the imaging unit on the display unit, and each frame included in the video data. Calculating means for calculating a feature value of a pixel value; generating means for generating a graph representing the feature quantity in time series by the feature quantity calculated by the calculation means and displaying the graph on the display means; and the display Frame specifying means for designating any part of the display area of the graph displayed by the means and displaying the frame corresponding to the part on the display means by the reproducing means, To do.

  According to the present invention, a desired video can be quickly displayed.

FIG. 1 is a diagram illustrating an example of the overall configuration of an image processing system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example in which video data is displayed on an application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 5 is a diagram illustrating an example in which an analysis region is specified in the video display unit on the application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 6 is a diagram illustrating an example in which a difference graph is displayed on the application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 7 is a diagram illustrating an example of displaying video data at a desired time based on the difference graph on the application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 8 is a flowchart illustrating an example of an image processing operation for displaying a difference graph in the image processing system according to the embodiment.

  Hereinafter, embodiments of an image processing apparatus, an image processing system, an image processing method, and a program according to the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

(Overall configuration of image processing system)
FIG. 1 is a diagram illustrating an example of the overall configuration of an image processing system according to an embodiment. The overall configuration of the image processing system 1 of the present embodiment will be described with reference to FIG.

  As illustrated in FIG. 1, the image processing system 1 according to the present embodiment includes imaging devices 2 a to 2 d, an information processing device 3, and a network 4.

  The imaging devices 2a to 2d capture (capture) a subject by converting light from the subject into an electrical signal, and are video images that are configured with a plurality of frames (image data) (for example, 10 [FPS]). A video camera that generates data. For example, the imaging devices 2a to 2d image production facilities that produce products and generate video data for detecting and analyzing unexpected movements.

  Note that the imaging devices 2a to 2d are simply referred to as “imaging device 2” when referred to without distinction or collectively. In FIG. 1, the image processing system 1 is configured to include four imaging devices 2, but is not limited thereto, and may be configured to include other numbers of imaging devices 2. Good.

  The information processing device 3 is a PC or a workstation that functions as an image processing device that performs image processing on video data captured by the imaging device 2. Image processing on the video data by the information processing apparatus 3 will be described later.

  The network 4 is, for example, an Ethernet (registered trademark) standard network for connecting the imaging devices 2 a to 2 d and the information processing device 3. In this case, in the network 4, data communication is performed using a protocol such as TCP (Transmission Control Protocol) / IP (Internet Protocol). In this case, the imaging devices 2a to 2d and the information processing device 3 have a MAC address for communication using the TCP / IP protocol and are assigned an IP address such as a private IP address. The specific configuration of the network 4 is, for example, a star wiring format in which the imaging devices 2a to 2d and the information processing device 3 are connected to a switching hub having a plurality of ports by a LAN (Local Area Network) cable.

  Note that the video data captured by the imaging devices 2a to 2d does not necessarily need to be stored in the information processing device 3. For example, a database server may be connected to the network 4 and video data captured by the imaging devices 2 a to 2 d may be stored in the database server via the network 4. In this case, the information processing device 3 may acquire target video data from the database server and perform image processing on the acquired video data.

  The network 4 shown in FIG. 1 is an example in which communication is performed by TCP / IP, but is not limited to this. For example, a VGA (Video Graphics Array) terminal or a USB (Universal) is provided on the information processing apparatus 3 side. (Serial Bus) ports may be provided, and a plurality of imaging devices 2 may be connected to the information processing device 3 with VGA cables or USB cables, respectively.

(Hardware configuration of information processing device)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. A hardware configuration of the information processing apparatus 3 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 2, the information processing apparatus 3 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an external storage device 104, a display 105, A network I / F 106, a keyboard 107, a mouse 108, and a DVD (Digital Versatile Disc) drive 109 are provided.

  The CPU 101 is a device that controls the overall operation of the information processing apparatus 3. The ROM 102 is a nonvolatile storage device that stores a program for the information processing device 3. The RAM 103 is a volatile storage device used as a work area for the CPU 101.

  The external storage device 104 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores video data captured by the imaging device 2 and various data such as setting information.

  The display 105 is a display device that displays various information such as a cursor, a menu, a window, characters or images, or a screen of an application that executes image processing by the information processing device 3. The display 105 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an organic EL (Electroluminescence) display, or the like. For example, the display 105 is connected to the main body of the information processing apparatus 3 by a VGA cable, an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, or the like, or by an Ethernet cable.

  The network I / F 106 is a NIC (Network Interface Card) for connecting to the network 4 and performing data communication. Specifically, the information processing device 3 acquires video data from the imaging device 2 via the network 4 and the network I / F 106.

  A keyboard 107 is an input device for selecting characters, numbers, various instructions, moving a cursor, selecting setting information, and the like. The mouse 108 is an input device for selecting and executing various instructions, selecting a processing target, moving a cursor, selecting setting information, and the like.

  The DVD drive 109 is a device that controls reading, writing, and deletion of data with respect to the DVD 110 as an example of a removable storage medium.

  The CPU 101, the ROM 102, the RAM 103, the external storage device 104, the display 105, the network I / F 106, the keyboard 107, the mouse 108, and the DVD drive 109 are connected to each other via a bus 111 such as an address bus and a data bus. . When the display 105 is connected by an Ethernet cable, it is connected to the network I / F 106. In this case, data communication is performed using a protocol such as TCP / IP.

(Block configuration of information processing device)
FIG. 3 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. The block configuration of the information processing apparatus 3 will be described with reference to FIG.

  As shown in FIG. 3, the information processing apparatus 3 includes a video reception unit 301, a storage unit 302, an input device 303 (frame designation unit, input unit, display operation unit), a setting unit 304, and a playback unit 305 ( Reproduction means), area designation unit 306 (area designation means), frame acquisition unit 307 (acquisition means), difference calculation unit 308 (an example of calculation means), graph generation unit 309 (generation means), and auxiliary information A generation unit 310 (an example of a threshold value calculation unit), a display control unit 311, and a display device 312 (display unit) are provided.

  The video reception unit 301 is a functional unit that performs data communication with the imaging device 2 via the network 4 and receives video data from the imaging device 2. The video receiving unit 301 stores the received video data in the storage unit 302. The video receiving unit 301 is realized by the network I / F 106 shown in FIG. In FIG. 3, the network 4 is not shown for the sake of simplicity.

  The storage unit 302 is a storage device that stores the video data received by the video receiving unit 301 and various setting information. The storage unit 302 identifies and stores which imaging device 2 the video data received by the video reception unit 301 is. The storage unit 302 is realized by, for example, the external storage device 104 illustrated in FIG.

  The input device 303 is a device for performing operation input for image processing performed by the information processing device 3. The input device 303 is realized by the keyboard 107 and the mouse 108 shown in FIG.

  The setting unit 304 is a functional unit that sets setting information and stores the setting information in the storage unit 302 in accordance with an operation signal from the input device 303 operated by the user. In addition, the setting unit 304 sends information that needs to be displayed among the set setting information from the storage unit 302 to the display control unit 311, and causes the display control unit 311 to display the setting information on the display device 312. The setting unit 304 is realized by a program executed by the CPU 101 shown in FIG.

  The reproduction unit 305 sends the video data acquired from the storage unit 302 to the display control unit 311 according to an operation signal from the input device 303 operated by the user, and reproduces the video data on the display device 312 for the display control unit 311. It is a functional part to be displayed. The reproduction unit 305 is realized by a program executed by the CPU 101 shown in FIG.

  The area designating unit 306 designates an analysis area to be analyzed in the video data display area (video display unit 401 described later in FIG. 4) on the display device 312 in accordance with an operation signal from the input device 303 operated by the user. It is a functional part. The area specifying unit 306 causes the storage unit 302 to store information on the specified analysis area. Here, the analysis indicates image processing for calculating a graph by calculating a feature amount of a frame such as a difference from a frame of video data. The analysis area information is, for example, information such as coordinates indicating the position of the analysis area in the display area of the video data and the shape of the analysis area. The area specifying unit 306 is realized by a program executed by the CPU 101 shown in FIG.

  The frame acquisition unit 307 is the video data reproduced by the reproduction unit 305 in accordance with an operation signal from the input device 303 operated by the user (specifically, an operation signal when an analysis start button 410 described later is pressed). Among these, it is a functional unit that acquires frames of a predetermined period (analysis time) to be analyzed. Specifically, the frame acquisition unit 307 receives date / time information of the video data reproduced by the reproduction unit 305 from the reproduction unit 305, and acquires frames from the date / time to a predetermined period before. The frame acquisition unit 307 is realized by a program executed by the CPU 101 illustrated in FIG.

The difference calculation unit 308 receives the frame acquired by the frame acquisition unit 307 and receives information on the analysis region stored in the storage unit 302, and the analysis region of the target frame and the analysis region of the frame to be compared It is a functional part which calculates the difference about these pixel values as the feature-value of a flame | frame. As a calculation method of the difference as the feature amount of the frame, for example, the pixel values in the analysis regions of the two frames are compared, and the total number of pixels having different pixel values is calculated as the difference. If the frame is a grayscale image, the pixel value matches the luminance value, so the luminance values in the analysis areas of the two frames are compared, and the total number of pixels with different luminance values is calculated as the difference. Also good. When the frame is an image composed of RGB pixel values, the pixel values of R (red), G (green), and B (blue) are compared, and the pixel value is the same as described above. The total number of different pixels may be calculated as a difference. Further, when the frame is an image composed of pixel values of the L * a * b * color system, after normalizing the L * component, the a * component, and the b * component, √ ((L1-L2) ^The color difference due to the Euclidean distance of three points may be calculated as a difference, such as ² + (a1−a2) ² + (b1−b2) ² ). The difference calculation unit 308 is realized by a program executed by the CPU 101 shown in FIG.

  Also, the difference calculation unit 308 calculates the difference between each frame (target frame) acquired by the frame acquisition unit 307, the frame immediately before the target frame, the frame before a predetermined time, or a specific fixed frame. The frame to be compared (reference frame).

  Note that although the difference calculation unit 308 calculates the difference between the pixel values in the analysis region of the two frames as described above, the present invention is not limited to this. That is, the difference calculation unit 308, for each frame acquired by the frame acquisition unit 307, if the frame is an image composed of RGB pixel values, any pixel value of R, G, or B of the analysis region May be calculated as the feature amount of the frame.

  The graph generation unit 309 is a functional unit that generates a graph representing the difference values in time series from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307. The graph generation unit 309 sends the generated difference value graph information to the display control unit 311, and displays the difference value graph for the display control unit 311 in the display area of the display device 312 (a graph display unit 402 described later in FIG. 4). To display. The graph generation unit 309 is realized by a program executed by the CPU 101 illustrated in FIG.

  Note that the graph generation unit 309 is not limited to generating a difference value graph from all the difference information calculated by the difference calculation unit 308. In other words, the graph generation unit 309 includes difference information corresponding to each frame included in a predetermined sampling width (for example, in the case of a frame rate of 10 [FPS], a frame for one second, that is, 10 frames). It is good also as what produces | generates a graph from. Here, the difference corresponding to each frame included in a predetermined sampling width (for example, 1 second as described above) is calculated by the difference calculation unit 308 for each frame included in the sampling width, for example. It is assumed that the maximum difference among the differences is sampled. Further, the difference graph generated by the graph generation unit 309 is generated from all the difference information calculated by the difference calculation unit 308, or the maximum sampled for each frame included in the predetermined sampling width as described above. The setting unit 304 may select and set whether to generate the difference information.

  The auxiliary information generation unit 310 has a function of detecting the maximum difference from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307 and generating an auxiliary line indicating the maximum difference. Part. In addition, the auxiliary information generation unit 310 obtains a value of a predetermined ratio (for example, 80%) of the maximum difference as a threshold value, and generates an auxiliary line indicating the threshold value. Then, the auxiliary information generation unit 310 sends the generated auxiliary line indicating the maximum difference and the auxiliary line information indicating the threshold to the display control unit 311, and displays both auxiliary lines to the display control unit 311. The image is displayed on a display area 312 (a graph display unit 402 described later in FIG. 4). The auxiliary information generation unit 310 is realized by a program executed by the CPU 101 shown in FIG. Note that the generation and display of the above-described auxiliary line by the auxiliary information generation unit 310 is not essential and may not be displayed.

  The display control unit 311 is a functional unit that controls the display operation of the display device 312. Specifically, the display control unit 311 includes setting information set by the setting unit 304, video data acquired by the reproduction unit 305, a difference value graph generated by the graph generation unit 309, and an auxiliary information generation unit 310. The auxiliary line generated by the above is displayed on the display device 312. The display control unit 311 is realized by a program (driver or the like) executed by the CPU 101 shown in FIG.

  The display device 312 is a device that displays various data according to the control of the display control unit 311. In the present embodiment, display device 312 particularly displays application screen 400 shown in FIG. 4 and the like described later by a program (application) executed by CPU 101 shown in FIG. The display device 312 is realized by the display 105 shown in FIG.

  The area specifying unit 306 is not limited to one analysis area that can be specified in the video data display area, and a plurality of analysis areas may be specified. In this case, it is only necessary that the input device 303 can select the analysis area that is the target of the difference calculation by the difference calculation unit 308 and the graph generation unit 309. As a result, even when there are a plurality of locations in the video data display area where it is desired to confirm the occurrence of motion, each analysis area can be designated and analyzed, thereby increasing the efficiency of the analysis work.

  In addition, some or all of the setting unit 304, the reproduction unit 305, the region specification unit 306, the frame acquisition unit 307, the difference calculation unit 308, the graph generation unit 309, the auxiliary information generation unit 310, and the display control unit 311 illustrated in FIG. May be implemented by a hardware circuit such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA) instead of a program that is software.

  The setting unit 304, the playback unit 305, the region specification unit 306, the frame acquisition unit 307, the difference calculation unit 308, the graph generation unit 309, the auxiliary information generation unit 310, and the display control unit 311 conceptually show functions. However, it is not limited to such a configuration.

  In addition, for the sake of simplification of the following description, the expression that the display control unit 311 displays the received data on the display device 312 is simply expressed by the functional unit that has sent the data to the display control unit 311. It is expressed that it is displayed on (or a screen displayed on the display device 312). For example, when the reproduction unit 305 sends the video data to the display control unit 311 and causes the display control unit 311 to reproduce and display the video data on the display device 312, the reproduction unit 305 simply displays the video data on the display device 312. It shall be expressed as a playback display.

(Application screen that displays a frame difference graph)
FIG. 4 is a diagram illustrating an example in which video data is displayed on an application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 5 is a diagram illustrating an example in which an analysis region is specified in the video display unit on the application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 6 is a diagram illustrating an example in which a difference graph is displayed on the application screen displayed on the display device of the information processing apparatus according to the embodiment. FIG. 7 is a diagram illustrating an example of displaying video data at a desired time based on the difference graph on the application screen displayed on the display device of the information processing apparatus according to the embodiment. An operation on the application screen 400, an operation for displaying a frame difference graph, and the like will be described with reference to FIGS.

  An application screen 400 shown in FIG. 4 is displayed on the display device 312 by a program (application) executed by the CPU 101. The application screen 400 is a screen that displays video data to be analyzed, a graph of feature amounts such as differences as analysis results, and the like. As shown in FIG. 4 and the like, the application screen 400 includes a video display unit 401, a camera selection unit 401a, a graph display unit 402, a play button 403a, a pause button 403b, a fast forward button 403c, and a fast reverse button. 403d, stop button 403e, marking button 403f, seek bar 404, slider 405, video enlargement button 406a, video reduction button 406b, span enlargement button 407a, span reduction button 407b, and time display unit 408 , An analysis time setting unit 409, an analysis start button 410, and an analysis stop button 411.

  The video display unit 401 is a display unit that reproduces and displays video data acquired from the storage unit 302 according to an operation signal from the input device 303 operated by the user by the playback unit 305. The camera selection unit 401a is an operation unit that selects which video data is to be displayed on the video display unit 401 among the video data of the imaging devices 2a to 2d that the user desires to analyze. For example, the video data captured by the imaging devices 2a to 2d are associated with “1” to “4”, respectively. In the example of FIG. 4, “3”, that is, the video data of the imaging device 2c is selected. An example is shown. In this case, when the “3” camera selection unit 401 a is selected and operated by the user operating the input device 303, the playback unit 305 displays the video data captured by the imaging device 2 c on the display device 312.

  The graph display unit 402 is a display unit on which a graph of difference values generated by the graph generation unit 309 is displayed.

  When the playback button 403a is pressed by a user's operation of the input device 303 (click operation using the mouse 108, etc.), the playback unit 305 starts playback of video data displayed on the video display unit 401. It is. The pause button 403b is a button that, when pressed by the user operating the input device 303, causes the playback unit 305 to pause the video data that is played back and displayed on the video display unit 401. The fast-forward button 403c is a button for fast-forwarding and displaying the video data reproduced and displayed on the video display unit 401 by the reproduction unit 305 when pressed by an operation of the input device 303 by the user. The rewind button 403d is a button for quickly rewinding and displaying the video data reproduced and displayed on the video display unit 401 by the reproduction unit 305 when pressed by the user operating the input device 303. The stop button 403e is a button for stopping the video data reproduced and displayed on the video display unit 401 by the reproduction unit 305 when pressed by an operation of the input device 303 by the user. As will be described later, the marking button 403f is a button that is displayed as a mark immediately above the slider 405 on the seek bar 404 by the auxiliary information generation unit 310 when pressed by an operation of the input device 303 by the user.

  The seek bar 404 is a bar-shaped body that indicates how long the video data reproduced and displayed on the video display unit 401 is based on the position of the slider 405 disposed thereon. The slider 405 is a shape body and operation unit that slides to a position on the seek bar 404 corresponding to the imaging time of the video data reproduced and displayed on the video display unit 401 (time when the displayed frame is captured). Conversely, when the slider 405 is slid by the operation of the input device 303 by the user, the playback unit 305 causes the video display unit 401 to generate a frame of video data at the imaging time corresponding to the position on the seek bar 404 where the slider 405 exists. Is displayed.

  The video enlargement button 406a is a button for enlarging and displaying the video data reproduced and displayed on the video display unit 401 by the reproduction unit 305 when pressed by an operation of the input device 303 by the user. For example, every time the video enlargement button 406a is pressed, the video data is enlarged and displayed at a predetermined enlargement ratio. For example, assuming that the normal display state is 100%, the display is enlarged as 120%, 140%,. The video reduction button 406b is a button for reducing and displaying video data reproduced and displayed on the video display unit 401 by the reproduction unit 305 when pressed by an operation of the input device 303 by the user. For example, every time the video reduction button 406b is pressed, the video data is reduced and displayed at a predetermined reduction rate. For example, assuming that the normal display state is 100%, the display is reduced like 80%, 60%,.

  The span expansion button 407a is a button for expanding the display span of the seek bar 404 by the playback unit 305 when pressed by the user operating the input device 303. The span reduction button 407 b is a button for reducing the display span of the seek bar 404 by the playback unit 305 when pressed by the user operating the input device 303. Note that it is not necessary for the playback unit 305 to enlarge or reduce the display span of the seek bar 404, and a different function unit (not shown) may be responsible for it.

  The time display unit 408 is a display unit that displays the imaging date and imaging time of the video data reproduced and displayed on the video display unit 401 by the reproduction unit 305.

  The analysis time setting unit 409 is a radio button for setting how many frames of video data reproduced and displayed on the video display unit 401 are to be analyzed. When a radio button of any time (analysis time) in the analysis time setting unit 409 is selected by an operation of the input device 303 by the user, the setting unit 304 stores the selection content as setting information in the storage unit 302. Is done. The setting unit 304 also displays on the analysis time setting unit 409 which analysis time is selected according to the analysis time setting information stored in the storage unit 302. In the example of FIG. 4, it is shown that “10 minutes” is selected as the analysis time. In this case, assuming that the frame rate of the video data is 10 [FPS], the frame acquisition unit 307 has 10 minutes of frames of video data reproduced and displayed on the video display unit 401, that is, 6000 frames. Get the frame.

  When the analysis start button 410 is pressed by the operation of the input device 303 by the user, the frame acquisition unit 307 acquires frames for the analysis time from the video data, the difference calculation unit 308 calculates the frame difference, and the graph generation unit This is a button for starting the generation of the difference value graph by 309 and the operation of generating the auxiliary line by the auxiliary information generating unit 310. The operation when the analysis start button 410 is pressed will be described later.

  When the analysis stop button 411 is pressed by an operation of the input device 303 by the user, the operation of the frame acquisition unit 307, the difference calculation unit 308, the graph generation unit 309, and the auxiliary information generation unit 310 is stopped, and the graph generation unit A button 309 is used to delete the difference value graph displayed on the graph display unit 402. Note that the analysis stop button 411 may be pressed even during the process started when the analysis start button 410 is pressed so as to exhibit the function.

  The application screen 400 shown in FIG. 4 displays the video data of the imaging device 2c acquired from the storage unit 302 by the playback unit 305 on the video display unit 401, and plays back and displays on the video display unit 401 on the time display unit 408. This shows a state in which the imaging date and imaging time of the video data being displayed are displayed. Further, the application screen 400 shown in FIG. 4 does not start the analysis operation for the difference in the frame of the video data when the analysis start button 410 is pressed, and the graph display unit 402 displays the difference value graph. Indicates a state that is not displayed.

  An application screen 400 shown in FIG. 5 shows a state in which an analysis area 501 to be analyzed for differences is specified in the video display unit 401 on which video data is reproduced and displayed. Specifically, the analysis area 501 is an area designated by the area designating unit 306 as an analysis target for frame differences in the video display unit 401 in accordance with an operation of the input device 303 by the user (for example, a drag operation with the mouse 108). It is. The area specifying unit 306 causes the storage unit 302 to store information on the specified analysis area 501. In addition, when the area designating unit 306 designates the analysis area 501, the video data displayed on the video display unit 401 may be in a reproduction state or a pause state. Further, in the application screen 400 shown in FIG. 5, as in FIG. 4, the analysis operation for the difference between the frames of the video data due to the pressing of the analysis start button 410 is not started, and the graph display unit 402 The graph of the difference value is not displayed.

  As an area designated as the analysis area 501, for example, a part is predicted to fall in a production facility, and a part where it is necessary to analyze when, how, and why the part falls is assumed. In addition, for example, in a signal tower having red, yellow, green and blue lamps provided in the production facility, when a specific abnormality of unknown cause occurs in the production facility, the operation is to turn on the blue lamp. It is also assumed that the video portion of the blue lamp is designated as the analysis region 501.

  As shown in FIG. 5, the analysis region 501 is designated as a rectangular shape, but is not limited to this. For example, the analysis region 501 may be designated with a polygonal shape, or may be designated with an arbitrary shape by freehand.

  In the application screen 400 shown in FIG. 6, when the analysis start button 410 is pressed by the user operating the input device 303, the frame acquisition unit 307 acquires a frame for the analysis time from the video data, and the difference calculation unit 308 The difference in the analysis area 501 of the frame is calculated, the graph of the difference value is generated by the graph generation unit 309, and the difference graph 601 that is the graph of the difference value is displayed on the graph display unit 402.

  Specifically, first, when the user presses the pause button 403b by operating the input device 303 at a timing that is a guideline for analysis while the video display unit 401 reproduces and displays video data, the playback unit 305 The video data reproduced and displayed on the video display unit 401 is paused. When the user selects any analysis time from the analysis time setting unit 409 by operating the input device 303, the setting unit 304 sets the selected analysis time as setting information and stores it in the storage unit 302. Let It goes without saying that the analysis time set by the user may be set in advance, not at this timing.

  Next, when the user presses the analysis start button 410 by operating the input device 303, the frame acquisition unit 307 causes the imaging time that is temporarily stopped among the video data reproduced and displayed on the video display unit 401, That is, frames from the time displayed on the time display unit 408 to the analysis time set by the setting unit 304 (10 minutes in FIG. 6) are acquired. Next, the difference calculation unit 308 receives the frame acquired by the frame acquisition unit 307 and receives the information of the analysis region 501 stored in the storage unit 302, and is compared with the analysis region 501 of the target frame. The difference between the pixel value and the analysis area 501 of the frame is calculated as the feature amount of the frame. At this time, the difference calculation unit 308 calculates the difference for all the frames acquired by the frame acquisition unit 307. Then, as illustrated in FIG. 6, the graph generation unit 309 generates a difference graph 601 that is a difference value graph from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307. And displayed on the graph display unit 402. In the above description, it is assumed that the user presses the analysis start button 410 when the video data displayed on the video display unit 401 is paused. However, the present invention is not limited to this, and the video data is being played back. Alternatively, the analysis operation may be started by pressing the analysis start button 410. In this case, the frame acquisition unit 307 extracts the imaging time of the video data at the timing when the analysis start button 410 is pressed, and acquires frames from the extracted imaging time until the analysis time set by the setting unit 304. That's fine.

  In addition, as illustrated in FIG. 6, the graph generation unit 309 causes the start time display unit 605 to display the imaging time corresponding to the forefront frame acquired from the video data, and ends the imaging time corresponding to the last frame. It is displayed on the time display unit 604.

  Further, the auxiliary information generation unit 310 detects the maximum difference from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307 and generates an auxiliary line indicating the maximum difference. To do. In addition, the auxiliary information generation unit 310 calculates a predetermined ratio value of the maximum difference value as a threshold value, and generates an auxiliary line indicating the threshold value. Then, as illustrated in FIG. 6, the auxiliary information generation unit 310 superimposes the difference graph 601 displayed on the graph display unit 402 to display a maximum difference value line 602 that is an auxiliary line indicating the maximum difference, A threshold line 603 which is an auxiliary line indicating the threshold is displayed. In this way, as shown in FIG. 6, the maximum difference value line 602 and the threshold line 603 are displayed so as to be superimposed on the difference graph 601 displayed on the graph display unit 402, thereby causing some kind of analysis in the video data analysis region 501. It becomes easy to specify the time when the movement occurred. Further, as described above, the threshold value of the threshold line 603 is set to a predetermined ratio of the maximum difference value of the maximum difference value line 602, and therefore any part of the difference graph 601 exceeds the threshold line 603. become. Therefore, by searching for a location that exceeds the threshold line 603 in the difference graph 601, there is a high possibility that the time when some movement has occurred in the analysis region 501 can be specified.

  Further, in the example in which the analysis area 501 described above is designated as a part where a part is expected to be dropped in the production facility, the difference graph 601, the maximum difference value line 602, and the threshold line 603 indicate the time when the movement occurs, , It will be easier to specify the time when the parts dropped. Further, in the example in which the video portion of the blue lamp of the signal tower provided in the above-described production facility is designated as the analysis region 501, the time when the movement occurs by the difference graph 601, the maximum difference value line 602, and the threshold line 603, that is, It becomes easier to specify the time when the blue lamp is lit.

  In the application screen 400 shown in FIG. 7, the user operates the input device 303 in the graph display unit 402, for example, the cursor 701 is moved by the mouse 108 and a click operation is performed on the graph display unit 402 on which the difference graph 601 is displayed. , A state in which a frame at a designated time T1, which is an imaging time corresponding to the point of the cursor 701 when the click operation is performed (or a frame in the number of video data corresponding to the point), is displayed on the video display unit 401. ing. In the application screen 400 shown in FIG. 7, the slider 405 is moved by the user operating the input device 303, the frame one minute after the specified time T1 is displayed on the video display unit 401, and the analysis area of the frame is displayed. A state in which a falling object is reflected in 501 is shown. Furthermore, in the application screen 400 shown in FIG. 7, when the marking button 403f is pressed by the user operating the input device 303, a mark 801 is displayed immediately above the slider 405 on the seek bar 404 at the time of pressing. Indicates the state.

  Specifically, first, when the user moves the cursor 701 with the mouse 108 serving as the input device 303 and performs a click operation on the graph display unit 402 on which the difference graph 601 is displayed, the playback unit 305 performs a click operation by the user. A frame at a designated time T1, which is an imaging time corresponding to the point of the cursor 701 at the time when the image is made, is displayed on the video display unit 401. In addition, the playback unit 305 causes the time display unit 408 to display the designated time T1 when the frame at the designated time T1 is displayed on the video display unit 401. Also, the playback unit 305 moves the slider 405 to the middle of the seek bar 404, that is, “0:00” when the frame at the specified time T1 is displayed on the video display unit 401. That is, “0:00” of the seek bar 404 corresponds to the designated time T1 displayed on the time display unit 408.

  Next, when the user slides the slider 405 on the seek bar 404 by operating the input device 303, the playback unit 305 corresponds to the time obtained by adding the time corresponding to the position of the slider 405 on the seek bar 404 from the specified time T1. The frame is displayed on the video display unit 401. In the application screen 400 shown in FIG. 7, the frame corresponding to the case where the user slides the slider 405 to the position “10:00” on the seek bar 404, that is, the designated time T <b> 1 (here, “17:59:11”). .28 ”), a frame corresponding to a time one minute after (18: 00: 11.28 in the example of FIG. 7) is displayed on the video display unit 401. In this case, as shown in FIG. 7, falling objects are reflected in the analysis area 501 of the frame displayed on the video display unit 401, and it was possible to search for a frame in which movement occurred in the analysis area 501. become. It is assumed that the difference value for the analysis region 501 of this frame is a large value, that is, this frame corresponds to the vicinity of the mountain part reaching the maximum difference value line 602 of the difference graph 601 of the graph display unit 402. It can be assumed that this is a frame to be played.

  Further, in the example in which the analysis area 501 described above is designated as a part where a part is expected to drop in the production facility, the portion of the difference graph 601 that exceeds the threshold line 603 is clicked, and the frame at that time is displayed. By displaying the image on the image display unit 401 and confirming the images before and after the time by the sliding operation of the slider 405, it is easy to identify how the component has dropped and why. In addition, in the example in which the image portion of the blue lamp of the signal tower provided in the above-described production facility is designated as the analysis region 501, similarly, the portion of the difference graph 601 that exceeds the threshold line 603 is clicked, and the time Is displayed on the video display unit 401, and by confirming the video before and after the time by sliding the slider 405, it becomes easy to analyze in which state and environment the abnormality has occurred, and the occurrence of the abnormality. It becomes easier to identify the cause of.

  In addition, as shown in FIG. 7, when the user presses the marking button 403 f by operating the input device 303 in a state where a fallen object is reflected in the analysis region 501, that is, in a state where movement occurs in the analysis region 501, auxiliary information The generation unit 310 causes the mark 801 to be displayed in a portion immediately above the slider 405 on the seek bar 404. By displaying this mark 801, it is possible to mark the time when the movement has occurred on the seek bar 404, which can be used as a clue for analyzing the cause of the movement.

  As described above, the difference calculation unit 308 receives frames for the analysis time acquired by the frame acquisition unit 307, and calculates feature amounts (differences, etc.) for pixel values between the target frame and the comparison target frame. The graph generation unit 309 generates a feature amount graph (for example, the difference graph 601) from the feature amount information calculated by the difference calculation unit 308, and displays the graph on the display device 312. This makes it easy to specify the time when the feature amount increases, and makes it easy to specify the time when some movement has occurred in the video data.

  Further, the auxiliary information generation unit 310 superimposes the difference graph 601 displayed on the graph display unit 402 to display a maximum value line (for example, the maximum difference value line 602) indicating the maximum feature amount, and to indicate a threshold value. A threshold line that is an auxiliary line (for example, a threshold line 603) is displayed. This makes it easier to specify the time when some movement has occurred in the difference graph 601.

  In addition, the area designating unit 306 designates an analysis area in the video data display area (video display unit 401), and the difference calculation unit 308 performs analysis of the target frame on the frame acquired by the frame acquisition unit 307. And the feature value for the pixel value of the analysis area of the frame to be compared is calculated. As a result, since it is possible to designate a portion where movement is considered to be an analysis area, a change in the characteristic amount when the movement occurs is significant for the characteristic amount calculated by the difference calculation unit 308. Therefore, it becomes easy to specify the time when the feature amount changes greatly, and it becomes easier to specify the time when some movement occurs in the video data. For example, in the case where the target of feature amount calculation by the difference calculation unit 308 is the entire frame, if there is a constant movement in the entire display area of the video data, a change in the feature amount due to the entire movement appears on the graph. Therefore, in the display area of the video data, even if the movement occurs in a place where the user desires to analyze and the movement (for example, the fall of an object) seems to occur, It may not appear on the graph. On the other hand, when the above-mentioned location is designated as the analysis area by the area designating unit 306, the movement that occurs in the analysis area is more likely to appear in the graph as a change in the feature amount, and the movement occurs. It becomes easy to grasp the state.

  Further, the operation of instructing the specific portion of the area of the graph display unit 402 on which the feature amount graph is displayed by the input device 303 (for example, a click operation with the mouse 108) causes the reproduction unit 305 to display the frame of the specified specific portion. Is displayed on the video display unit 401. Accordingly, the frame at the imaging time desired by the user can be quickly displayed, and the cause of the movement or the cause of the event accompanying the movement can be easily identified.

  Further, as described above, after the playback unit 305 displays the frame at the imaging time corresponding to the specified specific part on the video display unit 401, the user can move the slider 405 of the seek bar 404 by sliding the frame 405. The images before and after the imaging time can be checked. This makes it easier to identify the cause of the movement or the cause of the event associated with the movement.

(Image processing operation to display the difference graph)
FIG. 8 is a flowchart illustrating an example of an image processing operation for displaying a difference graph in the image processing system according to the embodiment. With reference to FIG. 8, the flow of the image processing (analysis) operation until the difference graph is displayed will be generally described.

<Step S11>
The reproduction unit 305 reproduces and displays the video data of the imaging device 2 acquired from the storage unit 302 on the video display unit 401 of the application screen 400. Then, the process proceeds to step S12.

<Step S12>
The area specifying unit 306 specifies an analysis area 501 that is a specified area for obtaining a frame difference in the video display unit 401 in accordance with an operation (for example, a drag operation) of the input device 303 by the user, and the specified analysis area 501 is specified. Is stored in the storage unit 302. Then, the process proceeds to step S13.

<Step S13>
When the analysis start button 410 is pressed by an operation of the input device 303 by the user, the frame acquisition unit 307 operates when the analysis start button 410 is pressed among the video data reproduced and displayed on the video display unit 401. Frames from the imaging time of the video data to the analysis time set by the setting unit 304 are acquired. Then, the process proceeds to step S14. Note that the timing of shifting to step S14 does not necessarily have to be after all frames before the set analysis time have been acquired by the detection area designating unit 307, but during the acquisition of frames. May be.

<Step S14>
The difference calculation unit 308 uses the frame immediately before the target frame or the frame before the predetermined time as a frame to be compared with respect to each frame (target frame) acquired by the frame acquisition unit 307. (Step S14: previous frame), the process proceeds to step S16. On the other hand, when the difference calculation unit 308 calculates a difference regarding the pixel value for each frame (target frame) acquired by the frame acquisition unit 307 using a specific fixed frame as a comparison target frame (step S14). : Specific frame), the process proceeds to step S15. Whether the frame to be compared is a frame immediately before the target frame, a frame before a predetermined time, or a specific fixed frame may be set by the setting unit 304 as setting information.

<Step S15>
The difference calculation unit 308 receives the frame acquired by the frame acquisition unit 307 and receives the information of the analysis region 501 stored in the storage unit 302, and the analysis region 501 of the target frame and the analysis region of the specific fixed frame The difference with respect to the pixel value from 501 is calculated as the feature amount of the frame. At this time, the difference calculation unit 308 calculates the difference for all the frames acquired by the frame acquisition unit 307. Then, the process proceeds to step S17.

<Step S16>
The difference calculation unit 308 receives the frame acquired by the frame acquisition unit 307 and receives the information of the analysis region 501 stored in the storage unit 302, the analysis region 501 of the target frame, the frame immediately before the target frame, Alternatively, a difference in pixel value from the analysis area 501 of the frame before a predetermined time is calculated as a frame feature amount. At this time, the difference calculation unit 308 calculates the difference for all the frames acquired by the frame acquisition unit 307. Then, the process proceeds to step S17.

<Step S17>
The auxiliary information generation unit 310 detects the maximum difference from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307, and generates an auxiliary line indicating the maximum difference. The auxiliary information generation unit 310 obtains a value of a predetermined ratio of the maximum difference value as a threshold value, and generates an auxiliary line indicating the threshold value. Then, the process proceeds to step S18.

<Step S18>
When the graph generation unit 309 generates a graph representing all the difference values in time series from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307 (step S18: No) When the process proceeds to step S19 and a graph representing the difference value corresponding to each frame included in the predetermined sampling width in time series is generated (step S18: Yes), the process proceeds to step S20. Whether to generate a graph of all difference values or to generate a graph of difference values corresponding to each frame included in a predetermined sampling width may be set as setting information by the setting unit 304. .

<Step S19>
The graph generation unit 309 generates a graph representing all the difference values in time series from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307 and displays the graph on the graph display unit 402. Let Then, the process proceeds to step S21.

<Step S20>
The graph generation unit 309 is a graph that represents the difference values corresponding to each frame included in the predetermined sampling width in time series from the difference information calculated by the difference calculation unit 308 for each frame acquired by the frame acquisition unit 307. Is generated and displayed on the graph display unit 402. Here, the difference corresponding to each frame included in a predetermined sampling width (for example, 1 second as described above) is calculated by the difference calculation unit 308 for each frame included in the sampling width, for example. It is assumed that the maximum difference among the differences is sampled. In this case, as described above with reference to FIG. 7, when a click operation is performed on the graph display unit 402 on which the difference graph is displayed, the maximum difference belonging to the sampling width including the imaging time corresponding to the point of the cursor 701 is included. The image capturing time of the frame corresponding to may be set as the designated time T1. Then, the process proceeds to step S21.

<Step S21>
The operation of the input device 303 by the user in the graph display unit 402, for example, when the cursor 701 is moved by the mouse 108 and the click operation is performed on the graph display unit 402 on which the difference graph 601 is displayed. A frame at the designated time T1 that is the imaging time corresponding to the point of the cursor 701 (or a frame in the number of video data corresponding to the point) is displayed on the video display unit 401. The image processing (analysis) operation for displaying the difference value graph and displaying the frame at the designated time T1 on the video display unit 401 is thus completed.

  In addition, at least one of the setting unit 304, the reproduction unit 305, the region specification unit 306, the frame acquisition unit 307, the difference calculation unit 308, the graph generation unit 309, the auxiliary information generation unit 310, and the display control unit 311 according to this embodiment. When realized by executing the program, the program is provided by being incorporated in advance in a ROM or the like. The program executed by the information processing apparatus 3 according to the present embodiment is an installable or executable file and is read by a computer such as a CD-ROM, flexible disk (FD), CD-R, or DVD. You may comprise so that it may record on a possible recording medium and provide. The program executed by the information processing apparatus 3 according to the present embodiment may be configured to be stored by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the information processing apparatus 3 of the present embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the information processing apparatus 3 according to the present embodiment includes the setting unit 304, the reproduction unit 305, the region specification unit 306, the frame acquisition unit 307, the difference calculation unit 308, the graph generation unit 309, and the auxiliary information generation unit. The module includes at least one of 310 and the display control unit 311. As actual hardware, the CPU reads a program from the above-described ROM and executes the program so that the above-described units are loaded onto the main storage device. Has been generated.

DESCRIPTION OF SYMBOLS 1 Image processing system 2, 2a-2d Image pick-up device 3 Information processing device 4 Network 101 CPU
102 ROM
103 RAM
104 External storage device 105 Display 106 Network I / F
107 keyboard 108 mouse 109 DVD drive 110 DVD
DESCRIPTION OF SYMBOLS 111 Bus 301 Image | video receiving part 302 Storage part 303 Input device 304 Setting part 305 Playback part 306 Area designation part 307 Frame acquisition part 308 Difference calculating part 309 Graph generation part 310 Auxiliary information generation part 311 Display control part 312 Display apparatus 400 Application screen 401 Video display unit 401a Camera selection unit 402 Graph display unit 403a Play button 403b Pause button 403c Fast forward button 403d Fast reverse button 403e Stop button 403f Marking button 404 Seek bar 405 Slider 406a Video enlarge button 406b Video reduce button 407a Span enlarge button 407b Span shrink Button 408 Time display section 409 Analysis time setting section 410 Analysis start button 411 Analysis stop button 501 Analysis area 601 Difference Graph 602 maximum difference value line 603 the threshold line 604 end time display part 605 starts the time display unit 701 cursor 801 marks T1 specified time

Japanese Patent No. 4706573

Claims (16)

Display means;
Reproduction means for reproducing and displaying the video data imaged by the imaging means on the display means;
A calculation means for calculating a feature value of a pixel value for each frame included in the video data;
Generating means for generating a graph representing the feature quantity in time series by the feature quantity computed by the computing means and displaying the graph on the display means;
A frame designating unit that designates any part of the display area of the graph displayed by the display unit, and causes the display unit to display the frame corresponding to the part;
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the frame designating unit causes the reproduction unit to display the frame at the imaging time corresponding to the portion on the display unit. An input means,
The image processing apparatus according to claim 1, wherein the frame specifying unit specifies the portion in accordance with an operation signal from the input unit. An area specifying means for specifying an analysis area in the display area of the video data displayed by the display means;
The image processing apparatus according to claim 1, wherein the calculation unit calculates the feature amount of the analysis region for each of the frames included in the video data. An input means,
The image processing apparatus according to claim 4, wherein the region specifying unit specifies the analysis region in accordance with an operation signal from the input unit.   The image processing apparatus according to claim 4, wherein the calculation unit calculates a difference between pixel values of the analysis region of the frame and the analysis region of a reference frame included in the video data as the feature amount. .   The image processing apparatus according to claim 6, wherein the reference frame is a frame immediately before or a predetermined time before the frame that is the calculation target of the difference by the calculation unit.   The image processing apparatus according to claim 6, wherein the reference frame is a specific fixed frame.   The image processing apparatus according to claim 4, wherein the calculation unit calculates a sum of pixel values relating to a predetermined color in the analysis region of the frame included in the video data as the feature amount.   The image processing apparatus according to claim 3, wherein the area designating unit designates a plurality of the analysis areas in the display area of the video data. An acquisition means for acquiring the frame included in a predetermined period of the frames included in the video data displayed by the display means;
The image processing apparatus according to claim 1, wherein the calculation unit calculates the feature amount for each of the frames acquired by the acquisition unit.   Threshold calculation means for obtaining a maximum value from the feature value calculated by the calculation means, calculating a value of a predetermined ratio of the maximum value as a threshold value, and superimposing the threshold value on the graph to display the auxiliary line of the threshold value on the display means The image processing apparatus according to any one of claims 1 to 11, further comprising:   The image processing apparatus according to claim 1, further comprising: a display operation unit that causes the display unit to display a video based on frames before and after the frame at the imaging time. One or more imaging means;
An image processing device according to any one of claims 1 to 13,
An image processing system. A reproduction step of causing the display means to reproduce and display the video data imaged by the imaging means;
A calculation step of calculating a feature value of a pixel value for each frame included in the video data;
A generation step of generating a graph representing the feature amount in time series by the calculated feature amount and displaying the graph on the display unit;
A frame designating step of designating any part of the display area of the graph displayed by the display means, and causing the display means to display the frame corresponding to the part;
An image processing method. On the computer,
A reproduction process for causing the display means to reproduce and display the video data imaged by the imaging means;
A calculation process for calculating a feature value of a pixel value for each frame included in the video data;
A generation process for generating a graph representing the feature amount in time series by the calculated feature amount and displaying the graph on the display unit;
A frame designation process for designating any part of the display area of the graph displayed by the display means, and causing the display means to display the frame corresponding to the part;
A program that executes

Images