JP6459256B2 - Image storage / playback control apparatus, image storage / playback control method, image storage / playback control system, and program - Google Patents

Description

  The present invention relates to an image storage / reproduction control apparatus, an image storage / reproduction control method, an image storage / reproduction control system, and a program for extracting an image from a monitoring target image according to a predetermined condition.

Conventionally, as an apparatus for storing images, for example, a PC (Personal Computer) as an external connection device and one or a plurality of cameras are connected, and image data captured by the cameras are stored / reproduced, and Network video recorders that transmit image data are known.
Since such a network video recorder can efficiently check images taken by one or a plurality of cameras, it is used in various situations such as defect detection during work and crime prevention.

  However, in order to detect malfunctions during work, etc., it is necessary to search for the images to be confirmed by playing back, fast-forwarding, rewinding, etc. the recorded images individually for a long time. Depending on the time, the time required for confirmation becomes enormous, and a great deal of labor is required.

  Therefore, when an intruder is detected using an infrared sensor, a temperature sensor, etc., the video data is controlled by starting recording and recording the number of the camera that took the video, the recording start time, etc. in the management table. Has been proposed (Patent Document 1).

  In Patent Document 1, the number of the camera that has captured the video, the recording start time, and the like are recorded in the management table. However, there are cases where the contents of the video data cannot be confirmed from those items, and it is necessary to reproduce the video data in order to confirm the contents of the video data.

  The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to make it possible to easily extract an image based on a condition designated by a user from a captured image.

The present invention provides an image extracting means for extracting an image from a monitoring target image obtained by photographing a monitoring target according to conditions set by a user with respect to a worker's work area or work time, and the worker's work area, work time, or a list generation means for associating a mark associated with a particular event set by the user to the images extracted by the image extracting unit, and stores generated in the storage device a list for work, stored in the storage device has been read the list, along with displays on the display device, wherein when the front Kemah over click from the list displayed on the display device is selected to identify the image associated with the mark, reproduction on said display device a reproduction control means for displaying, Ru image storing and reproducing controller der with.

  According to the present invention, a user can easily extract a video based on a condition designated by the user from a captured video.

1 is a schematic block diagram of an image storage / playback control apparatus according to an embodiment of the present invention. 1 is a functional block diagram of an image storage / playback control device according to an embodiment of the present invention. It is a figure which shows the image | video which image | photographed the operator's working area with the imaging device. It is a figure regarding the detection of the feature-value of the image storage reproduction control apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows a process until video data is listed and memorize | stored by the image memory | storage reproduction | regeneration control apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of a display of a marker list. It is a figure which shows the example of reproduction | regeneration of video data. It is a figure which shows the example of a display of the graph linked | related with video data. It is a figure which shows the example of a reproduction | regeneration in a split screen display. FIG. 3 is a flowchart showing video data reproduction control processing by the image storage reproduction control apparatus according to the first embodiment. It is a figure which shows the image | video which image | photographed the screw insert processing operation | work with the imaging device. It is a figure regarding the detection of the feature-value of the image storage reproduction control apparatus concerning Embodiment 2. It is a flowchart which shows a process until video data is listed and memorize | stored by the image memory | storage reproduction | regeneration control apparatus which concerns on Embodiment 2. FIG. It is a figure which shows the image | video which image | photographed the operation | work of the machine which installed the red warning light and the green work light with the imaging device. It is a figure regarding the detection of the feature-value of the image storage reproduction control apparatus which concerns on Embodiment 3. It is a flowchart which shows a process until video data is list-ized and memorize | stored by the image storage reproduction control apparatus which concerns on Embodiment 3.

Embodiments of the present invention will be described below with reference to the drawings. In the following description, the image is described as a video (moving image). However, the image is not necessarily limited to this and may be a still image.
FIG. 1 is a schematic block diagram of a network video recorder 100 according to an embodiment of the image storage / playback control apparatus of the present invention.
The network video recorder 100 configures the network video system 1 by connecting to the display device 200, the photographing device 300, the external storage device 400, the input device 500, and the external connection device 600.
The network video system 1 corresponds to the image storage / playback control system in the present invention.

As shown in FIG. 1, a network video recorder (hereinafter referred to as NVR) 100 can physically store video streams of a plurality of channels.
Further, the NVR 100 monitors a video based on a condition designated by the user, and when the video changes, the NVR 100 gives the video additional information associated with information (specific event) set by the user ( In this embodiment, it is marked with a mark), and further associated with the photographing time and the ID of the photographing apparatus 300 and stored in a list.
Note that predetermined software is installed in the NVR 100, and the stored video data and the like can be displayed on the display device 200. The function of the NVR 100 will be described with reference to FIG.

A display device (here, a monitor) 200 corresponds to the display means of the present invention, and is, for example, a liquid crystal display device. The monitor 200 displays the video currently captured by the imaging device 300 connected to the NVR 100, the video stored in the video data storage unit 150 (FIG. 2) in the NVR 100, and the like.
The display specification of the monitor 200 is the same as the display specification of the display or screen of the external connection device 600, and will be described later with reference to FIG.

The image capturing apparatus 300 is, for example, a video camera, an IP (Internet Protocol) camera, a web camera, or the like for capturing an object to be imaged. In addition, the image which image | photographed the to-be-photographed object for monitoring with the imaging device 300 is called monitoring object image in this invention.
The imaging device 300 includes a time management unit (not shown) that manages the imaging time and a unique ID unit (not shown) that holds an ID (identification information) unique to the imaging device 300. The shooting time and the unique ID are associated with the video data as meta information.
The video data is stored in the video data storage unit 150 (FIG. 2) inside the NVR 100 by connecting the photographing apparatus 300 to the NVR 100 via a hub (HUB) 330 that functions as a network device that connects the devices. .
In addition, the image capturing apparatus 300 functions as an external image recording unit separately from the image data storage unit 150 in the NVR 100, and the image capturing apparatus 300 itself can store image data at the time of image capturing.

The external storage device 400 is a large-capacity storage device such as a hard disk, and stores video data shot by a shooting device other than the shooting device 300.
Further, when shooting is performed with a shooting device other than the shooting device 300 and video data is stored in the shooting device itself, the shooting device may be the external storage device 400. In this case, by inserting the memory card of the photographing device into the memory slot 120 (FIG. 2), and connecting the USB cable to the USB connection interface (I / F) 130 (FIG. 2), Can be transferred.
The input device 500 is, for example, a keyboard, a mouse, etc., and is a device for the user to perform various operations on the NVR 100.

The external connection device 600 is, for example, a PC (Personal Computer), a tablet, a smartphone, or the like.
The external connection device 600 can view video data and the like stored in the NVR 100 by installing predetermined software.
Specifically, the external connection device 600 displays a selected function that can reproduce one selected video data, a function that can divide a display screen and reproduce a plurality of video data, and a list stored in the NVR 100. It has a function of playing back videos and / or graphs associated with the list.
In addition, the external connection device 600 is connected to the NVR 100 by wire such as a LAN (Local Area Network) cable or wirelessly such as a wireless LAN, Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), or the like.

FIG. 2 is a functional block diagram of the network video recorder 100 according to an embodiment of the image storage / playback control apparatus of the present invention.
The NVR 100 has an external video input unit 110, a memory slot 120, a USB (Universal Serial Bus) connection interface 130, a user interface 140, an external video output unit 160, a LAN port 170, a wireless interface 180 as input / output functions for video data and the like. As an image storage / reproduction control function, a video data storage unit 150, a time management unit 151, a setting information acquisition unit 152, a feature amount detection unit 153, a feature video acquisition unit 154, a graph generation unit 155, a video, A data list storage unit 156, a marker setting unit 157, and a display control unit 158 are provided.
Of the above units, the setting information acquisition unit 152, the feature amount detection unit 153, the feature video acquisition unit 154, the graph generation unit 155, the marker setting unit 157, and the display control unit 158 cause the computer of the NVR 100 to read the program. It is a function realization means produced | generated by this.

  The external video input unit 110 is an interface (channel) for connecting the NVR 100 and the imaging apparatus 300 (300A to 300F). As shown in FIG. 2, the NVR 100 includes, for example, six channels as the external video input unit 110, and the imaging devices 300A to 300F are connected to the respective channels. Thereby, video data for six channels can be simultaneously recorded on the NVR 100 and / or displayed on the monitor 200.

The memory slot 120 is a slot for inserting a memory card that has been photographed by the photographing device 300 or a photographing device other than the photographing device 300 and stored video data at the time of photographing.
The connection interface 130 such as a USB is an interface for connecting the input device 500 such as a keyboard and a mouse with a USB cable, and allows the user to operate the NVR 100. Further, the user can transfer the video data stored in the external storage device 400 to the NVR 100 by connecting the external storage device 400 to the connection interface 130 such as a USB with a USB cable.
The user interface 140 is an interface for the user to directly operate the NVR 100 such as power operation of the NVR 100, storage and playback control.

The video data storage unit 150 is a video storage unit inside the NVR 100 and is temporally synchronized with video data captured by the imaging devices 300A to 300F and input from the external video input unit 110 at a time such as the imaging time. Then, the video data is stored.
The video data storage unit 150 also stores video data transferred from the external storage device 400 via the memory slot 120 or the connection interface 130 such as a USB.
The time management unit 151 manages the time in the NVR 100, and manages time information associated with video data captured by the imaging apparatus 300 and stored in the NVR 100, and time information used when synchronizing the video data.
If the video data cannot be properly synchronized at the time managed by the time management unit 151 in the NVR 100 due to a delay in data communication or the like, it is managed by the time management unit provided in the photographing apparatus 300. The video data can be synchronized from the time associated with the video data.

The setting information acquisition unit 152 acquires information necessary for detecting a feature video from video data by a feature amount detection unit 153 described later.
For example, when the user wants to monitor a specific area from the video imaged by the imaging device 300, the setting information acquisition unit 152 acquires the area specified by the input device 500 or the user interface 140 as setting information.
The setting information acquisition unit 152 acquires, as setting information, the time (capacity of video data) stored as the feature video set by the user when the feature video is detected from the video data by the feature quantity detection unit 153. .

The feature amount detection unit 153 includes live video data captured by the imaging apparatus 300 connected to the external video input unit 110 or video data already stored in the video data storage unit 150 and video data to be compared with the video data. A feature amount is detected based on the difference (change amount).
That is, in the present embodiment, the difference is calculated by comparing the RGB values when detecting the difference between the video data. Here, when comparing with RGB values, first, with respect to each pixel of frame 1 (r1, g1, b1) and frame 2 (r2, g2, b2) to be compared, the absolute value of the difference between the RGB values (ie, | r1 -R2 |, | g1-g2 |, | b1-b2 |). Next, the number of pixels in which each calculation result is equal to or higher than the predetermined sensitivity set is counted, and the ratio of the counted number of pixels to the total number of pixels in the frame is calculated. When the ratio is equal to or greater than the set threshold value, it is determined that there is a feature amount in the video data to be detected, and it is determined that the video data (image) has changed.
However, the method of calculating the pixel difference is not necessarily limited to the method of comparing the RGB values of each pixel. The method of comparing the HSV (Hue Saturation Value) value of each pixel, video data (image) A method of comparing the luminance value of each pixel by making it gray scale, or a method of executing edge processing on video data (image) and comparing the edge intensity of each pixel may be used.

Note that the sensitivity for determining the change in RGB value of each pixel and the threshold for determining the change in frame are set by the user.
Specifically, the user can set by inputting the sensitivity and the threshold on the input screen of software installed in the NVR 100 or the external connection device 600.
In addition, before starting monitoring, the user acquires a feature video from the sample video data by the feature video acquisition unit 154 and further generates a graph by the graph generation unit 155, and then the feature video and / or graph. Is displayed on the monitor 200 or the display or screen of the external connection device 600, and the sensitivity and threshold value can be adjusted.

The feature video acquisition unit 154 corresponds to the image extraction unit of the present invention. When the feature amount detection unit 153 detects a feature amount (change), the information acquired by the setting information acquisition unit 152, that is, the condition set by the user Feature video data is acquired according to (various information).
The graph generation unit 155 corresponds to the display data generation unit of the present invention, and changes over time of the pixels of the video data, for example, the ratio of the number of pixels that exceeds the predetermined sensitivity with respect to the total number of pixels of the frame. A graph showing a change or the like is generated.
By displaying the graph generated by the graph generation unit 155 on the monitor 200, the user can visually recognize the exact time when the feature amount is detected and the degree of the feature amount.

The video data list storage unit 156 corresponds to image data list generation means for generating an image data list in the present invention. Additional information associated with information (specific event) set by the marker setting unit 157, here, The feature video acquired by the feature video acquisition unit 154 marked by the mark and the graph generated by the graph generation unit 155 are further associated with the shooting time and the ID of the shooting device 300, and a list is generated and stored.
The marker setting unit 157 corresponds to the additional information setting unit of the present invention, and marks (sets a mark) the video data stored in the video data list storage unit 156 according to the mark selected by the user.
It should be noted that after the user confirms playback of the video data stored in the video data list storage unit 156, the user adds a new mark through the marker setting unit 157 or has already added the mark. The mark can be changed.

  The display control unit 158 reads the video data stored in the video data list storage unit 156 according to the input information from the input device 500 or the external connection device 600, and displays the display area ( For example, video data or the like is displayed on a display, a screen, or the like. The display control unit 158 corresponds to the reproduction control means of the present invention.

The external video output unit 160 is an interface for connecting the NVR 100 and the display device 200.
The LAN port 170 is an interface for connecting the NVR 100 and the external connection device 600 by wired communication means such as a LAN cable.
The wireless interface 180 is an interface for connecting the NVR 100 and the external connection device 600 by wireless communication means such as Wi-Fi or Bluetooth (registered trademark).

(Embodiment 1)
In the present embodiment, the same video captured by the imaging device 300 (“same video” includes other video data of the same subject captured with the same settings in addition to the video stored in the same video data file. In the case where there is a change in the file), the NVR 100 acquires the video as a feature video, marks it with the marker setting unit 157, and then lists it in the video data list storage unit 156. And memorize it.
That is, for example, as shown in FIG. 3, when a predetermined area is selected from an image captured by the imaging apparatus 300 in accordance with conditions set by the user (here, the worker's work area, work time, etc.) (broken line in the figure) The NVR 100 uses the feature image acquisition unit 154 to acquire, as a feature image, an image from when the worker enters the selected area until the worker exits (that is, an image in which the RGB value has changed), and the marker setting unit 157. Then, after marking in association with conditions set by the user such as the work area and work time of the worker, the feature videos are listed according to the work time and work name and stored in the video data list storage unit 156.

Further, in this case, the NVR 100 can store the time when the worker enters and exits the selected area simultaneously with the acquisition of the feature video, and can measure the work time (work efficiency). Further, the NVR 100 can graph the measured work time and the like by the graph generation unit 155.
The work time or the like can also be measured from the feature images stored in a list in the video data list storage unit 156.

In FIG. 3, the shape of the monitoring target area selected by the user is indicated by a rectangle (rectangle), but the shape of the selectable area is not necessarily limited to this, and the user can select a circle, an ellipse, a distorted polygon, or the like. You can also select an area with.
In addition, the user can select a plurality of monitoring target areas from the video imaged by the imaging device 300. For example, in FIG. 3, the user can select the area around the person at the center of the screen as the monitoring target area in addition to the area selected in the lower left of the screen (broken line in the figure). In this case, the NVR 100 independently executes processing related to image storage and the like for each selected area.
Note that the shapes of areas that can be selected as monitoring targets and the independent processing for each selected area when a plurality of monitoring target areas are selected are the same in the second and third embodiments.

Next, feature amount detection will be described with reference to FIG.
FIG. 4 is a graph showing a change over time in the ratio of the number of pixels that is equal to or higher than a predetermined sensitivity with respect to the total number of pixels in the frame.
In FIG. 4, the horizontal axis indicates time, and the vertical axis indicates the ratio of the number of pixels that is equal to or higher than a predetermined sensitivity to the total number of pixels in the frame.
A broken line in the figure indicates a threshold value for determining whether or not a feature amount is detected, and is set to 20% in this embodiment.

In the present embodiment, the absolute value of the difference between the RGB values is calculated for each pixel of the frame positioned before and after in time, and the ratio of the number of pixels that is equal to or greater than a predetermined sensitivity to the total number of pixels in the frame is calculated. This calculation is repeatedly executed by counting up the frames in time.
Then, it is determined that the feature amount has been detected (that is, the image has changed) when, for example, three or more frames having a ratio of the number of pixels equal to or higher than the predetermined sensitivity are equal to or greater than the threshold value.
In FIG. 4, first, at the time when the ratio of the number of pixels equal to or higher than the predetermined sensitivity is equal to or higher than the threshold, only two frames are equal to or higher than the threshold, and it is not determined that the feature amount has been detected. This is due to the fact that noise is included in the video data, and the feature amount is as long as the ratio of the predetermined number of frames, for example, 3 frames or more, and the number of pixels continuously exceeding the predetermined sensitivity does not exceed the threshold value. This is to prevent detection.

Next, a processing procedure until the video data is listed and stored in the network video recorder 100 according to the present embodiment will be described with reference to FIG.
The NVR 100 acquires video data to be monitored from the imaging device 300 or the video data storage unit 150 (S101).

The acquired video data is displayed on the monitor 200, and the user designates an area to be monitored which is a condition set by the user from the display screen using the input device 500 (S102).
When the entire area (area) displayed on the monitor 200 is to be monitored, the process in step S102 is not necessary.

When the user designates the monitoring target area (S102), the user sets a predetermined sensitivity for determining a change in the RGB value of each pixel and a threshold for determining a change in the frame (S103).
Note that the monitoring target area, sensitivity, and threshold value can also be set from the externally connected device 600 connected to the NVR 100.

The NVR 100 confirms whether the monitoring target area, sensitivity, and threshold are set, and whether there is monitoring setting (S104).
When the monitoring setting is not made (No in S104), the NVR 100 cannot start monitoring, and therefore continuously displays the input screen regarding the monitoring setting.
When the NVR 100 confirms that the setting item is input (S104 Yes), the NVR 100 starts monitoring.

The NVR 100 acquires the first frame of the video data to be monitored acquired in step S101 and the next frame in time (S105), and the absolute value of the RGB value difference for each pixel in the frame Is calculated (S106).
Next, the NVR 100 calculates the number of pixels in which each of the calculated absolute values is equal to or higher than the predetermined sensitivity set (S107), and whether the ratio of the number of pixels to the total number of pixels in the frame is equal to or higher than a set threshold value. It is determined whether or not (S108).
In step S108, the number of times that the ratio of the number of pixels calculated in step S107 to the number of pixels in the entire frame is equal to or greater than a threshold value is counted, and it is simultaneously determined whether or not the counted number is equal to or greater than a predetermined number. To do. In this case, when the ratio of the number of pixels calculated in step S107 to the number of pixels in the entire frame is smaller than a set threshold value, the counted number is reset.

When the ratio of the total number of pixels calculated in step S107 is less than the threshold (No in S108), and when the ratio of the total number of pixels calculated in step S107 is equal to or greater than the threshold However, if the number of times exceeding the threshold does not exceed the predetermined number (No in S108), the NVR 100 counts up the frame to acquire the next frame after the frame acquired in step S105. (S109), the processing from step S105 is executed again.
For example, if the frames F1 and F2 positioned before and after in time are acquired in step S105, the NVR 100 calculates the processing from step S105 to S108 in step S107 while counting up the frames in step S109. The number of times that the ratio of the number of pixels to the total number of pixels in the frame is equal to or greater than the set threshold value is counted, and this is repeated until the number of times becomes equal to or greater than a predetermined number (S108 Yes).
Note that the degree of frame count-up (+1, +2, +3... + N...) Executed in step S109 takes into account the contents of video data to be monitored and the processing performance of the image control recording apparatus. To decide.

When “Yes” is set in step S108, the NVR 100 trims the video data according to the information (storage time) set in the setting information acquisition unit 152, and sets the information (specific event, For example, marking is performed with a mark associated with the entry of the worker to the selected area), and further, associated with the time and the ID of the photographing apparatus 300, and is stored in the video data list storage unit 156 (S110).
When the NVR 100 lists and stores the video data (S110), the NVR 100 checks whether or not there is a stop command from the user or information set in the setting information acquisition unit 152, and determines whether or not to end the storage of the video data. (S111).
When the storage of the video data is terminated by a stop command from the user (S111 Yes), when the monitoring is terminated and the storage of the video data is continued (S111 No), the NVR 100 counts up the frame at step S109, The processing from step S105 is repeatedly executed.

Next, playback / display control of video data stored in a list will be described.
FIG. 6 shows a display example of a marker list stored in the video data list storage unit 156.
As shown in FIG. 6, the marker list is a list in which a list of video data associated with the shooting time and unique ID of the shooting apparatus 300 is marked with a predetermined mark, and is displayed on the monitor 200 as a marker list window 210. .

In the list box 211 of the marker list window 210, the video data stored in the video data list storage unit 156 is displayed in association with the marker 212, the date and time 213, and the unique ID 214 of the photographing apparatus 300.
The markers 212 are indicated by ★, ▼, ●, ◆, etc. in FIG.
★ indicates video data to be warned. For example, it is marked in the video data when an operator enters an area that should not be entered.
The mark ▼ is marked on video data serving as a reference for tact (tact time; process work time).
● and ◆ indicate markers set by the user according to the type of video data. For example, when the worker is working beyond the scheduled time in a specific area, the worker Is marked when a different operation is performed in a specific area.
The date and time 213 in the list box indicates the time when the image was taken by the image taking device 300 or the time stored in the video data storage unit 150 of the NVR 100.

In the example of FIG. 6, a warning check box 215 and a tact check box 216 are set as check boxes at the top of the marker list window 210.
For example, if the warning check box 215 is checked, the video data targeted for warning can be displayed in the list box, and if the check box is cleared, the video data targeted for warning is not displayed. Can be.
As with the warning check box 215, the tact check box 216 can be set to display / hide by checking or unchecking it.

At the bottom of the marker list window 210, a date designation button 217 and a Today button 218 are set so that video data can be narrowed down by date.
The user can narrow down the video data displayed in the list box 211 in a calendar format by selecting the date designation button 217. Further, by selecting the Today button 218, it is possible to narrow down the video data captured on that day (current date).

The user can display the video data on the monitor 200 by selecting the video data displayed in the list box 211.
FIG. 7 is a diagram illustrating an example of reproduction of video data selected in the list box.
As shown in FIG. 7, the video playback window 220 mainly includes a video data playback area 221, a marker list display area 222, and an operation area 223 for performing various operations.

The video data playback area 221 is an area for playing back and displaying video data selected by the user. As will be described later, the video data reproduction area 221 includes the video data captured by the imaging devices 300A to 300F and / or the video data stored in the video data storage unit 150 on the display screen as necessary. Playback can be performed in a screen display or a split screen display.
The marker list display area 222 is an area for displaying the aforementioned marker list, and shows which video data is being reproduced and displayed in the video data reproduction area 221 by highlighting the video data being reproduced.

The operation area 223 includes buttons and slide bars related to playback control (operation), and other setting buttons.
The button 224 related to playback control includes a start button for starting playback of video data, a stop button for stopping playback, and buttons such as fast forward and rewind. In addition, the user can reproduce from a desired position by sliding the slide bar 225 related to reproduction control with the input device 500.

Examples of the setting button 226 include a marker list display button 226 (1), a display area change button 226 (2), and a graph display button 226 (3) as shown in FIG.
The user can set display / non-display of the marker list by selecting the marker list display button 226 (1), and can display the video data reproduction area 221 by selecting the display area change button 226 (2). The area can be enlarged / reduced (full screen display / split screen display can be set).

When the graph display button 226 (3) is selected, a screen as shown in FIG. 8 is displayed. The user can reproduce an image from the selected point by selecting an arbitrary point (time point) from the displayed graph, for example, described with reference to FIG.
Conversely, when the user confirms an abnormality in the video that is being played back and displayed, the user can select the display area of the video data playback area 221 to display the video playback position with an arrow on the graph.
In addition, regarding the video playback window 220, a graph may be set up by selecting a display area of the video data playback area 221. Further, the displayed graph can be displayed on the full screen in the same manner as the reproduced video data.

Furthermore, the user selects a plurality of video data from the video data list displayed in the marker list display area 222 of the video playback window 220, and as shown in FIG. 9, according to the number of selected video data. The video data reproduction area 221 can be divided and reproduced.
That is, the user can reproduce a plurality of video data captured by the imaging apparatuses 300A to 300F at the same time in time synchronization.
For example, when a problem occurs in the work process, the work situation is reproduced and displayed on the monitor 200 by checking and displaying images taken from various angles, thereby reducing the time required to analyze the problem and the accuracy of the analysis. Can be improved.
In addition, even when a plurality of pieces of video data shot at different shooting times and different shooting apparatuses (300A to 300F) are selected by the user from the video data list, the video data is displayed according to the number of selected video data. The reproduction area 221 can be divided and reproduced and displayed.

Next, a processing procedure related to reproduction control of listed video data in the image storage / reproduction control apparatus according to the present embodiment will be described with reference to the flowchart of FIG.
When the NVR 100 receives a video data playback command from the user (Yes in S201), the NVR 100 reads the marker list from the video data list storage unit 156 and causes the display control unit 158 to display the marker list window 210 shown in FIG. S202).

  The NVR 100 displays the marker list window 210 until there is a video data selection command from the user (No in S203). When there is a video data selection command from the user (S203 Yes), the NVR 100 acquires the video data selected from the video data list storage unit 156 by the display control unit 158 (S204), and the video data is displayed on the monitor 200. Reproduction display is performed (S205).

After that, when the graph display button 226 (3) is selected by the user to analyze the video data (Yes in S206), the NVR 100 displays a graph associated with the selected video data by the display control unit 158. That is, for example, a graph showing a change over time in the ratio of the number of pixels that is equal to or higher than a predetermined sensitivity to the total number of pixels in the frame shown in FIG. 4 is displayed on the monitor 200 (S207).
Further, when the user selects an arbitrary point on the graph, for example, a location where a problem has occurred (an inflection point of the ratio of the number of pixels) or the like (Yes in S208), the NVR 100 is selected by the display control unit 158. The playback position is adjusted to the point (S209), and the video data is played back and displayed on the monitor 200 from the adjusted playback position (S210).

(Embodiment 2)
In this embodiment, a normal time image of the same subject to be compared stored in the video data storage unit 150 and a video to be monitored captured by the image capturing device 300 are set at a predetermined time interval, Alternatively, when a comparison is made at a predetermined time, a difference in video data is detected, and a frame in which the ratio of the number of pixels equal to or greater than a predetermined sensitivity to the total number of pixels in the frame is equal to or greater than a threshold, for example, 3 frames or more, The NVR 100 acquires the video to be monitored as a feature video, marks it with the marker setting unit 157, and stores it in the video data list storage unit 156 as a list.
For example, as illustrated in FIG. 11, when a user selects a predetermined area (predetermined screw insertion hole) from an image captured by the image capturing apparatus 300 in the screw insert processing operation, the NVR 100 stores the image data in the image data storage unit 150. Comparing the stored normal insert processing video as the comparison target with the video as the monitoring target captured by the imaging device 300 at a predetermined time interval or at a predetermined time, and detecting a difference in the video data In this case, a video whose feature value is detected based on the difference is acquired as a feature video, marked by the marker setting unit 157, listed by work time, work name, etc., and stored in the video data list storage unit 156. Remember.

Note that, for example, in the case where the screw insertion failure is limited to the case where the screw is not inserted into the screw insertion hole, the comparison target video stored in the video data storage unit 150 is displayed when the same subject is abnormal. (In this example, an image in which no screw is inserted).
In this case, the NVR 100 compares the image to be monitored captured by the image capturing device 300 with a predetermined time interval or a predetermined time, detects a difference in the video data, and determines a predetermined amount for the total number of pixels of the frame. If the ratio of the number of pixels equal to or greater than the sensitivity is equal to or greater than the threshold value, it is determined that no abnormality has occurred due to the screw not being inserted. Conversely, when the ratio of the number of pixels that is equal to or greater than the predetermined sensitivity to the total number of pixels in the frame is less than the threshold value, the NVR 100 sets the image to be monitored as an abnormal image (that is, an image in which no screw is inserted). ) And marked by the marker setting unit 157, and then stored in a list in the video data list storage unit 156.

  In addition, by connecting the NVR 100 with a sensor installed as an existing facility, the time when the abnormality occurred is specified by the signal from the sensor, and images before and after that time are acquired and marked by the marker setting unit 157 In the above, it is also possible to make a list by work time, work name, etc., and store it in the video data list storage unit 156. In addition, the time at which an abnormality has occurred is identified in the same manner as described above based on the change in the image capturing the sensor, that is, the change from the normal display to the abnormal display of the sensor, and the images before and after that time are acquired. It can also be stored in the data list storage unit 156.

Next, feature amount detection in the second embodiment will be described with reference to FIG.
FIG. 12A is a graph showing changes in RGB values over time. In FIG. 12A, the horizontal axis represents time, and the vertical axis represents RGB values.
FIG. 12B is a graph showing the difference of the change over time of the RGB value. In FIG. 12B, the horizontal axis indicates time, and the vertical axis indicates the difference in RGB values.
FIG. 12C is a graph showing a change over time in the ratio of the number of pixels with a predetermined sensitivity or more with respect to the total number of pixels in the frame. In FIG. 12C, the horizontal axis represents time, and the vertical axis represents the ratio of the number of pixels with a predetermined sensitivity or more to the total number of pixels in the frame.
Note that the RGB values in FIGS. 12A and 12B indicate any one of the R value, the G value, and the B value (for example, the G value). A broken line in FIG. 12C indicates a threshold value for determining whether or not a feature amount is detected. In this embodiment, the broken line is set to 20%.

In the present embodiment, a frame at a predetermined time Tn is acquired for the normal video data and the video data captured by the imaging device 300, and the RGB value of each pixel in the frame is calculated (FIG. 12A). Next, the absolute value of the difference between the RGB values of each video data is calculated (FIG. 12B), and the ratio of the number of pixels equal to or higher than a predetermined sensitivity to the total number of pixels in the frame is calculated (FIG. 12C). This calculation process is executed at predetermined time intervals (Tn + 1−Tn, Tn + 2−Tn + 1,...).
Then, it is determined that the feature amount has been detected when the ratio of the number of pixels equal to or greater than the predetermined sensitivity is equal to or greater than the threshold, for example, 3 frames or more (that is, the video data (image) changes). Determined).

Next, processing until the video data is listed and stored by the network video recorder 100 according to the present embodiment will be described with reference to FIG.
The NVR 100 acquires video data to be monitored from the imaging device 300 or the video data storage unit 150 (S301).

The acquired video data is displayed on the monitor 200, and the user uses the input device 500 to specify conditions set by the user on the display screen, here, the area to be monitored (S302).
If the entire area (area) displayed on the monitor 200 is to be monitored, the process in step S302 is not necessary.

When the user designates a monitoring target area (S302), the user sets a predetermined sensitivity for determining a difference between RGB values of each pixel and a threshold for determining a frame difference (S303).
Note that the monitoring target area, sensitivity, and threshold value can also be set from the externally connected device 600 connected to the NVR 100.

The NVR 100 checks whether the monitoring target area, sensitivity, and threshold are set, and whether or not monitoring is set (S304).
When the monitoring setting is not made (No in S304), the NVR 100 cannot start monitoring, and continuously displays the monitoring setting input screen.

When the NVR 100 confirms that the setting item has been input (S304 Yes), the NVR 100 displays a list of video data stored in the video data storage unit 150 on the monitor 200.
When the user selects video data to be compared (S305), the NVR 100 acquires the video data to be compared (for example, normal video data) from the video data storage unit 150 (S306) and starts monitoring. To do.

The NVR 100 acquires the first frame of the video data to be monitored acquired in step S301 (S307).
Next, the NVR 100 acquires the first frame of the video data to be compared acquired in step S306 (S308).
When the video data frame to be monitored and the video data frame to be compared are acquired (S307 and S308), the absolute value of the RGB value difference between the frames of each pixel in each frame is calculated ( S309).

Next, the NVR 100 calculates the number of pixels in which each of the calculated absolute values is equal to or greater than a predetermined sensitivity (S310), and whether or not the ratio of the number of pixels to the number of pixels in the entire frame is equal to or greater than a set threshold value. Is determined (S311).
In step S311, the number of times that the ratio of the number of pixels calculated in step S310 to the number of pixels in the entire frame is equal to or greater than a set threshold value is counted, and whether or not the counted number is equal to or greater than a predetermined number is determined. Judge at the same time. In this case, when the ratio of the number of pixels calculated in step S310 to the total number of pixels in the frame is smaller than a set threshold value, the counted number is reset.

  When the ratio of the number of pixels calculated in step S310 is less than the set threshold (No in S311), the threshold of the ratio of the number of pixels calculated in step S310 to the total number of pixels is set. Even in the case where the number of times becomes equal to or greater than the threshold, if the number of times equal to or greater than the threshold does not exceed the predetermined number (No in S311), the NVR 100 sets the time elapsed for a predetermined time from the time the frame was acquired in step S307. After resetting (S312), the processing from step S307 is executed again.

For example, when a frame of video data to be monitored at time Tn set in step S307 is acquired, and then a frame of video data to be compared at time Tn is acquired (S308), and step When the processes of S309 and S310 are executed and “No” is selected in step S311, the NVR 100 resets the time Tn + 1 obtained by elapse of the set time of frame acquisition for a predetermined time in step S312, and step S307. The process of .about.S310 is repeated until the ratio of the number of pixels calculated in step S310 to the number of pixels in the entire frame is equal to or greater than a threshold value, and is repeated until the number is equal to or greater than a predetermined number (S311 Yes).
In step S312, the length of time to elapse is determined in consideration of the content of video data to be monitored and the processing performance of the image control recording apparatus.

  Then, when “Yes” is set in step S311, the NVR 100 trims the video data according to the information (storage time) set in the setting information acquisition unit 152, and sets the information (specific event, For example, marking is performed with a mark associated with a screw insertion defect), and the time and the ID of the image capturing apparatus 300 are associated with each other, and a list is created and stored in the video data list storage unit 156 (S313).

When the NVR 100 lists and stores the video data (S313), the NVR 100 checks whether there is a stop command from the user or information set in the setting information acquisition unit 152, and determines whether or not to end the storage of the video data. (S314).
When the storage of video data is terminated due to a stop command or the like from the user (S314 Yes), when the monitoring is terminated and the storage of video data is continued (S314 No), the NVR 100 sets the frame acquisition set time in step S312. After a predetermined time has elapsed, the processes from step S307 are repeatedly executed.

(Embodiment 3)
In the present embodiment, the NVR 100 calculates the RGB value of the pixel in the frame at a predetermined time interval in the video imaged by the imaging apparatus 300, and the calculated RGB value exceeds a predetermined sensitivity set in advance. Are counted. When the counted number of pixels is equal to or greater than the threshold, the NVR 100 acquires a video that is equal to or greater than the threshold as a feature video, marks it with the marker setting unit 157, and then lists the video data in the video data list storage unit 156. And memorize it.
For example, as shown in FIG. 14, when a red warning light and a green work light are installed in the machine, a red warning is used as a predetermined area which is a condition set by the user from an image shot by the shooting device 300. When the light lamp and the green work lamp are selected, the NVR 100 calculates the R value and the G value from the captured video data, and the calculated R value and G value are predetermined R values and G values. The number of pixels that are equal to or higher than the sensitivity is counted, and an image in which the ratio of the counted number of pixels to the total number of pixels in the frame is equal to or greater than a threshold is acquired as a feature image and marked by the marker setting unit 157 The video data list storage unit 156 stores the time and work names as a list.

Next, feature image detection will be described with reference to FIG.
FIG. 15A is a graph showing changes in RGB values over time. In FIG. 15A, the horizontal axis represents time, and the vertical axis represents RGB values.
FIG. 15B is a graph showing a change over time in the ratio of the number of pixels with a predetermined sensitivity or more with respect to the total number of pixels in the frame. In FIG. 15B, the horizontal axis represents time, and the vertical axis represents the ratio of the number of pixels with a predetermined sensitivity or more to the total number of pixels in the frame.
Note that the RGB values in FIGS. 15A and 15B indicate one of the R, G, and B values (for example, the G value). A broken line in FIG. 15B indicates a threshold value for determining whether or not a feature amount is detected, and is set to 20% in the present embodiment.

In this embodiment, a frame at a predetermined time Tn is acquired, and the R value, G value, and B value of each pixel of the frame are calculated (FIG. 15A).
Next, for each pixel, it is determined whether the R value or the G value is equal to or higher than a predetermined sensitivity, and the number of pixels that are equal to or higher than the predetermined sensitivity is counted. Is calculated with respect to the total number of pixels (FIG. 15B). This calculation process is executed at predetermined time intervals (Tn + 1−Tn, Tn + 2−Tn + 1,...).

Next, processing until the video data is listed and stored by the network video recorder 100 according to the embodiment of the image storage / playback control apparatus of the present invention will be described with reference to FIG.
The NVR 100 acquires video data to be monitored from the imaging device 300 or the video data storage unit 150 (S401).
The acquired video data is displayed on the monitor 200, and the user uses the input device 500 as a monitoring target area, which is a condition set by the user from the display screen, and a red warning light and a green work installed in the machine. A lamp is designated (S402).

When the user designates the monitoring target area (S402), a predetermined sensitivity for determining whether or not the R value or G value of the pixel is equal to or greater than a predetermined value and a threshold value for determining a frame difference are set. (S403).
Note that the setting of the monitoring target area, sensitivity, and threshold value can also be performed from the external connection device 600 connected to the NVR 100.

The NVR 100 checks whether the monitoring target area, sensitivity, and threshold are set, and whether or not monitoring is set (S404).
When the monitoring setting is not made (No in S404), the NVR 100 cannot start monitoring, and continuously displays the input screen related to the monitoring setting.
When the NVR 100 confirms that the setting item is input (S404 Yes), the NVR 100 starts monitoring.

The NVR 100 acquires the first frame of the video data to be monitored acquired in step S401 (S405), and calculates the R value, G value, and B value of each pixel in the frame (S406).
Next, the NVR 100 calculates the number of pixels in which the R value is equal to or higher than the predetermined sensitivity and the number of pixels in which the G value is equal to or higher than the predetermined sensitivity (S407), and the ratio of the number of pixels to the total number of pixels in the frame is set. It is determined whether or not it is equal to or greater than the threshold value (S408).

The ratio of the number of pixels with the R value equal to or higher than the predetermined sensitivity to the total number of pixels of the frame is less than the threshold, and the ratio of the number of pixels with the G value equal to or higher than the predetermined sensitivity to the total number of pixels is set. When it is less than the threshold (No in S408), the NVR 100 trims the video data according to the information (storage time) set in the setting information acquisition unit 152, and sets the information (specific event) set in the setting information acquisition unit 152. Here, marking is performed with a mark associated with the machine setup change), and further associated with the time and the ID of the photographing apparatus 300, and is stored in the video data list storage unit 156 (S409).
When the NVR 100 lists and stores the video data (S409), the NVR 100 checks the presence / absence of a stop command from the user or the information set in the setting information acquisition unit 152, and determines whether to end the storage of the video data. (S410).
When the storage of video data is terminated due to a stop command or the like from the user (S410 Yes), when monitoring is terminated and the storage of video data is continued (S410 No), the NVR 100 sets the frame acquisition time in step S416. The processing from step S405 is repeatedly executed with the passage of time.

In step S408, either the ratio of the number of pixels where the R value is equal to or higher than the predetermined sensitivity to the total number of pixels, or the ratio of the number of pixels where the G value is equal to or higher than the predetermined sensitivity to the total number of pixels of the frame. Is equal to or greater than the threshold value (S408 Yes), the NVR 100 determines which of the R value and the G value is equal to or greater than the threshold value (S411).
As a result of the determination, when the ratio of the number of pixels in which the R value is equal to or higher than the predetermined sensitivity to the total number of pixels in the frame is equal to or greater than the threshold (S411 R value), the NVR 100 issues a warning (S412).
In step S411 in FIG. 16, the processing related to noise countermeasures applied in step S108 in the first embodiment and step S310 in the second embodiment is not incorporated, but the processing is executed as in the first and second embodiments. May be.

When the NVR 100 issues a warning (S412), the NVR 100 trims the video data according to the information (storage time) set in the setting information acquisition unit 152, and sets the information (specific event, here in the setting information acquisition unit 152). , Marking with the mark associated with the machine failure), further associating with the time and the ID of the photographing apparatus 300, listing them, and storing them in the video data list storage unit 156 (S413).
When the NVR 100 lists and stores the video data (S413), the NVR 100 checks the presence / absence of a stop command from the user or the information set in the setting information acquisition unit 152, and determines whether to end the storage of the video data. (S410).
When the storage of video data is terminated due to a stop command or the like from the user (S410 Yes), when monitoring is terminated and the storage of video data is continued (S410 No), the NVR 100 sets the frame acquisition time in step S416. The processing from step S405 is repeatedly executed with the passage of time.

In step S411, when it is determined that the ratio of the number of pixels with the G value equal to or higher than the predetermined sensitivity to the total number of pixels in the frame is equal to or higher than the threshold (S411 G value), the NVR 100 displays the time management unit 151. The current time or the time associated with the video data is acquired.
Although not shown in FIG. 16, in step S411, the processing is continued after it is determined that the ratio of the number of pixels with the G value equal to or higher than the predetermined sensitivity to the total number of pixels in the frame is equal to or higher than the threshold. In step S408 or step S411, when the ratio of the number of pixels in which the G value is equal to or higher than the predetermined sensitivity to the number of pixels in the entire frame is less than the set threshold value (S408 No or S411 R value), the NVR 100 also The current time or the time associated with the video data is acquired from the time management unit 151. As described above, the NVR 100 can automatically calculate the operation rate of the machine by acquiring the lighting time and the extinguishing time of the work lamp. The calculated operation rate is stored in association with the video data.

When the NVR 100 acquires the time (S414), the NVR 100 trims the video data according to the information (storage time) set in the setting information acquisition unit 152, and sets the information (a specific event, here, set in the setting information acquisition unit 152). Marking is performed with a mark associated with (normal operation of the machine), and further associated with the time and the ID of the photographing apparatus 300, and is stored in the video data list storage unit 156 (S415).
When the NVR 100 lists and stores the video data (S415), the NVR 100 checks the presence / absence of a stop command from the user or the information set in the setting information acquisition unit 152, and determines whether to continue monitoring (S410). ).
When the storage of video data is terminated due to a stop command or the like from the user (S410 Yes), when monitoring is terminated and the storage of video data is continued (S410 No), the NVR 100 sets the frame acquisition time in step S416. The processing from step S405 is repeatedly executed with the passage of time.

As described above, according to the NVR according to the embodiment of the present invention, the difference between the captured videos (video data) is detected to extract the video based on the condition specified by the user, and the extracted video By marking a video with a mark associated with information (event) set by the user, and making a list and storing it, the user can easily check a characteristic video.
Therefore, by using the NVR according to the embodiment of the present invention, for example, in a production line monitoring system, it is possible to easily detect and confirm defects in the production line, and to shorten the work man-hours involved in the failure analysis. Can do.

  DESCRIPTION OF SYMBOLS 100 ... Network video recorder, 110 ... External video input part, 120 ... Memory slot, 130 ... Connection interface, such as USB, 140 ... User interface, 150 ... Video data storage part, 151 ... Time management part, 152 ... Setting information acquisition part, 153... Feature quantity detection unit, 154... Feature image acquisition unit, 155... Graph generation unit, 156... Video data list storage unit, 157. Port, 180 ... wireless interface, 200 ... display device, 300 ... photographing device, 400 ... external storage device, 500 ... input device, 600 ... external connection device.

Japanese Patent Publication No. 2006-197223

Claims (10)

Image extracting means for extracting an image from a monitoring target image obtained by photographing the monitoring target in accordance with conditions set by the user with respect to the worker's work area or work time ;
The operator's working area, the working time, or associate the mark associated with a particular event set by the user to the images extracted by the image extracting unit for work and generate and store device stores a list List generating means for
Read the list stored in the storage device, along with displays on the display device, wherein when the front Kemah over click from the list displayed on the display device is selected to identify the image associated with the mark, Playback control means for controlling playback and display on the display device ;
An image storage / reproduction control apparatus. The image storage / reproduction control device according to claim 1,
The image extraction / reproduction control apparatus extracts the image based on a change amount between the image stored in advance and the monitoring target image. The image storage / playback control device according to claim 2,
An image storage / reproduction control device in which the previously stored image is a normal or abnormal image of the same subject. The image storage / playback control device according to claim 2,
The image storage / reproduction control device, wherein the pre-stored image and the monitoring target image are images located before and after the same video in time. The image storage / reproduction control device according to claim 1,
The list generation means associates the photographing time of the image extracted by the image extraction means together with the mark with the image, generates and stores a list, and the reproduction control means stores a list to be displayed on the display device by a date. An image storage / reproduction control device for displaying a button for narrowing down on the display device. The image storage / reproduction control device according to any one of claims 2 to 5,
An image storage / reproduction control device having display data generation means for generating display data for displaying a change over time of the monitoring target image. The image storage / reproduction control device according to any one of claims 1 to 6,
An image storage / reproduction control device for controlling the display means so that the reproduction control means divides a display screen and reproduces and displays the plurality of images on the same display screen. An image extracting step in which an image extracting unit extracts an image from a monitoring target image obtained by photographing the monitoring target according to a condition set by a user with respect to a worker's work area or work time ;
List generating means, the operator's working area, the working time, or associate the mark associated with a particular event set by the user to the images extracted by the image extracting step for work and generates a list A list generation step of storing in the storage device ;
Reproduction control means reads the list stored in the storage device, along with displays on the display device, when the display front KOR over click from the list displayed on the device has been selected, associated with the mark A reproduction control step for specifying an image and controlling the reproduction and display on the display device;
An image storage / reproduction control method in an image storage / reproduction control apparatus. An image storage / reproduction control device according to any one of claims 1 to 7,
A photographing device for photographing an image stored in the image storage / reproduction control device;
An input device for pre-enter the Kijo matter and the event,
An image storage reproduction control system.   A program for causing a computer to function as each unit of the image storage / playback control device according to claim 1.

Images