JP2017098897A - Information processing system, information processing device and method of controlling the same, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology capable of performing all analysis processing even in a case where realtime image analysis that matches photographing cannot be completed.SOLUTION: An information processing system comprises a photographing device, a storage device, and an information processing device. The photographing device comprises: photographing means that photographs an image; and first analysis means that tries to analyze the image in response to the photographing of an image by the photographing means. The storage device stores the image photographed by the photographing means and analysis information indicating analysis contents by the first analysis means, of the image while being associated with each other. The information processing device comprises: determination means that refers the analysis information to determine an image that has not analyzed yet; and second analysis means that performs analysis for the image determined that it has not analyzed yet.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing apparatus, a control method thereof, and a computer program.

  The monitoring and monitoring camera system includes an imaging device, a recording device that records video output from the imaging device, and a display device that displays video from the imaging device or video recorded on the recording device. For example, in Patent Document 1, these devices are connected via a network, the imaging device directly writes the video to the recording device, and the display device reads the video directly from the recording device, so that no load is imposed on management. The system is described. Further, Patent Document 2 detects a video when an abnormality occurs by image recognition, records abnormal information at the same time as recording the video, and refers to a list of the abnormal information to quickly search for a video when the abnormality occurs. A system is described that allows it to be done.

JP 2005-184214 A JP-A-11-32321

  However, in the case of a system that performs video analysis processing at the same time as shooting with an imaging device and records video data and analysis data in a recording device connected via a network, the video analysis processing may be heavy depending on the subject. When the load of video analysis processing performed by the imaging device increases, the analysis processing cannot be processed in real time, and recording to a recording device connected via a network is not in time, and analysis data may be lost.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that enables all analysis processing to be performed even when analysis of a real-time image in accordance with shooting has not been completed. And

In order to achieve the above object, an information processing system according to the present invention comprises the following arrangement. That is,
An information processing system including an imaging device, a storage device, and an information processing device,
The imaging device
Photographing means for photographing an image;
A first analysis unit that attempts to analyze the image in response to the image being captured by the imaging unit;
The storage device stores an image captured by the imaging unit in association with analysis information indicating analysis content of the first analysis unit of the image,
The information processing apparatus includes:
Determination means for determining an image that has not been analyzed with reference to the analysis information;
Second analysis means for analyzing an image determined to have not been analyzed.

  According to the present invention, it is possible to provide a technique that makes it possible to perform all the analysis processing even when the real-time image analysis in accordance with the photographing cannot be completed.

The block diagram which shows the structure of a network camera system. The block diagram which shows the hardware structural example of a video analysis server apparatus. It is a figure which shows the recording folder structure of a recording device. The figure which showed the correspondence of a recorded image and analysis data. The figure which showed the content of the image processing result information. The flowchart which shows operation | movement of a video analysis server apparatus. The figure which shows the recording folder structure of a recording device.

  Hereinafter, the present invention will be described in detail based on the embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<Embodiment 1>
(Network camera system)
FIG. 1 is a block diagram showing a system configuration of a network camera system (information processing system) according to the first embodiment (Embodiment 1) of the present invention. In the network camera system, a network camera 100, a network storage 200, a video analysis server device 300, and a video information display device 400 are connected by a LAN 500 that is a network line.

  The network camera 100 is an imaging device (photographing device), and has a function of performing video analysis processing such as moving object detection, moving object tracking, human body detection, face recognition, and object detection on an image captured at the same time as photographing a subject. . The network storage 200 is a recording device that records video, and analysis data that has been subjected to video analysis processing simultaneously with the video shot by the network camera 100 is recorded via the LAN 500. The video analysis server device 300 is an information processing device that analyzes video. The video analysis server apparatus 300 includes a CPU or a GPU having high-speed calculation capability, and uses this to perform video analysis processing with capabilities and functions that exceed the video analysis processing performed inside the network camera 100. be able to. The video analysis server apparatus 300 can be realized by an information processing apparatus such as a PC (personal computer), an embedded system, or a tablet terminal.

  The video information display device 400 performs playback by superimposing video data recorded in the network storage 200 and analysis data. Furthermore, it has a function of an input means for performing a video search operation such as an event scene based on the analysis data. The video information display device 400 can also be realized by an information processing device such as a PC or a tablet terminal.

  In the present embodiment, an example in which each of the above devices is connected to the LAN 500 by a wired cable will be described. However, if the devices are connected in a protocol manner, the physical form is not particular. For example, when the video information display device 400 is realized by a tablet terminal that does not have a wired cable interface, the video information display device 400 may be configured to be connected wirelessly (for example, wireless LAN or Bluetooth (registered trademark)). Further, the number of network cameras 100, network storages 200, video analysis server devices 300, and video information display devices 400 connected to the LAN 500 is not limited to one as shown in the example of FIG. For example, there may be a plurality if it can be identified by an address or the like. Alternatively, all or part of these devices may be realized by the same general-purpose information processing device.

Network Camera As shown in FIG. 1, the network camera 100 includes an imaging sensor unit 101, a development processing unit 102, a video encoding unit 103, a video analysis processing unit 104, and a LAN I / F unit 105. The imaging sensor unit 101 is realized by an imaging element such as a CMOS that converts a light image formed on the imaging surface into a digital electric signal by photoelectric conversion. The development processing unit 102 performs predetermined pixel interpolation and color conversion processing on the digital electrical signal obtained by photoelectric conversion from the imaging sensor unit 101, and generates a digital image represented by a color space such as RGB or YUV. To do. Further, the development processing unit 102 performs predetermined calculation processing using the developed digital image, and performs image processing such as white balance, sharpness, contrast, and color conversion based on the obtained calculation result. The video encoding unit 103 compresses and encodes the digital video signal from the development processing unit 102. In the video encoding unit 103, digital video signal compression, frame rate setting, and the like are performed for distribution. The compression method for distribution here is, for example, a predetermined MPEG4, H.264, or the like. H.264, based on standards such as MJPEG or JPEG. Further, video data such as mp4 or mov format is also filed.

  The video analysis processing unit 104 uses the video from the development processing unit 102 to detect a moving object by, for example, a background difference method. Tracking of the moving object is performed by assigning a unique ID to the one identified from the positional relationship between the frames of the detected moving object. Hereinafter, a moving object to which an ID is assigned is referred to as an “object”. The video analysis processing unit 104 assigns attributes such as a person (male and female), an animal, a non-moving object, and the like to each object from the shape feature and behavior feature patterns. Further, a passing event in which the locus of the object passes a specific place on the screen and a leaving event in which the object stays at the same place on the screen for a predetermined time or more are detected. As a result of these processes, an analysis data file is created from the events and objects obtained. Furthermore, a processing result information file describing information on the results of these analysis processes is also created. This processing result information is analysis information indicating the analysis content of the captured image. Specifically, for which video file, what type of analysis, up to which frame, and further up to which part of the screen. Information describing whether analysis processing has been performed is described. The process of the video analysis processing unit 104 is not limited to the process exemplified here, and any process that analyzes video and creates analysis data such as inter-frame difference method or tracking of an object using an optical flow may be used. Further, the object attributes are not limited to the present method, and any method may be used as long as the method provides attributes such as attribute assignment by specific object detection processing using edge features.

  The LAN I / F unit 105 is a network interface for connecting to the LAN 500. The LAN I / F unit 105 constructs a network file system such as NFS or CIFS in cooperation with the LAN I / F unit 201 of the network storage 200. NFS is an abbreviation for Network File System, and CIFS is an abbreviation for Common Internet File System.

Network Storage The network storage 200 is a storage device that includes a LAN I / F unit 201, a video storage unit 202, and an analysis data storage unit 203. The LAN I / F unit 201 is a network interface for connecting to the LAN 500. The LAN I / F unit 201 constructs a network file system in cooperation with the LAN I / F unit 105 of the network camera 100 or the LAN I / F unit 301 of the video analysis server device 300. The video storage unit 202 records the video data file encoded and filed by the video encoding unit 103 of the network camera 100 using a network file system. The analysis data storage unit 203 records the analysis data file and the processing result information file, which have been subjected to video analysis processing by the video analysis processing unit 104 of the network camera 100 and converted into a file, using a network file system.

Video Analysis Server Device The video analysis server device 300 includes a LAN I / F unit 301, an acquisition unit (analysis processing result information acquisition unit) 302, a video decoding unit 303, a video analysis processing unit 304, and a video encoding unit 305. Is provided. The LAN I / F unit 301 is a network interface for connecting to the LAN 500, and constructs a network file system in cooperation with the LAN I / F unit 201 of the network storage 200. The acquisition unit 302 is a functional element that acquires information on the analysis processing result. Specifically, the acquisition unit 302 acquires the processing result information file recorded in the analysis data storage unit 203 of the network storage 200 using the network file system, and reads the analysis processing result information. Then, based on the read analysis processing result information, the video decoding unit 303 is instructed to acquire the corresponding video data file from the video storage unit 202 of the network storage 200. Further, the video analysis processing unit 304 is instructed to perform video analysis processing on the video data decoded by the video decoding unit 303.

  The video decoding unit 303 decompresses and decodes the video data file acquired by the network file system from the video storage unit 202 of the network storage 200 to obtain a digital video. The decoded digital video is output to the video analysis processing unit 304. The video analysis processing unit 304 basically has the same function as the video analysis processing unit 104 of the network camera 100. However, compared to resources that can be implemented in an embedded device such as the network camera 100, functions such as PCs and workstations that have analysis and calculation capabilities without restrictions can be implemented. The video encoding unit 305 re-encodes the video data decoded by the video decoding unit 303 and subjected to the analysis processing by the video analysis processing unit 304 to form a file, and again, the video storage unit of the network storage 200 202.

(Hardware configuration of video analysis server device)
FIG. 2 is a block diagram illustrating a hardware configuration example of the video analysis server device 300. In FIG. 2, a CPU 990 is a central processing unit and controls the overall operation of the video analysis server apparatus 300 in cooperation with other components based on a computer program. The ROM 991 is a read-only memory and stores basic programs, data used for basic processing, and the like. A RAM 992 is a writable memory and functions as a work area for the CPU 990.

  The external storage drive 993 can access a recording medium, and can load a computer program and data stored in a medium (recording medium) 994 such as a USB memory into this system. The storage 995 is a device that functions as a large-capacity memory such as an SSD (solid state drive). The storage 995 stores various computer programs and data.

  The operation unit 996 is a device that receives an instruction and a command input from a user, and corresponds to a keyboard, a pointing device, a touch panel, and the like. The display 997 is a display device that displays a command input from the operation unit 996 and a response output of the video analysis server device 300 in response thereto. An interface (I / F) 998 is a device that relays data exchange with an external device. The system bus 999 is a data bus that manages the flow of data in the video analysis server device 300.

  In addition, it can also be comprised as an alternative of a hardware apparatus by the software which implement | achieves a function equivalent to the above each apparatus. The video information display device 400 can also be realized by a hardware configuration equivalent to that of the video analysis server device 300.

(Folder structure)
FIG. 3 is a diagram showing a folder structure when a video data (mp4) file, an analysis data file, and a processing result information file are recorded in a hierarchy in the video storage unit 202 and the analysis data storage unit 203 of the network storage 200. . At the highest level (Layer 0 in FIG. 3), a folder is created for each network camera 100 that has captured the video to be recorded. Next, when the video data file and the analysis data file from the same network camera 100 are recorded, one folder of Layer 1 and Layer 2 is created. Each folder name is a number from 000 to 999 in the order of creation. Next, the video data file and the analysis data file are recorded in Layer 3 of the lowest layer. The video data file and analysis data file are created as one file for each preset number of frames. For example, in the case of a 10 fps image, an image for 10 frames, that is, one second is created as one file. Further, a processing result information file “INFO” in which analysis processing result information for the video data and analysis data recorded in this folder is recorded. Thereafter, when adding a file, if the number of files of the folder number with the largest number in Layer 2 is smaller than the upper limit number of layers according to a preset upper limit number of layers, the file is recorded in the folder. On the other hand, if the upper limit number of layers is exceeded, a new folder is created in Layer 2 and the file is recorded. However, if the number of folder names in Layer 2 exceeds 999, a new folder is created in Layer 1 and a folder is created and a file is recorded as described above. When the number of Layer1 file names exceeds the upper limit 999, the Layer1 000 folder is deleted and recorded in the 000 folder in the same manner. Thereafter, the recording is repeated after deleting in the same manner.

  In FIG. 3, ten frames of 10 fps, that is, one second of video is used as one file. In the example, six files are recorded in one folder of Layer 2 and a folder of Layer 2 is created every 6 seconds.

(file organization)
FIG. 4 is a file configuration diagram when video data is recorded as an mp4 file and analysis data is recorded as a linked file. The “Movie Header” of the video data file has an MP4BOX structure. The analysis data is obtained by arranging summary information at the head and continuously arranging analysis data created by the video analysis processing unit 304 of the network camera 100 thereafter. The video data file and analysis data file having the configuration shown in FIG. 4 are recorded with the same name and different extensions. Thereby, the video data and the analysis data can be associated with each other. Furthermore, summary information is created and updated when the analysis data file is recorded.

(Processing result information)
FIG. 5 is a diagram showing the contents of a processing result information file that is simultaneously created when the video analysis processing unit 104 of the network camera 100 performs video analysis processing simultaneously with shooting and creates analysis data. FIG. 5 exemplifies processing result information for six pieces of video data having file names “M0001.mp4” (51) to “M0006.mp4” (56). This processing result information is recorded as a file “INFO” in the folder shown in FIG. The processing result information describes the content and progress of video analysis processing performed on video data recorded in the same folder. In FIG. 5, the completion of the process is indicated by “completed”. For example, reference numeral 51 in FIG. 5 denotes processing for moving object detection, ID assignment for tracking a moving object, human body detection, animal detection, unmoving object detection for detecting leaving, etc. for video data “M0001.mp4”. Indicates that the process is complete.

  On the other hand, if the processing is not in time, and if the next video data is processed by rounding up the processing, it is up to which frame (or space area) on the screen. To be recorded.

  For example, the motion detection and motion ID assignment processing has been completed for 10 frames of video data of “M0004.mp4” (54) full HD size (1920 × 1080). On the other hand, the human body detection process is up to 900 × 450 for the sixth frame number, the animal detection process is up to 500 × 250 for the fifth frame number, and the unmoving object detection process is 0 × 0. It is shown that. Furthermore, it is indicated that the analysis data “M0005.meta” that should be created was not created for the video data “M0005.mp4” (55).

  As shown in FIG. 5, in the processing result information, the analysis content of the captured image is stored for each type of analysis. For this reason, the video analysis server device 300 receives the file “INFO” having the processing result information, determines a captured image that has not been analyzed for each type of analysis, and has not yet analyzed the captured image. A kind of analysis can be performed. Further, when the analyzed frame range is stored in the processing result information file by the number of analyzed frames, the video analysis server apparatus 300 completes the analysis by analyzing frames other than the number of frames. be able to. In addition, when a spatial region that has been analyzed in the captured image is stored in the processing result information file, the video analysis server device 300 identifies a spatial region that has not been analyzed in the captured image based on this spatial region. Thus, the analysis can be completed.

(Processing procedure)
Details of the operation of the network camera system according to the present embodiment having the above-described configuration will be described. In the present embodiment, the network camera 100 attempts image analysis in real time according to shooting, and stores the shot image, analysis data, and processing result information in the network storage 200. Then, the video analysis server device 300 refers to the processing result information to determine a captured image that has not been analyzed, and analyzes the captured image that has been determined to have not been analyzed. For this reason, even if the analysis in the network camera 100 is not completed, it is possible to complete all the analysis processes by complementing the analysis in the video analysis server device 300.

  FIG. 6 is a flowchart showing the operation of the video analysis server device 300 of the network camera system according to the present embodiment. Each step described below is executed based on the control of the CPU 990 of the video analysis server device 300.

  First, video analysis processing is performed in response to the start of shooting with the network camera 100. Video data files encoded and filed by the video encoding unit 103 are sequentially recorded in the video storage unit 202 of the network storage 200 via the LAN 500. At the same time, the video analysis processing unit 104 performs video analysis processing, and the created analysis data is sequentially recorded in the analysis data storage unit 203 of the network storage 200 via the LAN 500. When recording starts, the video analysis server device 300 receives a notification from the network camera 100 (S1).

  Upon receiving the notification, the video analysis server device 300 polls the first folder of Layer 2 shown in FIG. 3 in the analysis data storage unit 203 of the network storage 200, and refers to the recorded video data file and the recording status of the analysis data file. (S2). Then, it is determined whether or not the recording in the folder is finished and finally the processing result information file is recorded (S3). If it is confirmed that the processing result information file has been recorded (YES in S3), the process proceeds to S4. If not confirmed (NO in S3), the process returns to S2 and waits until the processing result information file is recorded. .

  In S4, the acquisition unit 302 of the video analysis server device 300 acquires the processing result information file from the analysis data storage unit 203 of the network storage 200, and confirms the content of the analysis processing result information. The contents of the analysis processing result information are confirmed, and it is determined whether or not the video analysis processing has been completed for all the video data files in the folder (S5). When it is determined that the video analysis processing has been completed for all the video data files (YES in S5), polling is performed on the next folder of Layer 2 (S12, S2). The content of the analysis processing result information is confirmed, and if there is video data that has not been subjected to video analysis processing (NO in S5), the process proceeds to S6.

  In S6, the corresponding video data file is acquired from the video storage unit 202 of the network storage 200. For example, the analysis processing result as shown in FIG. 5 indicates that the video analysis processing has not been completed for the video data files “M0003.mp4” (53) and “M0004.mp4” (54). Further, the video data files “M0005.mp4” (55) and “M0006.mp4” (56) are not subjected to video analysis processing at all. Therefore, in S6, these four files are acquired. The acquired video data is decoded by the video decoding unit 303 of the video analysis server device 300 and output to the video analysis processing unit 304.

  Next, the video analysis processing unit 304 performs video analysis processing (S7). Then, an analysis data file is created and overwritten and recorded in the analysis data storage unit 203 of the original network storage 200 (S8). Further, an analysis processing result information file having the contents for which all video analysis processing has been completed is created and similarly overwritten and recorded in the original analysis data storage unit 203 of the network storage 200 (S9). The video data for which all video analysis processing has been completed is encoded again by the video encoding unit 305, and similarly overwritten and recorded in the video storage unit 202 of the network storage 200 that originally existed (S10). At this time, the recording capacity of the network storage 200 can be increased by increasing the compression rate further than the capacity of the video data originally recorded in the analysis data storage unit 203 of the network storage 200.

  Next, it is determined whether the video analysis processing has been completed for all the video data recorded in the video storage unit 202 of the network storage 200 (S11). If the video analysis processing has not been completed for all the video data (NO in S11), a procedure for performing the same processing is entered for the next folder of Layer 2 shown in FIG. 3 (S12, S2). If all the processes have been completed (YES in S11), the network camera 100 is again in a standby state for receiving a notification (S13).

  As described above, the video analysis server device 300 refers to the network storage 200 in which the captured image and the analysis information indicating the analysis content for the captured image are stored in association with each other, and the analysis is performed based on the reference result of the analysis information. An incomplete photographed image is determined. And it analyzes about the picked-up image determined that the analysis is incomplete. For this reason, even if the real-time analysis corresponding to the shooting by the network camera 100 is not in time, the analysis can be supplemented and the analysis can be completed for all the shot images in the video analysis server device 300.

  The network storage 200 further stores analysis data that is an analysis result of a captured image by the network camera 100 in association with the captured image. Here, when the video analysis server device 300 analyzes the captured image in a complementary manner and creates analysis data as an analysis result, the video analysis server device 300 updates the analysis data stored in the network storage 200 with the analysis data. Furthermore, the processing result information stored in the network storage 200 is updated with the processing result information file indicating the analysis content of the captured image by the video analysis server device 300. For this reason, it is possible to store analysis data and processing result information for which all analysis has been completed in the network storage 200.

  Also, the video analysis server device 300 acquires a compressed image of the captured image from the captured image that has been analyzed, and updates the captured image that has been stored in the network storage 200 and has been analyzed with this compressed image. . For this reason, the storage usage of the network storage 200 can be saved. In this embodiment, an example in which the image to be captured is a moving image (video) composed of a plurality of frames has been described, but a still image may be used.

<Embodiment 2>
In the above system configuration, the case where the video analysis processing unit 104 of the network camera 100 and the video analysis processing unit 304 of the video analysis server device 300 perform analysis processing of the same function has been described. However, the video analysis processing function may be distributed to perform all processing in total. In this case, as shown in FIG. 7, all analysis data recorded in the Layer 3 is incomplete data, and the video analysis processing unit 304 of the video analysis server device 300 processes all the video data. The video analysis process is performed again. However, even if such an operation is performed, the above system configuration can be used without any problem. Which function of the video analysis processing between the video analysis processing unit 104 of the network camera 100 and the video analysis processing unit 304 of the video analysis server device 300 may be determined in advance, depending on the load status of the device You can change it.

  As described above, by dynamically changing the load of the image analysis processing between apparatuses according to the load state of the apparatus, it is possible to efficiently perform image analysis using the resources of the apparatus.

  As described above, according to each embodiment of the present invention, even if an image capturing apparatus that performs video analysis processing at the same time as shooting is performed, even if video analysis processing cannot be performed in real time, the analysis data obtained by performing all video analysis processing is It can be recorded on a recording device.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Network camera, 200: Network storage, 300: Video analysis server device, 301: LAN I / F unit, 302: Acquisition unit, 303: Video decoding unit, 304: Video analysis processing unit, 305: Video encoding unit

Claims (11)

An information processing system including an imaging device, a storage device, and an information processing device,
The imaging device
Photographing means for photographing an image;
A first analysis unit that attempts to analyze the image in response to the image being captured by the imaging unit;
The storage device stores an image captured by the imaging unit in association with analysis information indicating analysis content of the first analysis unit of the image,
The information processing apparatus includes:
Determination means for determining an image that has not been analyzed with reference to the analysis information;
An information processing system comprising: second analysis means for performing analysis on an image determined to be incomplete. A reference means for referring to a storage means in which a captured image and analysis information indicating analysis content for the captured image are stored in association with each other;
A determination unit that determines a captured image that has not been analyzed based on a result of referring to the analysis information by the reference unit;
An information processing apparatus comprising: an analysis unit configured to analyze a captured image that has been determined to be incomplete. The storage means further stores analysis data that is an analysis result of the captured image in association with the captured image,
The information processing apparatus according to claim 2, further comprising an update unit that updates the analysis data stored in the storage unit with analysis data that is an analysis result of the analysis unit.   The information processing apparatus according to claim 3, wherein the update unit updates the analysis information stored in the storage unit with analysis information indicating an analysis content of a captured image by the analysis unit. The storage means stores analysis information of the captured image for each type of analysis,
The determination means determines a captured image that has not been analyzed for each type of analysis,
The information processing apparatus according to any one of claims 2 to 4, wherein the analysis unit performs an analysis of a type that has not been analyzed for the captured image. The captured image is a moving image composed of a plurality of frames,
The storage means stores the analyzed frame range as the analysis information,
The information processing apparatus according to claim 2, wherein the analysis unit performs analysis on a frame that has not been analyzed based on the range of the frame.   The information processing apparatus according to claim 6, wherein the analyzed frame range is indicated by the number of analyzed frames. In the storage means, a spatial region analyzed in the captured image is stored as the analysis information,
The information processing apparatus according to any one of claims 2 to 7, wherein the analysis unit performs analysis on a spatial region in which the analysis of the captured image has not been completed based on the spatial region. Compression means for obtaining a compressed image of the captured image from the captured image that has been analyzed;
The information processing apparatus according to claim 2, further comprising: an update unit that updates a captured image stored in the storage unit that has been analyzed with a compressed image of the captured image. A reference step for referring to a storage means in which a captured image and analysis information indicating analysis content for the captured image are stored in association with each other;
A determination step of determining a captured image that has not been analyzed based on a result of the reference of the analysis information by the reference step;
An information processing apparatus control method comprising: an analysis means; or an analysis step of analyzing a photographed image determined to be incomplete.   The computer program for functioning a computer as each means with which the information processing apparatus of any one of Claim 2 to 9 is provided.

Images