JP6423023B2 - Video stream processing apparatus, video stream processing program, and video monitoring system - Google Patents

Description

  The present invention relates to a video stream processing device, a video stream processing program, and a video monitoring system.

  Devices that perform monitoring based on a video stream in which a plurality of images are continuous are known. For example, Patent Document 1 describes a technique for monitoring an intruder based on video from a TV camera. Patent Document 2 describes a technique for detecting a face, a license plate, or the like from an image.

Japanese Patent Laid-Open No. 3-097080 Japanese Patent Laying-Open No. 2015-508591

  However, the conventional technique has a problem that it is difficult to appropriately combine the determination results in time when different monitoring processes are performed using a plurality of applications.

  For example, when an intruder that has not been recognized before is recognized for the first time in a certain frame, a determination result indicating that the intruder has been recognized and time information corresponding to the frame are output. This time information may be relatively changed depending on transmission / reception processing of a video stream, processing delay in an application, and the like. Due to this, when merging determination results from a plurality of applications, there may be a difference between the time at which one application refers to a frame and the time at which another application refers to the same frame. In such a case, it is difficult to display the respective determination results by appropriately fusing them in one video stream.

  The present invention has been made in order to solve such problems, and in the case where different monitoring processes are performed using a plurality of applications, a video that enables the determination results to be appropriately fused in time. It is an object to provide a stream processing device, a video stream processing program, and a video monitoring system.

In order to solve the above-described problems, a video stream processing apparatus according to the present invention provides:
A function for acquiring a plurality of images constituting a video stream;
A function of associating each image with a unique identifier in the video stream;
A function of transmitting an associated image and identifier pair to an image analysis device via a communication network;
A function of generating a low-rate video stream consisting of only a part of the image;
A function of transmitting, for the low-rate video stream, an associated image and identifier pair to an image analysis device via a communication network;
Is provided.
According to a particular aspect,
A function of generating a background difference image and an inter-frame difference image corresponding to each image;
For each image, a function of transmitting a set of a corresponding background difference image or a corresponding inter-frame difference image and an identifier associated with the image to the image analysis apparatus via a communication network;
Is further provided.
According to a particular aspect,
The identifier is acquired for each image after acquiring the image and before generating the background difference image and the inter-frame difference image.
According to a particular aspect, the identifier is a time stamp .

  The video stream processing program according to the present invention causes a computer to function as the above-described video stream processing apparatus.

The video surveillance system according to the present invention is
The video stream processing device described above;
The plurality of image analyzers;
An image monitoring system comprising an image processing device,
The video stream processing device further includes a function of transmitting the video stream to the image processing device,
Each image analysis apparatus has a function of outputting an identifier and a determination result associated with an image that satisfies a predetermined determination criterion among images,
The image processing device has a function of processing each image in accordance with an output from the image analysis device.

  According to the video stream processing device, the video stream processing program, and the video monitoring system according to the present invention, since a unique identifier is associated with each image, the determination result can be synchronized based on the identifier, and the determination result is temporally Can be fused appropriately.

It is a figure which shows the example of a structure of the video surveillance system containing the video stream processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the data which the video stream processing apparatus of FIG. 1 handles. 3 is a flowchart illustrating an example of a flow of processing when the video stream processing apparatus of FIG. 1 acquires an image. 3 is a flowchart illustrating an example of a flow of processing when the video stream processing apparatus of FIG. 1 transmits an image. It is a figure which shows the example of a structure of the video monitoring system containing the video stream processing apparatus which concerns on the modification of Embodiment 1 of this invention.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
Embodiment 1 FIG.
FIG. 1 shows an example of the configuration of a video surveillance system 100 according to Embodiment 1 of the present invention. The video monitoring system 100 includes a video stream processing device 10, a human detection device 31, a leaving detection device 32, and an image processing device 50. In addition, the video monitoring system 100 may include a camera 20, a processing result storage DB 40, and the like as illustrated. The human detection device 31 and the abandonment detection device 32 are examples of image analysis devices, and the plurality of image analysis devices constitute the image analysis device group 30.

  In this specification, “image” means, for example, an image or data representing a still image, and “video stream” means a set or data of an image representing, for example, a moving image or video. That is, it can be said that a plurality of images constitute a video stream.

  The video stream processing apparatus 10 has a configuration as a known computer, and includes a calculation unit 11 that performs a calculation and a storage unit 12 that stores information. Although not particularly shown, the video stream processing apparatus 10 includes an input unit that receives a user operation, an output unit that outputs information, and a communication unit that inputs and outputs information to an external communication network.

  The storage unit 12 stores a video stream processing program in addition to information necessary for the operation of the video stream processing apparatus 10 described in this specification. The video stream processing apparatus 10 implements the functions described in the present specification by the arithmetic means 11 executing this video stream processing program. That is, this video stream processing program causes a computer to function as the video stream processing apparatus 10 described in this specification.

  The human detection device 31, the abandonment detection device 32, and the image processing device 50 also have a configuration as a known computer. In addition, each image analysis device (for example, the human detection device 31 and the leaving detection device 32) included in the image analysis device group 30 may be a different computer, or the same computer executes different application programs. It may be a configured device.

  The video stream processing device 10, the camera 20, the human detection device 31, the abandonment detection device 32, the processing result storage DB 40, and the image processing device 50 respectively transmit, receive, and write information as described in this specification. Or it can be read. In particular, the video stream processing device 10 can transmit and receive information to and from the human detection device 31, the abandonment detection device 32, and the image processing device 50 via a communication network.

  The processing result storage DB 40 may be a storage device configured as a computer together with the human detection device 31 or the abandonment detection device 32, or may be configured as a computer operable as an independent storage device.

  FIG. 2 shows an example of data handled by the video stream processing apparatus 10. In this example, seven images are acquired from the camera 20, and a time stamp is associated with each. This example corresponds to a video stream of about 30 frames per second (30 fps). For example, each image is received every 30 milliseconds. Here, since each image is received at a different timing (that is, it is received at a different time even if it is periodically received), the time stamp is used as an identifier for uniquely identifying each image. be able to.

  Note that the number of an image represented by a number such as “image # 1” is given for convenience of explanation in the present specification and drawings, and need not be attached to actual image data.

  As shown in FIG. 2, in the first embodiment, the video stream processing apparatus 10 uses a video stream having the same frame rate as the acquired video stream (“normal rate video stream”) and a frame lower than the acquired video stream. A function of generating a rate video stream (“low rate video stream”) is provided. Here, the low-rate video stream is a video stream composed of only a part of the images constituting the acquired video stream, that is, a video stream obtained by thinning out a part of the images.

  Hereinafter, an example of the operation of the video stream processing apparatus 10 will be described. The video stream processing apparatus 10 generates the normal rate video stream and the low rate video stream as shown in FIG. 2 by executing the processing shown in FIGS. 3 and 4, for example.

  FIG. 3 is a flowchart illustrating an example of a processing flow when the video stream processing apparatus 10 acquires an image. This process is executed, for example, when the video stream processing apparatus 10 receives a video stream from the camera 20 or the like.

  Further, the processing of FIG. 3 is executed independently for each image, for example. When the processing speed of the video stream processing device 10 is sufficiently high, when the images are continuously received from the camera 20, the processing of FIG. 3 is completed for the immediately preceding image before the next image is received. However, the embodiment is not limited to this.

  First, the video stream processing apparatus 10 acquires an image (step S1). Next, the video stream processing apparatus 10 acquires a time stamp and associates the time stamp with the image (step S2). This time stamp is acquired as representing the time at which step S1 or step S2 is executed, for example. The video stream processing apparatus 10 may include a clock device that generates a clock signal serving as a reference for determining or calculating time, and acquires a time stamp by generating a time stamp based on such a clock signal. May be. Further, the video stream processing apparatus 10 may store the image and the time stamp in the storage unit 12 in association with each other.

  Next, the video stream processing apparatus 10 generates a background difference image corresponding to the image (step S3). The method for generating the background difference image is arbitrary. For example, a predetermined background image may be stored in advance in the storage unit 12 and may be generated by calculating a difference from the background image. “Difference” refers to a difference between corresponding pixels in each image, for example, regarding a numerical value (RGB value or the like) representing each pixel. Or you may use the method described in patent document 1 as a production | generation method of a background difference image. In FIG. 2, this difference is represented by using a minus sign. For example, “image # 1 -background” represents the difference between the first image and the background image. The video stream processing apparatus 10 may store the generated background difference image and time stamp in the storage unit 12 in association with each other.

  Next, the video stream processing apparatus 10 generates an inter-frame difference image corresponding to the image (step S4). An inter-frame difference image is an image representing a difference between two images in a video stream. For example, an image representing a difference between an image being processed and a previous image (for example, an image received immediately before). Means. In FIG. 2, similarly to the background difference image, the inter-frame difference image is represented using a minus sign. The video stream processing apparatus 10 may store the generated inter-frame difference image and the time stamp in the storage unit 12 in association with each other.

  Next, the video stream processing apparatus 10 generates a low-rate inter-frame difference image (step S5). A low-rate inter-frame difference image is an inter-frame difference image in a low-rate video stream. If the image acquired in step S1 is an image that is not included in the low-rate video stream, the process in step S5 may be omitted. In the example of FIG. 2, this process is omitted for even-numbered images. The video stream processing apparatus 10 may store the generated low-rate inter-frame difference image and the time stamp in the storage unit 12 in association with each other.

  In this way, the video stream processing apparatus 10 generates a normal rate video stream and a low rate video stream to which time stamps are added.

  FIG. 4 is a flowchart illustrating an example of a processing flow when the video stream processing apparatus 10 transmits an image. This process is executed, for example, when the video stream processing device 10 transmits a video stream to the human detection device 31 or the leaving detection device 32. The process of FIG. 4 may be partially or entirely executed in parallel with the process of FIG.

  First, the video stream processing apparatus 10 receives an image transmission request (step S11). The image transmission request is a request transmitted from another computer (for example, the human detection device 31, the leaving detection device 32, or the image processing device 50), and includes information for specifying the type of image to be transmitted.

The type of image to be transmitted is specified as one of the following patterns, for example.
-Pattern 1: Normal rate video stream image, background difference image and inter-frame difference image-Pattern 2: Low rate video stream image, background difference image and inter-frame difference image-Pattern 3: Normal rate video stream image Only-Pattern 4: Only the background difference image of the normal rate video stream-Pattern 5: Only the difference image between frames of the normal rate video stream-Pattern 6: Only the image of the low rate video stream-Pattern 7: Background difference of the low rate video stream Image only-Pattern 8: Inter-frame difference image of low-rate video stream only-Other combinations of the above

  Next, for the requested type of image, the video stream processing apparatus 10 sets the associated image and time stamp pair via the communication network to the request source (that is, the person detection apparatus 31 that has transmitted the image transmission request). And sent to the abandonment detection device 32 or the image processing device 50) (step S12).

  Here, the processing of FIG. 4 is executed independently for each image transmission request. That is, for example, when the human detection device 31 and the abandonment detection device 32 transmit image transmission requests one after another, the video stream processing device 10 sends the requested types of images to the plurality of image analysis devices. And a set of time stamps can be transmitted.

The image to be transmitted differs depending on the content of the image transmission request. For example, the video stream processing apparatus 10 selectively executes one or more of the following processes according to the content of the image transmission request.
-For each image in the normal rate video stream, send the image and the time stamp associated with that image to at least one image analyzer-For each image in the normal rate video stream, the corresponding background difference Send a set of images or corresponding inter-frame difference images and identifiers associated with the images to at least one image analyzer-for each image in the low-rate video stream, the image and associated with the image A set of time stamps to at least one image analysis device-for each image in the low-rate video stream, a corresponding background difference image or a corresponding inter-frame difference image and an identifier associated with the image Send the set to at least one image analyzer

  In this way, for example, the image of pattern 1 (the image of the normal rate video stream, the background difference image and the inter-frame difference image) is transmitted to the human detection device 31, and the pattern 2 image (low image) is transmitted to the abandonment detection device 32. The image of the rate video stream, the background difference image, and the inter-frame difference image) can be transmitted, and the image of the pattern 3 (only the image of the normal rate video stream) can be transmitted to the image processing device 50.

  A specific transmission / reception method can be arbitrarily designed. For example, when the human detection device 31 and the abandonment detection device 32 are independent computers on the communication network, different IP addresses are assigned to each other, and the video stream processing device 10 performs transmission processing on different IP addresses. May be. Alternatively, regardless of whether the human detection device 31 and the abandonment detection device 32 are independent computers on the communication network, different communication ports are allocated, and the video stream processing device 10 has different communication ports (or sockets). You may perform a transmission process with respect to.

  The video stream processing apparatus 10 continues the process of step S12 until a predetermined image stop request is received from the communicating image analysis apparatus. When the image stop request is received (step S13), the video stream processing apparatus 10 ends the process of FIG. Note that the image stop request does not need to take the form of a packet that is explicitly transmitted, and may be, for example, some information that implicitly indicates a communication interruption.

  The operations of the human detection device 31 and the abandonment detection device 32 are not particularly shown in the flowchart, but operate corresponding to the processing of the video stream processing device 10 shown in FIG. For example, the human detection device 31 and the abandonment detection device 32 transmit an image transmission request to the video stream processing device 10 when activated, and receive the image when the video stream processing device 10 starts transmitting an image accordingly. The type of image to be requested may be defined in advance according to the contents of the analysis processing executed by the human detection device 31 and the abandonment detection device 32, respectively.

  The human detection device 31 and the abandonment detection device 32 output a time stamp and a determination result associated with an image that satisfies a predetermined determination criterion among images.

  For example, the human detection device 31 analyzes the image and determines whether or not a person is shown. If a person is shown, the position or area is determined. The position is determined as a two-dimensional coordinate value (for example, X coordinate and Y coordinate). The region is determined as a rectangular region, for example, and can be represented by two sets of two-dimensional coordinate values (for example, the X and Y coordinates of the upper left vertex of the rectangle and the X and Y direction sizes of the rectangle).

  Similarly, for example, the abandonment detection device 32 analyzes the image and determines whether or not the left object is reflected. Further, when the left object is reflected, the position or area is determined. The position is determined as a two-dimensional coordinate value (for example, X coordinate and Y coordinate). The region is determined as a rectangular region, for example, and can be represented by two sets of two-dimensional coordinate values (for example, the X and Y coordinates of the upper left vertex of the rectangle and the X and Y direction sizes of the rectangle).

  Specific contents (determination criteria and the like) of the analysis processing of the human detection device 31 and the leaving detection device 32 can be designed based on, for example, a known technique. For example, the technique described in Patent Document 1 or Patent Document 2 may be used.

  The human detection device 31 and the abandonment detection device 32 store the determination result in the processing result storage DB 40. The format of the determination result can be designed arbitrarily. For example, the determination result by the human detection device 31 includes a flag indicating whether or not a person is reflected in the time stamp of each image, and the position when it is reflected. Alternatively, it can be expressed as a set associated with information representing a region. An image need not be included in the determination result, but may be included.

  Here, since the time stamp of each image is given by the video stream processing device 10 and is common to the image analysis device group 30, direct image synchronization processing or the like is performed between the human detection device 31 and the leaving detection device 32. Even if not, the determination result is synchronized. For example, even when there is a difference in processing speed between the person detection device 31 and the abandonment detection device 32, the time when a certain person is detected and the time when a certain abandoned object is detected are respectively detected images. Can be accurately compared based on the time stamps.

  The image analysis device group 30 includes other image analysis devices (a person identification device, a shutter open / close determination device, a stay detection device, etc.) instead of or in addition to the human detection device 31 and the leaving detection device 32. May be included.

  When activated, the image processing device 50 transmits an image transmission request to the video stream processing device 10, and receives the image when the video stream processing device 10 starts transmitting an image accordingly. The requested image type is, for example, only an image of a normal rate video stream.

  Further, the image processing device 50 processes the image received from the video stream processing device 10 based on the determination result in units of images stored in the processing result storage DB 40.

  The content of the processing can be arbitrarily designed, but for example, a predetermined mark, symbol, figure or the like may be superimposed on the image according to the determination result. Specifically, for an image that is determined to show a human being, an image that is determined to have left behind by overlaying a rectangular frame of a specific color (for example, yellow) on the corresponding area. A rectangular frame of another specific color (for example, red) may be superimposed on the corresponding area.

  The image processing device 50 stores the processed image. Further, the image processing device 50 displays the processed image in accordance with a user operation. Since a frame indicating the position of a person or an object is drawn on the displayed image, a user of the image processing apparatus 50 (for example, a facility supervisor) can select a person or an object shown in the video of the camera 20. It can be easily recognized.

  Also here, the time stamp of each image is given by the video stream processing device 10 and is common to the image analysis device group 30 and the image processing device 50, and therefore directly between the image analysis device group 30 and the image processing device 50. Even if the image synchronization process is not performed, the determination result and the processing process are synchronized. For example, even when there is a difference in processing speed between the transmission processing of the image from the video stream processing device 10 to the image processing device 50 and the writing processing of the determination result to the processing result storage DB 40 by the human detection device 31, When a person is detected in an image, it is possible to display a frame indicating the person at an accurate timing.

  As described above, according to the video stream processing apparatus 10, the video stream processing program, and the video monitoring system 100 according to the first embodiment, each image is associated with a time stamp as a unique identifier. The determination results can be synchronized, and the determination results can be appropriately fused in time.

  In the example of FIG. 3, the time stamp is acquired for each image after the image is acquired and before the background difference image, the inter-frame difference image, and the low-rate inter-frame difference image are generated. For this reason, even if the processing time of steps S3 to S5 varies from image to image, the time stamp represents the time when the image occurred relatively accurately.

In the first embodiment, the following modifications can be made.
The number of image analysis devices included in the image analysis device group 30 may be at least one. When there is one image analysis device, the video stream processing device 10 transmits the associated image and time stamp pair to the image analysis device via the communication network. In addition, the video stream processing device 10 transmits, for each image, a set of a corresponding background difference image or a corresponding inter-frame difference image and a time stamp associated with the image to the image analysis device via the communication network. To do.

  When there are a plurality of image analysis apparatuses as in the first embodiment, the video stream processing apparatus 10 transmits a set of associated images and time stamps to the plurality of image analysis apparatuses via the communication network. In addition, the video stream processing apparatus 10 uses, for each image, at least one image analysis of a set of a corresponding background difference image or a corresponding inter-frame difference image and a time stamp associated with the image via a communication network. Send to device.

  The image processing device 50 may output the processed image by another method instead of displaying the processed image. For example, it may be stored in a storage means or transmitted to another computer via a communication network.

  The image processing device 50 may be omitted. The determination result stored in the processing result storage DB 40 can be used for other purposes.

  A device that performs more complicated determination processing based on the processing result of the image analysis device group 30 may be added. For example, a device that determines whether or not a “walking person” is reflected based on the human coordinates detected in each image may be added. The determination result of such a device may also be stored in the processing result storage DB 40 in the same format as the output of the human detection device 31 or the like.

  The order of the steps shown in FIG. 3 is not limited to this. The order of steps S3 to S5 can be arbitrarily changed. Further, the process of step S2 may be performed during or after steps S3 to S5.

  Any or all of steps S3 to S5 in FIG. 3 may be omitted. For example, in Embodiment 1, the video stream processing apparatus 10 has a function of generating a low-rate video stream, but this function may be omitted. For example, when all of the image analysis devices support a relatively high rate video stream, a low rate video stream is not necessary. In the first embodiment, the video stream processing apparatus 10 has a function of generating a background difference image and an inter-frame difference image, but may have only a function of generating one of them. Alternatively, no function is provided, and step S12 may be executed only for the image acquired in step S1. In that case, it is not necessary to include information specifying the type of image to be transmitted in the image transmission request.

  In the first embodiment, a time stamp is used as an image identifier. However, any identifier can be used as long as it is a unique identifier in the video stream. It may be a unique identifier common to all video streams handled by the video stream processing apparatus 10 such as a time stamp, or may be unique only within a specific video stream, such as an order in the video stream. It may be an identifier.

  In the first embodiment, the video stream processing apparatus 10 acquires an image from the camera 20, but any video stream output apparatus or video stream storage apparatus may be used instead of the camera 20.

  The overall configuration may be as shown in FIG. As shown in FIG. 5, a camera image is input and an image in which an analysis result is drawn is output. The video GW function (corresponding to the video stream processing device 10 of the first embodiment) acquires each image constituting the camera video, and each analysis function (each device included in the image analysis device group 30 of the first embodiment). Corresponding) has a function to distribute frame images. There are five types of analysis functions: leaving / sitting / stagnation detection, in-region state determination, in-region detection, abnormal behavior determination, and possession determination, and only necessary ones can be executed. In the processing result integration function (corresponding to the processing result storage DB 40 and the image processing apparatus 50 of the first embodiment), each analysis result and the result of combining them are enclosed in a rectangle on the original image and labeled. Merge. The merged image is distributed using the video distribution function, and the system user can check the analysis result.

  The five analysis functions have various processing speeds and operate asynchronously. In order to integrate the results processed asynchronously, not only an image but also an ID (corresponding to the time stamp of the first embodiment) is assigned to the video GW, and the image is distributed to each function. By assigning an ID, it is possible to synchronize the processing results of each function.

  By using the method of the first embodiment or this modified example, it becomes possible to link the distribution image from the video GW and each analysis result in the processing result integration function, and a plurality of analysis results are drawn together on the distribution image. Output is possible.

  It is also possible to create a detection condition that combines a plurality of analysis results. For example, a large load or a cane is determined in the possession determination, and a sitting person is detected in the leaving / sitting / staying detection, and then these determination results are combined. As a result, it becomes possible to detect and assist a person who seems to be in trouble with a large baggage or an elderly person who seems tired. For example, the stroller determination can be performed by detecting the stroller that has not been left or pushed by a person by detecting the stroller by carrying out possession determination, by detecting a left-behind object by detecting leaving, sitting, or staying.

  In order to respond to various detection needs in a unified manner, this system is designed so that detection conditions can be concentrated in the processing result integration function unit and individual conditions can be incorporated.

  10 video stream processing device, 31 human detection device (image analysis device), 32 detection device (image analysis device), 50 image processing device, 100 video monitoring system.

Claims (6)

A function for acquiring a plurality of images constituting a video stream;
A function of associating each image with a unique identifier in the video stream;
A function of transmitting an associated image and identifier pair to an image analysis device via a communication network;
A function of generating a low-rate video stream consisting of only a part of the image;
A function of transmitting, for the low-rate video stream, an associated image and identifier pair to an image analysis device via a communication network;
A video stream processing apparatus comprising: A function of generating a background difference image and an inter-frame difference image corresponding to each image;
For each image, a function of transmitting a set of a corresponding background difference image or a corresponding inter-frame difference image and an identifier associated with the image to the image analysis apparatus via a communication network;
Further comprising
The video stream processing apparatus according to claim 1. The identifier is acquired for each image after acquiring the image and before generating the background difference image and the inter-frame difference image.
The video stream processing apparatus according to claim 2.   The video stream processing apparatus according to claim 1, wherein the identifier is a time stamp. A video stream processing program for causing a computer to function as the video stream processing device according to any one of claims 1 to 4 . The video stream processing device according to any one of claims 1 to 4 ,
The image analyzer;
An image monitoring system comprising an image processing device,
The video stream processing device further includes a function of transmitting the video stream to the image processing device,
The image analysis device has a function of outputting an identifier and a determination result associated with an image that satisfies a predetermined determination criterion among images,
The image processing device has a function of processing each image according to an output from the image analysis device,
Video surveillance system.

Images