JP7458138B2 - Information processing system, information processing device, terminal device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing system, an information processing device, a terminal device, an information processing method, and a program.

In recent years, technology for detecting objects in videos has become widely used.
For example, technology has been put into practical use that automatically detects abnormal objects in images captured by surveillance cameras.
Furthermore, with the development of AI (Artificial Intelligence), technology is being developed to detect objects in a video by image recognition using AI or the like.
Note that a technique for detecting an object in a video is described in, for example, Patent Document 1.

JP2019-092154A

However, when detecting the movement of an object within a video, various disturbances may reduce the accuracy of detecting the movement of the object.
For example, when detecting movement of an object in a specific direction within a video, there is a possibility that the direction of movement of the object may be erroneously detected due to the influence of disturbance.
In addition, when there is a large disturbance such as when a camera is installed outdoors, using AI to detect the movement of an object in a specific direction requires a large computational processing load, and sample data for learning There are cases where implementation is difficult due to problems such as the need to prepare a large number of.
Furthermore, in the approach of expressing the target to be detected using a program (rule-based approach), it is necessary to optimally set many operating parameters, but the large number of parameters makes it difficult to set them intuitively.

An object of the present invention is to estimate the velocity of an object in an image robustly against disturbances.

In order to solve the above problems, an information processing system according to one aspect of the present invention includes:
An information processing system including a terminal device used by a user and a server configured to be able to communicate with the terminal device,
The terminal device is
comprising a detection requesting means for requesting the server to perform detection processing for detecting an object moving within a set detection area in a video to be detected;
The server is
In response to a request from the terminal device, in the video to be detected, based on the correlation between a pixel in the detection region and data of pixels around the pixel in data of the video at different times, Object detection means for performing the detection process of detecting a moving object;
Detection result presentation means for presenting a detection result of an object by the object detection means to the terminal device;
It is characterized by having the following.

According to the present invention, the velocity of an object in an image can be estimated robustly against disturbances.

1 is a schematic diagram showing the overall system configuration of an information processing system 1 according to the present invention. It is a diagram showing the hardware configuration of an information processing device 800 that configures each device. FIG. 2 is a block diagram showing the functional configuration of the server 20. FIG. 3 is a schematic diagram showing an example of a moving average feature amount R(t) displayed as a heat map in an N×N matrix. 1 is a block diagram showing the functional configuration of a user terminal 10. FIG. FIG. 3 is a schematic diagram showing an example of a live view screen. FIG. 7 is a schematic diagram showing an example of a live view screen when a plurality of object detection areas are set. FIG. 13 is a schematic diagram illustrating an example of a detection log screen. FIG. 3 is a schematic diagram showing an example of a condition setting screen. 3 is a flowchart showing the flow of UI control processing executed by the user terminal 10. FIG. 5 is a flowchart showing the flow of object detection processing executed by the server 20. FIG. FIG. 8 is a block diagram showing the functional configuration of a stand-alone information processing device 800 that executes object detection processing and display of processing results.

Embodiments of the present invention will be described below with reference to the drawings.
The information processing system according to this embodiment can move within any area in a video at a specified speed and in a constant direction with intuitive settings without having to prepare a huge number of learning samples for application to AI. For example, it detects objects moving in a specific direction, such as at entrances to expressways, building entrances, etc.
Furthermore, in the information processing system according to the present embodiment, it is possible to set conditions for selectively detecting objects moving in a specific direction in a video, and a user who can easily set such conditions can Provide an interface.
Thereby, the information processing system according to the present embodiment makes it possible to more easily detect movement of an object in a specific direction in a video with high accuracy.
Furthermore, it is possible to estimate the velocity of an object in an image robustly against disturbances.

[composition]
[System configuration]
FIG. 1 is a schematic diagram showing the overall system configuration of an information processing system 1 according to the present invention.
As shown in FIG. 1, the information processing system 1 includes a plurality of user terminals 10, a server 20, and an imaging device C. The plurality of user terminals 10 and the server 20 are connected to a network 30 such as the Internet. It is configured to be able to communicate via. Note that the information processing system 1 can be provided with an imaging device C, such as a network camera, as needed. This imaging device C is configured to be able to communicate with the user terminal 10 and the server 20 via the network 30. In the information processing system 1, any of the images acquired (imaged or read out, etc.) by the user terminal 10, the server 20, or the imaging device C can be used as a target for detecting the movement of an object in a specific direction in the image. It is possible.

The user terminal 10 is an information processing device used by a user, and accepts input of various information (such as video data and conditions related to the object to be detected) for detecting the movement of an object in a video, and makes requests to the server 20 regarding the detection of an object in the video. The user terminal 10 also displays the detection results of an object in the video sent from the server 20 (specifically, a UI screen that displays the detection results), and downloads data (log data) related to a series of detection results of an object in the video.

When the user terminal 10 makes a request for detecting an object in the video, the server 20 detects an object in the video that meets the conditions according to the conditions set in the user terminal 10. Specifically, the server 20 detects objects moving in a specific direction in the video, and detects objects that meet conditions set by the user on the user terminal 10, such as the moving speed, size, or moving area. Detect movement of. Then, the server 20 sequentially transmits data representing the detection results to the user terminal 10, or transmits data of the detection results regarding the entire video data to the user terminal 10 as log data.
The imaging device C is installed at a position where it can image an area where an object is detected, and transmits the captured video data to the user terminal 10 or the server 20 via the network 30.

[Hardware configuration]
Next, the hardware configuration of each device in the information processing system 1 will be explained.
In the information processing system 1, each device is constituted by an information processing device such as a PC, a server computer, or a tablet terminal, and the basic configuration thereof is the same.

FIG. 2 is a diagram showing the hardware configuration of the information processing device 800 that constitutes each device.
As shown in FIG. 2, the information processing device 800 that constitutes each device includes a CPU (Central Processing Unit) 811, a ROM (Read Only Memory) 812, a RAM (Random Access Memory) 813, a bus 814, and an input It includes a section 815, an output section 816, a storage section 817, a communication section 818, a drive 819, and an imaging section 820.

The CPU 811 executes various processes according to a program recorded in the ROM 812 or a program loaded from the storage unit 817 to the RAM 813 .
The RAM 813 also stores data and the like necessary for the CPU 811 to execute various processes.

The CPU 811, ROM 812, and RAM 813 are interconnected via a bus 814. An input section 815, an output section 816, a storage section 817, a communication section 818, a drive 819, and an imaging section 820 are connected to the bus 814.

The input unit 815 is made up of various buttons and the like, and is used to input various pieces of information in response to instruction operations.
The output unit 816 is composed of a display, a speaker, etc., and outputs video and audio.
In addition, when the information processing device 800 is configured as a smartphone or a tablet terminal, the displays of the input unit 815 and the output unit 816 may be arranged so as to overlap each other to configure a touch panel.
The storage unit 817 is configured with a hard disk or a dynamic random access memory (DRAM) or the like, and stores various data managed by each server.
The communication unit 818 controls communications with other devices via the network 30 .

A removable medium 831 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 819 as appropriate. The program read from the removable medium 831 by the drive 819 is installed in the storage unit 817 as necessary.
The imaging unit 820 is configured by an imaging device including a lens, an image sensor, etc., and captures a digital image of a subject.
Note that when the information processing device 800 is configured as a tablet terminal, it is also possible to configure the input unit 815 with a touch sensor and arrange it over the display of the output unit 816 to provide a configuration with a touch panel. be.

[Functional configuration]
Next, the functional configuration of each device in the information processing system 1 will be explained.
[Configuration of server 20]
FIG. 3 is a block diagram showing the functional configuration of the server 20.
As shown in FIG. 3, the CPU 811 of the server 20 includes a data acquisition section 211, a parameter setting section 212, an image filter processing section 213, an image binarization processing section 214, a pixel value calculation section 215, and a condition A determination unit 216, a visualization processing unit 217, and a detection result presentation unit 218 function. Further, in the storage unit 817 of the server 20, a video data storage unit 271, a parameter storage unit 272, and a log data storage unit 273 are formed.

The video data storage unit 271 stores video data in which an object is detected. The video data storage unit 271 stores video data captured in real time and video data captured in the past transmitted from the user terminal 10, as well as video data acquired by the server 20 from sources other than the user terminal 10 (imaging device C, etc.). Video data shot in real time, video data shot in the past, etc. can be stored. The video data storage unit 271 can also store video data shot in real time by the server 20 and video data shot in the past. Note that the video data in this embodiment can be data in various forms in which the movement of an object is recorded, such as moving image data and still image data that are continuously photographed.

The parameter storage unit 272 stores various parameters transmitted from the user terminal 10 (for example, operation parameters indicating conditions for detecting an object in a video).
The log data storage unit 273 stores detection result data (log data) obtained by detecting an object in a video.

The data acquisition unit 211 acquires video data transmitted from the user terminal 10 or another device, and data such as various parameters set for detecting objects in the video transmitted from the user terminal 10. Further, the data acquisition unit 211 stores the acquired video data in the video data storage unit 271 and also stores the acquired data of various parameters in the parameter storage unit 272. In this embodiment, the data acquisition unit 211 can acquire video data frame by frame in time series. Furthermore, if the image data of the acquired frame is a color image, the data acquisition unit 211 converts it into a grayscale image.
Here, the data of the gray scale image acquired at time t is two-dimensional data of W pixels in the horizontal direction and H pixels in the vertical direction, and the luminance value for the coordinates (x, y) is expressed as I(t , x, y). Note that the brightness value takes a value from 0 (black) to 255 (white).

The parameter setting unit 212 sets the conditions for detecting an object in the video based on the operation parameters transmitted from the user terminal 10. Note that if the parameter setting unit 212 receives operation parameters from the user terminal 10 while detecting an object in the video (i.e., if the conditions for detecting an object are changed), the parameter setting unit 212 updates the conditions for detecting an object in the video based on the new operation parameters.

The image filter processing unit 213 performs various filter processing on the image of the frame in which an object is to be detected, according to the user's settings. For example, if the user has set the noise filter to be applied, the image filter processing unit 213 applies the noise filter to the image of the frame in which the object is detected. Further, the image filter processing unit 213 applies the illumination variation filter to the frame of the image in which the object is to be detected, when the user has set the illumination variation filter to be applied.

として模式的に表すことができる。ただし、ｎはフレーム間の時間差を表し、ｔｈは閾値であり映像内における物体の検知を行う際の条件を示す動作パラメータとしてユーザによって設定されたパラメータの１つである。
さらに、画像２値化処理部２１４は、生成した２値化フレーム間差分画像において、ユーザによって設定された物体検知領域以外の画素の値を０に設定する。これにより、不要な情報を除去し、物体の検知における誤判定が生じることを抑制できる。
また、ユーザが指定した監視エリアを示すマスク情報と、２値化フレーム間差分画像のAND演算でＩ’（ｔ，ｘ，ｙ）を更新することで指定のエリア内の動きにのみ着目することができる。 The image binarization processing unit 214 generates a binarized inter-frame difference image between frames in an image of a frame of a video in which an object is detected. When the frame image data at time t is I(t), the luminance value I'(t, x, y) for the frame data coordinates (x, y) of the binarized inter-frame difference image is

It can be schematically expressed as. However, n represents the time difference between frames, and th is a threshold value, which is one of the parameters set by the user as an operation parameter indicating conditions for detecting an object in a video.
Further, the image binarization processing unit 214 sets the values of pixels other than the object detection area set by the user to 0 in the generated binarized inter-frame difference image. As a result, unnecessary information can be removed and erroneous determinations in object detection can be suppressed.
In addition, by updating I'(t, x, y) by ANDing the mask information indicating the monitoring area specified by the user and the binarized inter-frame difference image, it is possible to focus only on the movement within the specified area. Can be done.

The pixel value calculation unit 215 calculates a feature quantity that quantifies the moving speed (moving direction and speed) of an object moving within the object detection area. As this feature quantity, values calculated by various calculation methods can be used as long as the moving speed (moving direction and speed) of the object can be quantified. However, in this embodiment, a binary value is used. Correlation between the target pixel value (luminance value) I'(t, x, y) in the binarized inter-frame difference image and the binarized inter-frame difference image I'(t-1, x, y) acquired at the previous time We will use the value. The correlation value can be calculated using the formula r(t, dx, dy)=Σ _x Σ _y I'(t, x, y) I'(t-1, x+dx, y+dy). Here, (dx, dy) indicates displacement with respect to two-dimensional coordinates within the image.
When calculating the correlation value within N×N pixels around the pixel of interest, the feature amount is N This becomes an N-dimensional matrix r(t).

Furthermore, the pixel value calculation unit 215 calculates a moving average feature amount R(t) of the calculated feature amount r(t) using the set time window. The feature quantity r(t) is a value representing the feature related to the movement of the object, but by calculating the moving average feature quantity R(t) of the feature quantity r(t), it is possible to detect the movement of the object in a specific direction due to noise. This can prevent false detections.

FIG. 4 is a schematic diagram showing an example in which the moving average feature amount R(t) is displayed as a heat map in an N×N matrix.
As in the example in Figure 4, when the moving average feature R(t) calculated for a scene in which a bicycle approaches is displayed as a heat map in an N×N matrix, It can be seen that the value tends to increase.
Using this tendency, the pixel value calculation unit 215 calculates the product of the calculated moving average feature amount R(t) and the N×N-dimensional weight matrix W for each element {R(t, i, j)× W(i,j)} is the feature amount R'(t). For example, in the moving average of the calculated feature amounts, the weight for the direction or speed that the user wants to detect can be set to 1.0, and the other weights can be set to 0.0. Regarding the weighting of the speed, the weighting W(i, j) can be set such that the address at the center of N×N pixels has a speed of 0, and the further away from the center the address, the higher the moving speed.

Further, the pixel value calculation unit 215 calculates the sum s(t) of each element value of the calculated weighted feature amount R'(t)=Σ _mx Σ _my R'(t, mx, my). This value pseudo-represents the size of the object to be detected. Here, in FIG. 4, mx and my refer to coordinate information on the heat map.
Then, the pixel value calculation unit 215 calculates the maximum element coordinates {(dx, dy)}=arg max {R'(t, dx, dy)}, (dx, dy)∈N Detect the direction vector from the center coordinates (0, 0) to ² and obtain it as the moving speed.
Alternatively, a method may be adopted that can handle a plurality of objects having different moving speeds by obtaining local maximum values (modes).

The pixel value calculation unit 215 uses the set time window to calculate the time integral S(t) of the sum s(t) of the elements of the weighted feature amount. By calculating the moving average of the sum s(t) of the elements of the weighted feature amount, false detection of movement of an object in a specific direction due to noise is suppressed.
The condition determination unit 216 determines whether or not the time integral S(t) of the sum s(t) of the elements of the weighted feature satisfies the conditions for detecting it as an abnormal value (for example, if it exceeds a set threshold). ). Note that the time integral S(t) of the sum s(t) of the elements of the weighted feature has significance as a score indicating the possibility of a detected abnormality.

If the condition determining unit 216 determines that the time integral S(t) of the sum of elements of the weighted feature amount s(t) meets the conditions for detecting it as an abnormal value, the visualization processing unit 217 An alert indicating that the object in the video is in an abnormal state (for example, exceeds a set threshold) is output to the terminal 10.
Furthermore, the visualization processing unit 217 generates data of various information representing the detection results of objects in the video (for example, detection result data included in the live view screen described later), and sequentially stores them in the log data storage unit 273. .

In addition, in order to make it possible to confirm that the operation parameter settings are appropriate, the visualization processing unit 217 converts the binarized inter-frame difference image of the object detection area and the element of the feature amount r(t) into 2. A dimensional matrix display, a vector representing the moving direction of the object, and the time integral S(t) (score indicating the possibility of detected anomaly) of the sum of weighted feature elements s(t) will be described later. It can be displayed on the live view screen.

The detection result presentation unit 218 sequentially transmits data of various information representing the detection result of the object in the image generated by the visualization processing unit 217 to the user terminal 10. Further, when a download of log data is requested from the user terminal 10, the detection result presentation unit 218 reads the log data stored in the log data storage unit 273 and transmits it to the user terminal 10.

[Configuration of user terminal 10]
FIG. 5 is a block diagram showing the functional configuration of the user terminal 10.
As shown in FIG. 5, in the CPU 811 of the user terminal 10, a UI control section 111 and a data management section 112 function. Further, the storage unit 817 of the user terminal 10 includes a video data storage unit 171, a parameter storage unit 172, and a log data storage unit 173.

The video data storage unit 171 stores video data in which object detection is performed. In this embodiment, the video data for which object detection is performed can include both video data shot in real time and video data shot in the past, and the data is stored in the video data storage unit 171. The data of these images are stored.

The parameter storage unit 172 stores various parameters set for detecting an object in a video (for example, various parameters representing conditions related to the object to be detected, various parameters for displaying the detection result of the object in the video, etc.). is memorized.
The log data storage unit 173 stores detection result data (log data) obtained by the server 20 detecting an object in a video.

The UI control unit 111 controls the display of various input/output screens (hereinafter referred to as "UI screens") for inputting and outputting parameters for detecting objects in images. For example, the UI control unit 111 displays a UI screen (hereinafter referred to as a "live view screen") that displays video data in real time and the detection results of objects in the video (hereinafter referred to as "live view screen"), A UI screen that displays a log of detection results (hereinafter referred to as the "detection log screen"), a UI screen that sets conditions regarding the object to be detected (hereinafter referred to as the "condition setting screen"), or notification as an alert. Displays a UI screen (hereinafter referred to as "alert setting screen") for setting the content to be displayed. Note that the UI control unit 111 displays these screens according to the user's selection. When displaying the live view screen is selected, the UI control unit 111 makes a request to the server 20 regarding detection of an object in the video.

FIG. 6 is a schematic diagram showing an example of a live view screen.
As shown in FIG. 6, the live view screen includes a video display area D1 where a video in which an object is detected is displayed, and a detection result display area D2 where information regarding the detection result of the object in the video is displayed. include. In the example shown in FIG. 6, an image near the exit from the motorway to the general road is displayed in the image display area D1.

In the video display area D1, the video in which the object is detected is displayed as a moving image or a series of still images. In addition, in the video display area D1, an area set as a target for object detection (hereinafter referred to as "object detection area d") is displayed. Furthermore, when an object that matches the set conditions (an object moving in a specific direction) is detected in the video display area D1, a marker that identifies the detected object (hereinafter referred to as "detected object identification marker M") is displayed. As an example, the object detection area d can be displayed as a green frame, and the detected object identification marker M can be displayed as a red frame that surrounds the object.

In the detection result display area D2, various types of information representing the detection results of objects in the video are displayed. For example, the detection result display area D2 includes information such as abnormal values (for example, values exceeding a set threshold), presence or absence of alerts, detailed information on detection results, time zone, set threshold, and execution frame rate. Is displayed. As detailed information on the detection results, for example, the moving direction set as the detection direction, the moving direction of the detected object, the moving speed, or the size of the abnormality (degree of exceeding the threshold) can be displayed. . In the example shown in FIG. 6, coordinates with the set moving direction as the + direction of the vertical axis are displayed in the detection result display area D2 as detailed information of the detection result (at the right end of the detection result display area D2), and the setting If the moved direction has a width, the width is displayed as a fan-shaped spread from the + direction of the vertical axis. Furthermore, in the example shown in FIG. 6, the moving direction of the detected object is displayed as a straight line, and the moving speed of the detected object is expressed as the length of the straight line. Note that the size of the abnormality can be displayed depending on the display format of the straight line representing the moving direction of the detected object. For example, the straight line representing the moving direction of the detected object can be displayed to correspond to the size of the abnormality. It can be displayed in different colors.

In addition, on the live view screen, it is possible to set multiple object detection areas d in the image display area D1, and in this case, a detection result display area D2 showing the detection results of each of the set object detection areas is displayed within the live view screen.
FIG. 7 is a schematic diagram showing an example of a live view screen when a plurality of object detection areas are set.
7, two object detection areas d1 and d2 are set within image display area D1, with object detection area d1 indicated by a solid line and object detection area d2 indicated by a dashed dotted line. Furthermore, in detection result display area D2, the detection results of object detection area d1 and object detection area d2 are displayed side by side, either vertically or horizontally.

FIG. 8 is a schematic diagram showing an example of a detection log screen.
As shown in FIG. 8, on the detection log screen, log data in which object detection results are recorded is displayed.
In the example shown in FIG. 8, a list of dates and times when an abnormality was detected is displayed in response to an operation instructing display of log data.
In addition to displaying the log data as shown in FIG. 8, it is also possible to download the log data as a file to the user terminal 10.

FIG. 9 is a schematic diagram showing an example of a condition setting screen.
As shown in Figure 9, on the condition setting screen, the input image is subjected to various filter processing (processing using a noise filter or illumination variation filter, etc.), and further, binarized inter-frame difference processing is performed on the input image. The detected image, detailed information on the detection result, and abnormal value indicators are displayed. The condition setting screen also displays button icons for making settings for each category of "basic settings,""time zone settings,""detectionsettings," and "maintenance."

When "basic settings" is selected, settings for the detection area and frame rate are accepted. Specifically, when the user performs an operation to set the detection area on the condition setting screen, the setting of the object detection area in the image is accepted, and the user sets the object detection area using a pointing device such as a mouse. can do. In addition, when the user performs an operation to set the frame rate on the condition setting screen, the frame rate setting of the video for object detection is accepted, and the user can set the frame rate according to the purpose. .

When "time zone setting" is selected, the time zone setting indicating the time zone of the day is accepted. Specifically, when the user performs an operation to set the time zone on the condition setting screen, the setting for the time zone (time zone) in which objects in the video are detected is accepted, and the time to specify the time zone is accepted. input is accepted. In the time zone set at this time, conditions for detecting an object can be set for each time zone, as described later.
Therefore, by setting the time zone, it becomes possible to detect objects in a video under more appropriate conditions in response to changes in sunlight conditions, etc.

When "Detection Settings" is selected, the settings of the conditions for detecting objects for each set time zone are accepted. Specifically, when the user performs an operation on the condition setting screen to set the conditions for detecting objects for each time zone, the selection of the time zone is accepted, and further, the input of the operating parameters for the selected time zone is accepted.

Operational parameters include noise filter, illumination variation filter, binarization threshold, horizontal block size, vertical block size, feature size, moving average of features, moving average of abnormal values, detection direction, and abnormal value determination. It is possible to set the threshold value for the alert, the alert delay time, etc.
In the noise filter settings, you can set whether to apply a noise filter (smoothing filter, etc.), and also set the size of the noise filter (3 x 3 pixels or 5 x 5 pixels, etc.). .

In the illumination fluctuation filter setting, it is possible to set whether or not to apply an illumination fluctuation filter (such as a Sobel filter).
In the binarization threshold setting, a threshold for generating a binarized inter-frame difference image for an input image can be set.
In the horizontal block size setting, the number of blocks into which the image is divided in the horizontal direction can be set.
In the vertical block size setting, the number of blocks into which an image is divided in the vertical direction can be set.
Note that object detection is performed for each block determined by the set horizontal block size and vertical block size.

In setting the feature amount size, it is possible to set the maximum displacement amount when the feature amount is calculated by the pixel value calculation unit 215 in the server 20 described above.
In setting the moving average of the feature amount, it is possible to set the size of the time window for integration processing for the calculated feature amount.
When setting the moving average of abnormal values, it is possible to set the size of the time window for integration processing for detected abnormal values.

In setting the detection direction, it is possible to set the moving direction of the object to be detected in the video. When setting the detection direction, the angle representing the moving direction of the object to be detected (any of 360 degrees with the 0 o'clock direction as 0 degrees), the allowable error angle (width relative to the moving direction), the lower or upper speed threshold, It is possible to set whether or not the object to be detected moves in both directions (whether movement in the opposite direction is also detected).
When setting a threshold value for determining an abnormal value, the threshold value for determining an abnormal value can be arbitrarily set according to the user's purpose.

In setting the alert delay time, it is possible to set a time for determining how long a state determined to be an abnormal value continues before an alert is output.
In FIG. 9, when "maintenance" is selected, an operation to return the setting of the operating parameter to the initial value or an operation to display a log screen showing the transition of abnormal values is accepted.

Returning to FIG. 5, the data management unit 112 stores and reads video data in the video data storage unit 171, stores and reads various parameters in the parameter storage unit 172, and sends and receives data to and from the server 20. to manage. For example, the data management unit 112 reads video data to be displayed on the live view screen from the video data storage unit 171, or stores data on various parameters set on the UI screen in the parameter storage unit 172. Further, when requesting the server 20 to detect an object in the video, the data management unit 112 transmits various parameters stored in the parameter storage unit 172 to the server 20, or sends data of the detection results of the object in the video to the server 20. from the server 20.

[motion]
Next, the operation of the information processing system 1 will be explained.
[UI control processing]
FIG. 10 is a flowchart showing the flow of the UI control process executed by the user terminal 10.
The UI control process is started in response to an instruction to execute the UI control process being input via the input unit 815 of the user terminal 10 .

When the UI control process is started, in step S1, the UI control unit 111 displays an initial screen of a user interface for detecting an object in the video. Icons for transitioning to various UI screens are displayed on the initial screen of the user interface.

In step S2, the UI control unit 111 determines whether the user has selected an icon for transitioning to a live view screen.
If the user has not selected an icon for transitioning to the live view screen, a negative determination is made in step S2, and the process proceeds to step S4.
On the other hand, if the user selects the icon for transitioning to the live view screen, YES is determined in step S2, and the process proceeds to step S3.

In step S3, the UI control unit 111 displays a live view screen. When the live view screen is displayed, various operations related to the live view screen are appropriately accepted. When an icon for transitioning to the live view screen is selected, the UI control unit 111 makes a request to the server 20 regarding detection of an object in the video.
In step S4, the UI control unit 111 determines whether or not the user has selected an icon for transitioning to the detection log screen.
If the user has not selected an icon for transitioning to the detection log screen, the result of step S4 is determined to be NO, and the process proceeds to step S6.
On the other hand, if the user selects the icon for transitioning to the detection log screen, the result of the determination in step S4 is YES, and the process proceeds to step S5.

In step S5, the UI control unit 111 displays a detection log screen. When the detection log screen is displayed, various operations related to the detection log screen (log data download request, etc.) are accepted as appropriate.
In step S6, the UI control unit 111 determines whether an icon for transitioning to the condition setting screen has been selected.
If the icon for transitioning to the condition setting screen is not selected, the determination in step S6 is NO, and the process moves to step S8.
On the other hand, if the icon for transitioning to the condition setting screen is selected, YES is determined in step S6, and the process moves to step S7.

In step S7, the UI control unit 111 displays a condition setting screen. When the condition setting screen is displayed, various operations related to the condition setting screen are accepted as appropriate.
In step S8, the UI control unit 111 determines whether an instruction has been given to end the UI control process.
If the end of the UI control process has not been instructed, a negative determination is made in step S8, and the process proceeds to step S1.
On the other hand, if an instruction is given to end the UI control process, a YES determination is made in step S8, and the UI control process ends.

Next, the processing executed by the server 20 will be explained.
[Object detection processing]
FIG. 11 is a flowchart showing the flow of object detection processing executed by the server 20.
The object detection process is started when an instruction to execute the object detection process is input from the input unit 815 of the server 20 or another device.

When the object detection process is started, in step S31, the data acquisition unit 211 acquires video data for detecting an object. At this time, the data acquisition unit 211 acquires video data one frame at a time in time series, and if the image data of the acquired frame is a color image, converts it into a grayscale image.
In step S32, the image filter processing unit 213 performs various filter processing (processing using a noise filter, an illumination variation filter, etc.) on the image of the frame in which an object is to be detected, according to the user's settings.

In step S33, the image binarization processing unit 214 generates a binarized inter-frame difference image between frames in the image of the frame of the video in which the object is detected.
In step S34, the image binarization processing unit 214 sets the values of pixels other than the object detection area set by the user to 0 in the generated binarized inter-frame difference image.
In step S35, the pixel value calculation unit 215 calculates a feature amount r(t) that quantifies the moving speed (moving direction and speed) of the object moving within the object detection area.

In step S36, the pixel value calculation unit 215 calculates a moving average feature amount R(t) of the calculated feature amount r(t) using the set time window.
In step S37, the pixel value calculation unit 215 weights each element in the calculated moving average feature amount R(t), and the sum s of each element value of the weighted feature amount R'(t), which is the weighted result. Calculate (t)=Σ _x Σ _y R'(t, x, y). In addition, in step S37, the pixel value calculation unit 215 calculates the maximum element coordinate {(dx, dy)}=arg max {R'(t, dx, dy)} in the calculated weighted feature amount R'(t). , (dx, dy)∈N ² The direction vector from the center coordinates (0, 0) is detected and obtained as the moving speed.
In step S38, the pixel value calculation unit 215 uses the set time window to calculate the time integral S(t) of the sum s(t) of the elements of the weighted feature amount.

In step S39, the condition determination unit 216 determines whether or not the time integral S(t) of the sum s(t) of the elements of the weighted feature satisfies the condition for detecting it as an abnormal value (for example, if the (Whether the threshold value is exceeded or not) is determined.
If the time integral S(t) of the sum s(t) of the elements of the weighted feature does not meet the conditions for detection as an abnormal value, the determination in step S39 is NO, and the process moves to step S41. do.
On the other hand, if the time integral S(t) of the sum s(t) of the elements of the weighted feature satisfies the conditions for detection as an abnormal value, the determination in step S39 is YES, and the process moves to step S40. do.

In step S40, the visualization processing unit 217 outputs an alert to the user terminal 10 indicating that the object in the video is in an abnormal state (for example, exceeds a set threshold).
In step S41, the visualization processing unit 217 generates data of various information representing the detection results of objects in the video (for example, detection result data included in the live view screen), and the detection result presentation unit 218 performs visualization processing. The unit 217 sequentially transmits data of various types of information representing the detection results of objects in the video to the user terminal 10 .

In step S42, the data acquisition unit 211 determines whether or not the video data for object detection has ended.
If the image data for object detection has not ended, a NO determination is made in step S42, and the process proceeds to step S31.
On the other hand, if the video data for object detection has been exhausted, a YES determination is made in step S42, and the object detection process ends.

Through such processing, it is possible to detect the movement of an object in a specific direction in a video with intuitive settings, without having to prepare a huge number of learning samples as in the case of AI applications. Further, it is possible to provide a user interface that allows easy setting of conditions for selecting and detecting objects moving in a specific direction in a video.
Therefore, according to the information processing system 1 according to the present embodiment, movement of an object in a video in a specific direction can be detected more easily and with high accuracy.
Furthermore, it is possible to estimate the velocity of an object in an image robustly against disturbances.

Note that it is possible to perform multiple object detection processes on one video data. For example, in road video data, the up lane area and the down lane area may be treated as separate object detection areas. The object detection process may be performed for each object detection area.
Furthermore, it is also possible to execute object detection processing by switching operation parameters depending on the time zone.

[Modification 1]
In the embodiment described above, a client-server type information processing system 1 is constructed, the server 20 executes object detection processing, and the result of the object detection processing is transmitted to the user terminal 10.
In contrast, by providing a single device (for example, user terminal 10, etc.) with a function for executing object detection processing and a function for displaying the result of object detection processing, the functions of information processing system 1 can be integrated into a single device. It is also possible to implement the information processing apparatus 800 (that is, implement it as a stand-alone system).

FIG. 12 is a block diagram showing the functional configuration of a stand-alone information processing device 800 that executes object detection processing and display of processing results.
As shown in FIG. 12, when configured as a standalone type, a single information processing device 800 includes a UI control section 111 of the user terminal 10, a data acquisition section 211 of the server 20, a parameter setting section 212, and an image filter processing section 213. , the functions of the image binarization processing section 214, pixel value calculation section 215, condition determination section 216, visualization processing section 217, and detection result presentation section 218 are provided in the CPU 811, and the video data storage section 271 and parameter storage provided in the server 20 are provided. The storage unit 817 may include the storage unit 272 and the log data storage unit 273.

[Modification 2]
In the embodiment described above, it is possible to provide the imaging device C with a function for executing the object detection process of the server 20.
In this case, the imaging device C can be configured as a camera having a function of automatically detecting movement of an object in a specific direction within the video.
Then, by transmitting the processing results of the object detection processing executed in the imaging device C to the user terminal 10 or the server 20, the processing results can be displayed or recorded.

As described above, the information processing system 1 according to the present embodiment includes the user terminal 10 used by the user and the server 20 configured to be able to communicate with the user terminal 10.
The user terminal 10 includes a UI control section 111.
The UI control unit 111 requests the server 20 to perform object detection processing to detect an object moving within a set detection area in the video to be detected.
The server 20 includes a pixel value calculation section 215, a condition determination section 216, and a detection result presentation section 218.
In response to a request from the user terminal 10, the pixel value calculation unit 215 and the condition determination unit 216 calculate the correlation between the data of the pixels in the detection area and the pixels around the pixel in video data at different times in the video to be detected. Based on this, an object detection process is executed to detect an object moving within the detection area.
The detection result presentation unit 218 presents (for example, transmits) the object detection result obtained by the condition determination unit 216 to the user terminal 10 .
This makes it possible to estimate the object velocity in the video robustly against disturbances.

The pixel value calculation unit 215 and the condition determination unit 216 acquire a correlation value for each pixel between data of a pixel within the detection area and pixels around the pixel in video data at different times, and calculate a correlation value consisting of the acquired correlation value. An object moving in a specific direction within a set detection area is detected based on a feature quantity obtained by weighting data in accordance with movement in a specific direction.
As a result, movement of an object in a video in a specific direction can be detected more accurately and more easily with intuitive settings without having to prepare a huge number of learning samples for AI learning.

The pixel value calculation unit 215 and the condition determination unit 216 obtain the moving direction and moving speed of any object in the video based on the element values of the weighted feature amount R'(t).
This makes it possible to detect with higher accuracy an object moving in the direction and at the speed set by the user.

The UI control unit 111 displays a condition setting screen for setting detection conditions including the moving direction of the object to be detected.
Further, the UI control unit 111 requests the server 20 to perform object detection processing according to the detection conditions set on the condition setting screen.
This makes it possible to set detection conditions through intuitive operations and detect movement of an object in a specific direction within a video.

The UI control unit 111 displays the video and detection area based on the detection results presented by the server 20, and also identifies and displays the detected object.
With this, it is possible to display an object moving within the video that is detected by the object detection process in an easy-to-understand format.

The UI control unit 111 sets a plurality of detection areas in the video to be detected, and requests the server 20 to perform object detection processing.
The pixel value calculation unit 215 and the condition determination unit 216 detect objects moving within each of the plurality of detection areas.
The detection result presentation unit 218 presents the object detection results obtained by the pixel value calculation unit 215 and the condition determination unit 216 for each of the plurality of detection areas to the user terminal 10.
This makes it possible to set more complex detection conditions and detect the movement of an object within a video.

Further, the information processing device 800 (the server 20 or the information processing device 800 of the first modification) according to the present embodiment includes a pixel value calculation section 215, a condition determination section 216, and a detection result presentation section 218.
The pixel value calculation unit 215 and the condition determination unit 216 determine the detection area based on the correlation between pixels within the set detection area and data of pixels around the pixel in video data at different times in the video to be detected. Execute object detection processing to detect objects moving within the vehicle.
The detection result presenting unit 218 presents (for example, displays or transmits) the object detection result obtained by the condition determining unit 216.
This makes it possible to estimate the object velocity in the video robustly against disturbances.

It should be noted that the present invention is not limited to the above-described embodiments, and any modifications, improvements, etc. that can achieve the purpose of the present invention are included in the present invention.
For example, in the above embodiment, when configuring the client-server type information processing system 1, the combination of information processing apparatuses configuring the system is not limited to the example shown in the above embodiment.
For example, it is possible to distribute and implement the functions provided in the server 20 to more servers, or to collectively implement the functions of the user terminal 10, server 20, or imaging device C in fewer devices. .

The series of processes described above can be executed by hardware or software.
In other words, the functional configurations in the embodiments described above are merely examples and are not particularly limited. In other words, it is sufficient that any computer constituting the information processing system 1 is equipped with a function that can execute the series of processes described above as a whole, and it is not particularly important what kind of functional blocks are used to realize this function. It is not limited to the example shown.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.

Furthermore, the recording medium containing the program for executing the series of processes described above is not only configured as a removable medium that is distributed separately from the device itself in order to provide the program to the user, but also as a removable medium that is pre-installed in the device itself. It consists of recording media, etc. that are provided to users in a state in which they are

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Further, the effects described in this embodiment are only a list of the most preferable effects resulting from the present invention, and the effects according to the present invention are not limited to those described in this embodiment.

1 information processing system, 10 user terminal, 20 server, 30 network, C imaging device, 111 UI control unit, 112 data management unit, 171, 271 video data storage unit, 172, 272 parameter storage unit, 173, 273 log data storage unit, 211 data acquisition unit, 212 parameter setting unit, 213 image filter processing unit, 214 image binarization processing unit 215 pixel value calculation unit, 216 condition determination unit, 217 visualization processing unit, 218 detection result presentation unit, 800 information processing Device, 811 CPU, 812 ROM, 813 RAM, 814 Bus, 815 Input section, 816 Output section, 817 Storage section, 818 Communication section, 819 Drive, 820 Imaging section, 831 Removable media

Claims (9)

An information processing system including a terminal device used by a user and a server configured to be able to communicate with the terminal device,
The terminal device is
comprising a detection requesting means for requesting the server to perform detection processing for detecting an object moving within a set detection area in a video to be detected;
The server is
In response to a request from the terminal device, in a difference image representing a difference between two frames included in data of the video to be detected, data of a pixel of interest within the detection area in one difference image, and data of the pixel of interest within the detection area in one difference image; executing the detection process of detecting an object moving within the detection area based on correlation with data of pixels located in a predetermined range around the pixel of interest in another difference image at a different time from the two difference images; an object detection means;
Detection result presentation means for presenting a detection result of an object by the object detection means to the terminal device;
Equipped with
The object detection means calculates a correlation value between data of the pixel of interest in the detection area in the one difference image and data of pixels located in a predetermined range around the pixel of interest in the other difference image. Based on the feature amount obtained by weighting data consisting of the correlation values obtained for each pixel in the range and corresponding to movement in a specific direction, the An information processing system that detects objects moving in a specific direction . The information processing system according to claim 1 , wherein the object detection means acquires the moving direction and moving speed of an arbitrary object in the video based on the element value of the feature amount. The terminal device is
comprising a condition setting screen display means for displaying a condition setting screen for setting detection conditions including a moving direction of the object detected by the object detecting means;
3. The information processing system according to claim 1 , wherein the detection requesting means requests the server to perform the detection processing according to the detection conditions set on the condition setting screen. The terminal device is
2. The apparatus according to claim 1, further comprising a detection result display means for displaying the image and the detection area, and identifying and displaying the detected object based on the detection result presented from the server. 3. The information processing system according to any one of 3 . The detection requesting means of the terminal device sets a plurality of detection areas in the video to be detected and requests the server to perform the detection processing;
The object detection means of the server detects an object moving within the detection area for each of the plurality of detection areas,
5. The detection result presentation means presents the detection results of the object by the object detection means for each of the plurality of detection areas to the terminal device, according to any one of claims 1 to 4 . information processing system. A terminal device constituting the information processing system according to claim 1,
a condition setting screen display means for displaying a condition setting screen for setting detection conditions for an object to be detected by the object detection means;
a detection result display means for displaying the image and the detection area based on the detection result presented by the server, and for identifying and displaying the detected object;
Equipped with
The terminal device is characterized in that the detection request means makes a request for the detection process to the server in accordance with the detection conditions set on the condition setting screen. In a difference image representing the difference between two frames included in data of a video to be detected, data of a pixel of interest within a set detection area in one difference image and data at a time different from that of the one difference image. Object detection means for performing detection processing for detecting an object moving within the detection area based on correlation with data of pixels located in a predetermined range around the pixel of interest in other difference images ;
Detection result presentation means for presenting a detection result of an object by the object detection means;
Equipped with
The object detection means calculates a correlation value between data of the pixel of interest in the detection area in the one difference image and data of pixels located in a predetermined range around the pixel of interest in the other difference image. Based on the feature amount obtained by weighting data consisting of the correlation values obtained for each pixel in the range and corresponding to movement in a specific direction, the An information processing device characterized by detecting an object moving in a specific direction . An information processing method executed by an information processing system, the method comprising:
In a difference image representing the difference between two frames included in data of a video to be detected, data of a pixel of interest within a set detection area in one difference image and data at a time different from that of the one difference image. an object detection step of performing a detection process to detect an object moving within the detection area based on correlation with data of pixels located in a predetermined range around the pixel of interest in another difference image ;
a detection result presentation step of presenting the object detection result in the object detection step;
including;
In the object detection step, the correlation value between the data of the pixel of interest in the detection area in the one difference image and the data of pixels located in a predetermined range around the pixel of interest in the other difference image is determined by the predetermined value. Based on the feature amount obtained by weighting data consisting of the correlation values obtained for each pixel in the range and corresponding to movement in a specific direction, the An information processing method characterized by detecting an object moving in a specific direction . to the computer,
In a difference image representing the difference between two frames included in data of a video to be detected, data of a pixel of interest within a set detection area in one difference image and data at a time different from that of the one difference image an object detection function that executes a detection process to detect an object moving in a specific direction within the detection area based on correlation with data of pixels located in a predetermined range around the pixel of interest in other difference images; ,
a detection result presentation function that presents a detection result of an object by the object detection function;
Realize ,
The object detection function calculates a correlation value between data of the pixel of interest in the detection area in the one difference image and data of pixels located in a predetermined range around the pixel of interest in the other difference image. Based on the feature amount obtained by weighting data consisting of the correlation values obtained for each pixel in the range and corresponding to movement in a specific direction, the A program that detects objects moving in a specific direction .

Images