JP5834196B2 - MONITORING DEVICE, MONITORING SYSTEM, AND MONITORING METHOD - Google Patents

Description

  The present invention relates to a monitoring apparatus, a monitoring system, and a monitoring method for generating and outputting an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image.

  In stores such as convenience stores, a monitoring system that installs a camera that captures the inside of the store and monitors the person in the store with the image of the camera is widely used. When using captured images for the purpose of improving services, it is necessary to protect the privacy of customers.

  Under such circumstances, the whole person area in each image of the video shot by the camera is displayed on the image converted into an image filled with a predetermined color, and the feature point is displayed to determine how the person moves. A technique for grasping is known (see Patent Document 1). In addition, a background image without a foreground image of a person is generated, and the foreground image is subjected to mask processing or filter processing that is partially or wholly transparent so that the face or figure of the person cannot be specified. There is also known a technique for generating (see Patent Document 2).

JP 2013-186838 A Japanese Patent No. 5159381

  Now, in order to grasp the action of a person from the video, the person is seen while distinguishing the person from the background. However, as in the conventional technique, by replacing the image area of the person with a specific image, And a person can be easily identified, and it becomes easy to grasp the behavior of the person.

  On the other hand, there is a request to grasp how the product changes with the action of the person, but the conventional technology does not give any consideration to such a request, and the work to grasp the change state of the product There was a problem that it could not be performed efficiently. In particular, in the video that can grasp the behavior of the person, it is difficult to understand the state of the product. A technique that can be performed efficiently is desired.

  The present invention has been devised to solve such problems of the prior art, and a monitoring device that allows a user to efficiently perform a task of grasping a change state of a product together with a person's action, The main purpose is to provide a monitoring system and a monitoring method.

The monitoring device of the present invention is a monitoring device that generates and outputs a mask video in which a person's image area detected from a video image capturing a monitoring area is changed to a mask image , and detects a person from the video image A person detection unit that acquires region information for each person, an attribute determination unit that determines an attribute for each person detected from the moving image , and an article in a background image region in the moving image A background image analysis unit that acquires article change information relating to a change state of the image, a mask image setting unit that sets the mask image for each attribute, and the region information and the attribute for each person output from the person image analysis unit the basis to generate the contour of each person, along with changing the internal contour lines for each of the mask image the attributes, exits from the background image analysis unit Based on the article change information, and a video output controller for generating and outputting said mask image change status of the article in the image region is identifiably displayed background, person by reproducing the mask image It is set as the structure which can grasp | ascertain the behavior tendency and the change condition of the said article | item .

The monitoring system of the present invention is a monitoring system that generates and outputs a mask video in which a person's image area detected from a moving image captured in the monitoring area is changed to a mask image, and the camera that shoots the monitoring area And a plurality of information processing devices, and any one of the plurality of information processing devices detects a person from the moving image and acquires area information for each person, and the moving image A person image analysis unit having an attribute determination unit for determining an attribute for each detected person, a background image analysis unit for acquiring article change information regarding a change state of an article in a background image area in the video image , and the attribute and the mask image setting unit for setting different the mask image, on the basis of the area information and the attributes of each person that is output from the personal image analysis unit, Generating an outline of each object, as well as change the internal contour lines for each of the mask image the attribute, on the basis of the article change information output from said background image analyzing section, of the article in the image area of the background A video output control unit that generates and outputs the mask video in which the change status is displayed in an identifiable manner, and is configured to be able to grasp the behavior tendency of the person and the change status of the article by playing the mask video .

Further, the monitoring method of the present invention is a monitoring method for causing an information processing apparatus to perform processing for generating and outputting a mask video in which a person's image area detected from a moving image captured in a monitoring area is changed to a mask image. Te, a step having the step, and the step of determining the attribute of each detected person from the moving image to obtain the area information of each person by detecting a person from the video image, the background image in the moving image Acquiring article change information relating to an article change state in an area; setting the mask image for each attribute ; generating an outline for each person based on the area information and the attribute for each person; with changing the internal contour lines for each of the mask image the attribute, on the basis of the article change information, put in the image area of the background Anda step of generating and outputting the mask image which is identifiably displayed changes status of the article, configuration and which enables grasping a change status behavioral trends and the article of the person by the reproduction of the mask image To do.

According to the present invention, since the mask video in which the inside of the contour line for each person is changed to the mask image for each attribute is displayed, it is possible to easily grasp the attributes of the person appearing in the video, It is possible to efficiently perform a task of grasping a person's behavior tendency for each attribute. And since the change state of an article is displayed in an identifiable manner, the user can identify a changing article, thereby enabling the user to efficiently perform the task of grasping the change state of the product together with the behavior of the person. Can do.

Overall configuration diagram of the monitoring system according to the first embodiment Floor plan of the store explaining the layout of the store and the installation status of the camera 1 Functional block diagram showing schematic configuration of PC3 Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing which shows the principal part of the image | video displayed on the video display part 48 of a monitoring screen. Functional block diagram which shows schematic structure of PC3 in the monitoring system which concerns on 2nd Embodiment. Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing showing images of two shooting dates and times Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Functional block diagram which shows schematic structure of PC3 in the monitoring system which concerns on 3rd Embodiment. Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Explanatory drawing which shows another example of the monitoring screen displayed on the monitor 4

1st invention made | formed in order to solve the said subject is a monitoring apparatus which produces | generates and outputs the mask image | video which changed the image area of the person detected from the moving image image | video which image | photographed the monitoring area into the mask image , a portrait image analysis unit having the person detection section that acquires area information for each person to detect a person from the video image, and determines the attribute determination unit attributes of each detected person from the moving image, the moving image A background image analysis unit that obtains article change information related to a change state of an article in a background image area, a mask image setting unit that sets the mask image for each attribute, and a person output from the person image analysis unit based on the area information and the attributes of each, to generate a contour of each person, along with changing the internal contour lines for each of the mask image the attributes, the back Based on the article change information output from the image analysis unit, and a video output controller for generating and outputting said mask image change status of the article in the image region is identifiably displayed background, the mask It is configured such that a person's behavior tendency and the state of change of the article can be grasped by reproducing the video .

According to this, since the mask video in which the inside of the contour line for each person is changed to a mask image for each attribute is displayed, the attributes of the person appearing in the video can be easily grasped. In addition, it is possible to efficiently perform the work of grasping the person's behavior tendency. And since the change state of an article is displayed in an identifiable manner, the user can identify a changing article, thereby enabling the user to efficiently perform the task of grasping the change state of the product together with the behavior of the person. Can do.

  Moreover, 2nd invention is set as the structure which the said video output control part displays so that at least any one of the loss | disappearance and appearance of the said article can be identified as a change condition of the said article.

  According to this, since it is displayed so that at least one of the disappearance and appearance of the article can be identified, it is possible to grasp the change state of the product together with the action of the person.

  According to a third aspect of the present invention, the mask image setting unit sets the mask image for each attribute according to an input operation of a user who selects the mask image for each attribute.

  According to this, since the user can freely set the mask image, the convenience for the user can be improved.

  According to a fourth aspect of the invention, in the mask image, the attribute can be identified by changing a display element of at least one of color, shading, pattern, and contour line according to the attribute. To do.

  According to this, persons with different attributes can be easily identified due to the difference in mask images.

Further, the fifth invention, the video output control unit, in order to distinguishably display the change status of the article in the image area of the background, the mask changing the image area of the changed articles transmissive mask image It is configured to generate a video .

  According to this, the user can easily identify the changed article. Since the mask image of the article is transmissive, the background image can be seen through, so that the current situation of the article arrangement area (display shelf, etc.) shown in the background image can be grasped.

The sixth aspect of the invention further includes a background image processing unit that generates a background image from the moving image , and the video output control unit displays the mask video obtained by superimposing the transparent mask image on the background image. A configuration to generate.

  According to this, since the mask image of the person is transmissive, the background image can be seen through, so that the action of the person can be observed while observing the situation of the monitoring area shown in the background image.

According to a seventh aspect of the present invention, there is provided a monitoring system for generating and outputting a mask video in which an image area of a person detected from a moving image captured in a monitoring area is changed to a mask image , and a camera for shooting the monitoring area; And a plurality of information processing devices, and any one of the plurality of information processing devices detects a person from the moving image and acquires area information for each person, and is detected from the moving image. A person image analysis unit having an attribute determination unit that determines an attribute for each person, a background image analysis unit that acquires article change information related to a change state of an article in a background image area in the moving image , and and the mask image setting unit configured to set the mask image, on the basis of the area information and the attributes of each person that is output from the personal image analysis unit, the contour of each person Form, together with changing the internal contour lines for each of the mask image the attribute, on the basis of the article change information output from said background image analysis unit, to be identified change status of the article in the image area of the background And a video output control unit that generates and outputs the displayed mask video , and is configured to be able to grasp a behavior tendency of a person and a change state of the article by reproducing the mask video .

  According to this, similarly to the first invention, the user can efficiently perform the work of grasping the change state of the product together with the behavior of the person.

The eighth invention is a monitoring method for causing an information processing apparatus to perform processing for generating and outputting a mask video in which a person's image area detected from a moving image captured in a monitoring area is changed to a mask image. the steps having the step, and the step of determining the attribute of each detected person from the moving image to obtain the area information of each person by detecting a person from the moving image, the image area of the background in the video image Acquiring an article change information related to the change status of the article in step, setting the mask image for each attribute, generating a contour line for each person based on the region information and the attribute for each person, with changing the internal contour for each of the mask image the attribute, on the basis of the article change information, varying of the article in the image area of the background Comprising a step of generating and outputting the mask image displayed distinguishably the situation, and to ascertain possible and with the structure changes conditions of behavior trends and the article of the person by the reproduction of the mask image.

  According to this, similarly to the first invention, the user can efficiently perform the work of grasping the change state of the product together with the behavior of the person.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of a monitoring system according to the first embodiment. This monitoring system is constructed for a retail chain store such as a convenience store. A camera (imaging device) 1, a recorder (image recording device) 2, a PC (monitoring) provided for each of a plurality of stores. Device) 3 and a monitor (display device) 4, and a PC 11 and a monitor 12 provided in a headquarters for generalizing a plurality of stores.

  The camera 1 is installed at an appropriate place in the store, the inside of the store is imaged by the camera 1, and image information obtained thereby is recorded in the recorder 2. With the PC 3 provided in the store and the PC 11 provided in the headquarters, the store image captured by the camera 1 can be viewed in real time, and the past store image recorded in the recorder 2 can be viewed. It is possible to check the situation in the store at the store or the headquarters.

The PC 3 installed in the store is configured as a monitoring device that monitors the state of persons and products in the store, and the monitoring information generated by the PC 3 can be browsed by the PC 3 itself, and further installed in the headquarters. Transmitted to the PC 11 and can be browsed by the PC 11 , and the PCs 3 and 11 are configured as browsing devices for browsing the monitoring information.

  Next, the store layout and the installation status of the camera 1 will be described using a convenience store as an example. FIG. 2 is a plan view of the store explaining the store layout and the installation status of the camera 1.

  The store is provided with an entrance, a display shelf (display area), and a cashier counter. Display shelves are divided into different types of products such as bento boxes, plastic bottles, and rice balls. Next to the checkout counter is a fast food display shelf.

  The customer enters the store from the entrance, moves through the passage between the display shelves, finds the desired product, and goes to the checkout counter with the product and pays at the checkout counter (payment). After exiting, exit from the entrance.

  In the store, a plurality of cameras 1 that capture images of the store (monitoring area) are installed. In particular, in the example shown in FIG. 2, an omnidirectional camera having a 360-degree shooting range using a fish-eye lens is adopted as the camera 1, and the camera 1 can image products displayed on the display shelf, It is also possible to take an image of a person staying in front of a display shelf or a cashier counter.

  Next, processing performed by the PC 3 shown in FIG. 1 will be described based on a functional block diagram. FIG. 3 is a functional block diagram showing a schematic configuration of the PC 3.

  The PC 3 includes a monitoring unit 31. The monitoring unit 31 causes the PC 3 to function as a monitoring system for monitoring the inside of the store. The operation of the camera 1 and the recorder 2 is controlled by the monitoring unit 31, and the inside of the store imaged by the camera 1 is controlled. The image can be browsed in real time, and the in-store image recorded in the recorder 2 can be browsed.

  The PC 3 includes a person image analysis unit 32, a background image processing unit 33, a video output control unit 34, and a mask image setting unit 35.

  The person image analysis unit 32 analyzes information captured by the camera 1 and acquires information about a person, and includes a person detection unit 37 and an attribute determination unit 38.

  The person detection unit 37 performs processing for detecting a person from the captured video and acquiring area information for each person (position information of the person's image area). The attribute determination unit 38 performs processing for determining attributes (such as gender and age) for each person detected from the captured video. The processing performed by the person detection unit 37 and the attribute determination unit 38 may use a known image recognition technique. The person image analysis unit 32 desirably performs person tracking processing for associating a person detection result for each frame of the captured video, thereby enabling highly accurate attribute determination.

  The mask image setting unit 35 performs processing for setting a mask image for each attribute. In particular, in the present embodiment, processing for setting a mask image for each attribute is performed in accordance with an input operation of a user who selects a mask image for each attribute of a person. A mask image for each attribute may be set in advance.

  In the background image processing unit 33, processing for generating a background image from the captured video is performed. This background image is obtained by removing a person image (foreground image) from a photographed video, removes a person image from each of the most recently photographed videos, and synthesizes the background image from the plurality of images. You can do that. Moreover, you may make it prepare previously as a background image the picked-up image when a person does not exist, for example, the picked-up image before a store opening.

  The video output control unit 34 generates and outputs an output video in which the person's image area is changed to a mask image based on the area information and attributes for each person output from the person image analysis unit 32. In the present embodiment, a mask image having an outline corresponding to an image area of a person in a captured video is generated, and a process of superimposing the mask image on the background image generated by the background image processing unit 33 is performed.

  The mask image is one in which attributes can be identified by changing display elements of at least one of color, shading, pattern (pattern), and outline according to the attributes. It is assumed that an outline of a person image area is generated from area information for each person (position information of the person image area), and the inside of the outline is filled with a transparent image. Note that the contour line may not be displayed, and the contour line may be displayed so as to be distinguishable from the filled region by changing the color, shading, or the like.

  Next, the monitoring screen displayed on the monitor 4 shown in FIG. 1 will be described. FIG. 4 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  This monitoring screen is used by the user to check the situation in the store for the purpose of marketing analysis and service improvement. The monitoring screen includes a store selection unit 41, a start button 42, a setting button 43, and an operation. A mode selection unit 44, a date display unit 45, a date setting button 46, a legend display unit 47, a video display unit 48, and a display time operation unit 49 are provided.

  In the store selection unit 41, the user selects a store from a pull-down menu. The start button 42 is used to start the monitoring process on the PC. The setting button 43 is used to set various conditions in the monitoring process. In this embodiment, when the setting button 43 is operated, a mask image setting screen (see FIG. 5) is displayed in a pop-up. The operation mode selection unit 44 selects an operation mode, and operation modes such as sales floor monitoring, product monitoring, display alert, and shortage alert are prepared. Here, sales floor monitoring is selected.

  The date and time display unit 45 displays the date and time. The date and time setting button 46 is for setting the date and time. When the date / time setting button 46 is operated, a date / time setting screen (not shown) is displayed. By selecting a date / time here, the selected date / time is displayed on the date / time display unit 45, and a video of the selected date / time is displayed on the video display unit 48. Is displayed.

  The legend display unit 47 is for explaining a mask image displayed on the video display unit 48. In the present embodiment, a mask image set for each person attribute (such as gender and age) is displayed on the legend display unit 47. In the example shown in FIG. 4, the ages are all generations. However, when a mask image is set for each generation, for example, teens to 60s, the legend display unit 47 displays a mask image of each generation.

  The video display unit 48 displays store videos (moving images). In particular, in the example shown in FIG. 4, two images obtained by panoramic development of an image captured by the camera 1 that is an omnidirectional camera are displayed. The entire store is reflected in these two images, and the status of the entire store can be confirmed. On the video display unit 48, the video captured by the camera 1 is displayed as it is in the normal playback mode, and the video in which the image area of the person is changed to a mask image is displayed in the mask playback mode. The video displayed on the video display unit 48 will be described in detail later.

  The display time operation unit 49 adjusts the time of the video displayed on the video display unit 48. By operating the display time operation unit 49, the video can be switched to a video at a desired time. Specifically, when the slider 50 is shifted using the input device 6 such as a mouse, an image at the time indicated by the slider 50 is displayed on the video display unit 48. Thereby, the situation of the store at the required time can be confirmed on the video of the video display unit 48. In addition, when the video on the video display unit 48 is played, the operation button 51 is used to enable fast-forward playback and rewind playback in addition to normal playback. In this case, even when a video recorded for a long time is played back in the mask playback mode, the video can be confirmed efficiently.

  Next, an operation for setting a mask image for each attribute will be described. FIG. 5 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  In this mask image setting screen, a user specifies a mask image for each attribute of a person. In this mask image setting screen, a person attribute selecting unit 61, a mask image selecting unit 62, a setting button 63, A setting content display section 64 is provided.

  In the person attribute selection unit 61, the user selects a person attribute (gender and age) from a pull-down menu. The person attribute selection unit 61 can select, for example, each generation of teens to 60s and all generations. It should be noted that the attributes of the person can be adults and children. In the mask image selection unit 72, the user selects a mask image corresponding to the attribute selected by the person attribute selection unit 61 from the tile menu.

  When the person attribute selection unit 61 and the mask image selection unit 62 select a person attribute and the corresponding mask image and operate the setting button 63, the setting content display unit 64 displays the setting content, that is, the mask for each attribute. An image is displayed. In the example shown in FIG. 5, mask images of men and women are set for all generations.

  Next, the video displayed on the video display unit 48 of the monitoring screen shown in FIG. 4 will be described. FIG. 6 is an explanatory diagram showing the main part of the video displayed on the video display unit 48 of the monitoring screen shown in FIG.

  As shown in FIG. 6A, in the normal playback mode, a video image taken by the camera 1 is displayed as it is. On the other hand, in the mask reproduction mode after setting the mask image, as shown in FIG. 6B, an image in which the person's image area is changed to a mask image is displayed. In this video, the image area of the person in the captured video of FIG. 6A is replaced with a mask image for each attribute (here, male and female). Further, since the mask image is transmissive, the background image can be seen through.

  When playing back a captured video, a playback mode identification icon may be superimposed on the video so that the user can identify which playback mode is being played back. Further, the playback mode may be identified by displaying the video display frame in the mask playback mode in a different color from the color of the video display frame in the normal playback mode.

  As described above, in the first embodiment, since different mask images are displayed for each attribute, for example, men and women, it is possible to easily grasp the attributes of the person appearing in the video. It is possible to efficiently perform a task of grasping a person's behavior tendency for each attribute.

  In addition, although sex and age were demonstrated as an example of a person's attribute, various directions, such as a person's direction, a race, an adult / child, a type (OL and office worker), etc. can be acquired as an attribute obtained using a well-known person recognition technique. It is possible to set the attribute. In particular, if the attribute of the direction of the person is used, it is possible to grasp the preference for the product for each person even in the mask reproduction mode.

(Second Embodiment)
Next, a second embodiment will be described. The points not particularly mentioned here are the same as in the first embodiment.

  FIG. 7 is a functional block diagram showing a schematic configuration of the PC 3 in the monitoring system according to the second embodiment. In the second embodiment, similarly to the first embodiment, the PC 3 includes a monitoring unit 31, a person image analysis unit 32, a background image processing unit 33, a video output control unit 34, a mask image setting unit 35, and the like. Further, in the second embodiment, the PC 3 includes a date / time setting unit 71.

  In the second embodiment, similarly to the first embodiment, it is possible to display mask images by person attributes (such as sex and age), but it is also possible to display mask images by date and time. Note that it is possible to display only mask images by date and time without displaying mask images by person attributes.

  The date and time setting unit 71 performs a process of setting a plurality of dates and times in accordance with a user input operation that specifies the shooting date and time. The mask image setting unit 35 performs a process of setting a mask image for each shooting date and time and for each person attribute according to a user input operation for selecting a mask image for each shooting date and time and a person attribute. The person image analysis unit 32 acquires captured images of a plurality of shooting dates and times set by the date and time setting unit 71 and acquires person area information (position information of person image areas) for each person for each shooting date and time. Is done.

  In the video output control unit 34, based on the region information for each person output from the person image analysis unit 32, the person image region for each of a plurality of shooting dates and times is changed to a mask image for each shooting date and time and for each person attribute. Processing to generate and output an output video is performed, and the output video is a mask image for each person at a plurality of dates and times displayed on one background image so that the shooting date and time and the attributes of the people can be identified. .

  In the second embodiment, the mask image can identify the attribute and the shooting date / time by changing the display element of at least one of color, shading, pattern, and contour line according to the attribute and the shooting date / time. In particular, in this embodiment, it is assumed that the inside of the outline of the person's image area is filled with a transparent image.

  Next, the monitoring screen displayed on the monitor 4 will be described. FIG. 8 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  Also in the second embodiment, a monitoring screen that is substantially the same as that of the first embodiment (see FIG. 4) is displayed on the monitor 4. In this monitoring screen, the legend display unit 47 displays a plurality of shooting dates and times and individual attributes. The set mask image is displayed.

  Next, an operation for setting a mask image for each attribute and shooting date / time will be described. FIG. 9 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  In this mask image setting screen, the user designates a mask image for each person attribute and shooting date / time. This mask image setting screen includes a person attribute selection unit 81, a date / time selection unit 82, and a mask image selection. A section 83, a setting button 84, and a setting content display section 85 are provided.

  In the person attribute selection unit 81, the user selects a person attribute (gender and age) from a pull-down menu. In the date and time selection unit 82, the user selects the date and time (year, month, day and time zone) from the pull-down menu. In the date and time selection unit 82, the user selects a priority order from a pull-down menu. In the mask image selection unit 83, the user selects a mask image corresponding to the attribute selected by the person attribute selection unit 81 and the shooting date / time selected by the date / time selection unit 82 from the tile menu.

  When the person attribute selection unit 81, the date and time selection unit 82, and the mask image selection unit 83 select a person attribute and shooting date and time and a corresponding mask image and operate the setting button 84, the setting content display unit 85 is displayed. The setting contents, that is, mask images for each attribute and shooting date / time are displayed. This setting operation is performed at least twice, and a plurality of mask images with shooting dates and times are set. In the example shown in FIG. 9, the same time zone of different dates is set as the two shooting dates and times.

  Next, an outline of processing performed in the video output control unit 34 and video displayed on the video display unit 48 of the monitoring screen will be described. FIG. 10 is an explanatory diagram showing images of two shooting dates and times. FIG. 11 is an explanatory diagram showing a main part of the video display unit 48 of the monitoring screen shown in FIG.

  FIG. 10A shows a photographed image at the first photographing date and time. FIG. 10B shows a photographed image at the second photographing date and time. The video output control unit 34 generates a mask image having a contour corresponding to the image area of the person in the captured image at the first shooting date and time, and the image area of the person in the captured image at the second shooting date and time. A mask image having an outline corresponding to is generated, and a process of superimposing the mask image on a corresponding position on the background image generated by the background image processing unit 33 is performed. As a result, as shown in FIG. 11, the mask image is displayed on the video display section 48 of the monitoring screen.

  Here, in this second embodiment, mask images with different shooting dates and times are displayed at the same time. Mask images with different shooting dates and times may overlap each other. As a method for displaying mask images with different shooting dates and times, In the second embodiment, as shown in FIG. 11A, a normal display mode in which display based on priority is not performed, and a priority display mode in which display based on priority is performed as shown in FIG. 11B. And can be selected.

  As shown in FIG. 11A, in the normal display mode in which display based on the priority order is not performed, an overlapping portion of mask images having different dates and times is a portion that does not overlap another portion of the mask image, that is, another person's mask image. And identifiable. In the example shown in FIG. 11A, the overlapping portion of the mask image is displayed as an image obtained by combining the mask images. For example, the overlapping portion of the red mask image and the blue mask image is displayed in purple, and the overlapping portion of the mask image is displayed with a transmittance obtained by adding the transmittance of each mask image.

  As shown in FIG. 11B, in the priority display mode in which the display based on the priority order is performed, the mask image with the higher priority date and the mask image with the lower priority date and time are the parts where the mask images with different date and time overlap. Is displayed so as to cover up.

  As described above, in the second embodiment, since different mask images are displayed for a plurality of shooting dates and times, for example, today and yesterday, it is possible to easily grasp the shooting date and time of the person appearing in the video. Thus, it is possible to efficiently perform the work of grasping the behavior tendency of the person by date and time. Here, if a plurality of shooting dates and times are set to the same time on different days, it is possible to compare a person's behavior tendency for each day, and if a plurality of shooting dates and times are set to morning and evening time zones on the same day, It is possible to compare a person's behavior tendency in the morning and evening time zones.

(Third embodiment)
Next, a third embodiment will be described. The points not particularly mentioned here are the same as in the first embodiment.

  FIG. 12 is a functional block diagram showing a schematic configuration of the PC 3 in the monitoring system according to the third embodiment. In the third embodiment, the PC 3 is similar to the first embodiment in that the monitoring unit 31, the person image analysis unit 32, the background image processing unit 33, the video output control unit 34, the mask image setting unit 35, and the like. Further, in the third embodiment, the PC 3 includes a background image analysis unit 91.

  The background image analysis unit 91 performs a process of acquiring article change information related to the disappearance and appearance of the product in the background image area in the captured video. In this background image analysis processing, a change state (disappearance or appearance) of a product is detected by comparing a plurality of background images obtained by removing a person image (foreground image) from a captured video, and region information (product Position information of the image area).

  The video output control unit 34 generates and outputs an output video that displays the change (disappearance or appearance) of the product in the background image area so as to be identifiable based on the article change information output from the background image analysis unit 91. Is done. Here, when the product disappears, the image area where the product existed in the video before disappearing may be changed to a transparent mask image. When the product appears, the image area of the product Should be highlighted (highlighted).

  For highlighting when a product appears, various appropriate methods may be used. For example, the outline of the image area of the product may be displayed, and the product is small and difficult to understand. Alternatively, an animation such as blinking may be displayed. Further, when a product appears, the image area of the product may be changed to a transmissive mask image in the same manner as when the product disappears. In addition, an appropriate mark image, for example, an image of a symbol such as ○ or × may be displayed at the position of the product that has changed (disappeared or appeared).

  In the third embodiment, the change state (disappearance or appearance) of the product can be displayed in an identifiable manner, and a mask image of a person can be displayed for each attribute, as in the first embodiment. In the unit 35, a process of setting a mask image is performed in response to a user's input operation for selecting a mask image for each attribute of the person. In the attribute determination unit 38, the attribute (gender) of each person detected from the captured video is processed. In addition, the video output control unit 34 generates and outputs an output video in which the person's image area is changed to a mask image for each attribute.

  Next, the monitoring screen displayed on the monitor 4 will be described. FIG. 13 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  Also in the third embodiment, a monitoring screen that is substantially the same as that of the first embodiment (see FIG. 4) is displayed on the monitor 4. In this monitoring screen, the mask image set for each attribute of the person on the legend display unit 47. Then, a mask image representing the lost product is displayed.

  Next, an operation for setting a mask image for each person attribute and a mask image for a change state (disappearance and appearance) of a product will be described. FIG. 14 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  This mask image setting screen sets a mask image for each attribute of a person and sets a mask image of a change state (disappearance and appearance) of a product. The mask image setting screen includes a person setting unit 101 and a mask image setting screen. A product setting unit 102 and a setting content display unit 103 are provided.

  As in the first embodiment (see FIG. 5), the person setting unit 101 is provided with a person attribute selection unit 104, a mask image selection unit 105, and a setting button 106, and the person attribute selection unit 104 is provided. When the person's attributes (gender and age) are selected with the button, the mask image corresponding to the person's attribute is selected by the mask image selection unit 105 and the setting button 106 is operated, the setting content display unit 103 displays the setting contents, A mask image for each attribute is displayed.

  The product setting unit 102 includes a change state selection unit 107, a mask image selection unit 108, and a setting button 109. In the change state selection unit 107, the user selects a change state (disappearance and appearance) from a pull-down menu. In the mask image selection unit 108, the user selects a mask image corresponding to the change state selected by the change state selection unit 107 from the tile menu. When the change state and the mask image selection unit 108 select the change state and the corresponding mask image and operate the setting button 109, the setting content display unit 103 displays the setting content, that is, the change state of the product. A mask image is displayed.

  Next, an outline of processing performed in the video output control unit 34 and video displayed on the video display unit 48 of the monitoring screen illustrated in FIG. 13 will be described. FIGS. 15 and 16 are explanatory diagrams showing the main part of the video display unit 48 of the monitoring screen shown in FIG.

  In the third embodiment, similarly to the first embodiment (see FIG. 6A), in the normal playback mode, the video image taken by the camera 1 is displayed as it is on the video display unit 48, and the mask image is set. In the mask reproduction mode, as shown in FIGS. 15A and 15B, an output video in which a person's image area is changed to a mask image is displayed.

  Here, when the customer in front of the display shelf takes away the product as shown in FIG. 16A-1 and the product disappears from the display shelf as shown in FIG. 16A-2, background image analysis is performed. The unit 91 detects the disappearance of the product, and the video output control unit 34 performs a process of changing the image area of the disappeared product on the background image to a mask image. Thereby, in the mask reproduction mode, as shown in FIG. 16B, the mask image is displayed at the position where the product is present.

  It should be noted that the display of the mask image and the highlight display indicating the change state (disappearance and appearance) of the product may be terminated when a predetermined time has elapsed from the time of the change (disappearance and appearance) of the product. Similarly to the mask image of a person, the product mask image generates an outline of the product image area from the product area information (position information of the product image area), and the inside of the outline is transparent. It is assumed that the image is filled with

  As described above, in the third embodiment, since the mask image is displayed for each attribute, it is possible to easily grasp the attribute of the person appearing in the video, thereby grasping the behavior tendency of the person for each attribute of the person. Work can be performed efficiently. And since the change state of an article is displayed in an identifiable manner, the user can identify a changing article, thereby enabling the user to efficiently perform the task of grasping the change state of the product together with the behavior of the person. Can do.

  In the third embodiment, both the disappearance and appearance of the product are detected, and the output video is displayed so that both the disappearance and appearance can be identified. It is good also considering only one side.

  FIG. 17 is an explanatory diagram illustrating another example of the monitoring screen displayed on the monitor 4. In the above example, the panoramic image of the image captured by the camera 1 that is an omnidirectional camera is displayed on the image display unit 48. In the example shown in FIG. 17, the image is captured by the camera 1 that is an omnidirectional camera. The video is displayed as it is.

  As mentioned above, although this invention was demonstrated based on specific embodiment, these embodiment is an illustration to the last, Comprising: This invention is not limited by these embodiment. In addition, each component of the monitoring device, the monitoring system, and the monitoring method according to the present invention described in the above embodiment is not necessarily essential, and can be appropriately selected as long as it does not depart from the scope of the present invention. It is.

  For example, in each of the first to third embodiments, an example of a retail store such as a convenience store has been described. However, the present invention is not limited to such a retail store, and is applied to a store having a business form other than a retail store. It is also possible. Furthermore, it is possible to use for monitoring areas other than stores.

  In each of the first to third embodiments, as shown in FIG. 2, the camera 1 is an omnidirectional camera having a 360-degree shooting range using a fisheye lens, but a camera having a predetermined angle of view. A so-called box camera is also possible.

  In each of the first to third embodiments, the output video is generated by superimposing the mask image for each person on the background image. However, only the mask image is displayed and the background image is not displayed. It is.

  In the first to third embodiments, necessary setting items are input by selecting a pull-down menu or a tile menu. However, the present invention is not limited to this, and an appropriate input unit is provided on the screen. What is necessary is just to be able to select with a radio button etc., or to be able to input a numerical value directly.

  Moreover, in each of the first to third embodiments, various configurations having characteristics are shown, but these configurations are not limited to the combinations shown in the first to third embodiments. It is also possible to appropriately combine the configurations shown separately in the first to third embodiments.

  Further, in each of the first to third embodiments, the processing necessary for monitoring is performed by the PC 3 provided in the store, but this necessary processing is performed in the headquarters as shown in FIG. You may make it carry out to the cloud computer 21 which comprises provided PC11 and a cloud computing system. In addition, necessary processing may be shared by a plurality of information processing apparatuses, and information may be transferred between the plurality of information processing apparatuses via a communication medium such as an IP network or a LAN. In this case, the monitoring system is composed of a plurality of information processing apparatuses that share necessary processing.

  In such a configuration, it is preferable to cause the PC 3 provided in the store to perform at least a process with a large calculation amount among processes necessary for monitoring, for example, a person detection process. With this configuration, the amount of information required for the remaining processing can be reduced, so that the remaining processing is performed by an information processing apparatus installed at a location different from the store, for example, the PC 11 installed at the headquarters. Even if it does in this way, since communication load can be reduced, operation of the system by a wide area network connection form becomes easy.

  In addition, among the processes necessary for monitoring, at least a process with a large calculation amount, for example, a person detection process may be performed by the cloud computer 21. If comprised in this way, since the amount of calculations of the remaining processes may be small, a high-speed information processing apparatus will become unnecessary for users, such as a shop, and the cost which a user will bear can be reduced.

  In addition, the cloud computer 21 may perform all the necessary processes, or at least the video output control process among the necessary processes may be shared by the cloud computer 21. With this configuration, the store or the head office In addition to the PCs 3 and 11 provided in the mobile phone, the mobile terminal such as the smartphone 22 or the tablet terminal 23 can display the video. You can check the situation in the store.

  In the present embodiment, the PC 3 installed in the store performs processing necessary for monitoring, and the monitoring screen and mask image setting screen are displayed on the monitor 4 of the PC 3 to perform necessary input and output on the PC 3. However, the necessary input and output are performed on an information processing apparatus different from the information processing apparatus that performs processing necessary for monitoring, for example, a portable terminal such as the PC 11 or the tablet terminal 23 installed in the headquarters. Also good.

  In recent years, 4K televisions and the like that have pursued high image quality have been developed as monitors, and by employing a camera that supports such 4K televisions, the ability to identify people can be improved, and there are many people. Even in such an environment, the user can easily grasp the person's behavior tendency from the mask reproduction video.

  The monitoring device, the monitoring system, and the monitoring method according to the present invention have an effect that allows a user to efficiently perform a task of grasping a change state of a product along with a person's action, and are detected from a captured image of a monitoring area. The present invention is useful as a monitoring device, a monitoring system, a monitoring method, and the like that generate and output an output video in which a human image region is changed to a mask image.

1 Camera 2 Recorder 3 PC
4 monitor 6 input device 12 monitor 21 cloud computer 22 smartphone 23 tablet terminal 32 human image analysis unit 33 background image processing unit 34 video output control unit 35 mask image setting unit 37 human detection unit 38 attribute determination unit 71 date and time setting unit 91 background image Analysis part

Claims (8)

A monitoring device that generates and outputs a mask video in which an image area of a person detected from a video image capturing a monitoring area is changed to a mask image ,
A portrait image analysis unit having the person detection section that acquires area information for each person to detect a person from the video image, and determines the attribute determination unit attributes of each detected person from said video image,
A background image analysis unit for acquiring article change information relating to a change situation of an article in a background image area in the video image ;
A mask image setting unit for setting the mask image for each attribute;
Based on the region information and the attributes for each person output from the person image analysis unit , a contour line for each person is generated, the inside of the contour line is changed to the mask image for each attribute, and the background image Based on the article change information output from the analysis unit, and a video output control unit that generates and outputs the mask video that displays the change state of the article in a background image region in an identifiable manner, and
A monitoring apparatus characterized in that a person's behavior tendency and change state of the article can be grasped by reproducing the mask video .   The monitoring apparatus according to claim 1, wherein the video output control unit displays at least one of disappearance and appearance of the article as a change state of the article in an identifiable manner.   The said mask image setting part sets the said mask image according to the said attribute according to the input operation of the user who selects the said mask image for every said attribute, The Claim 1 or Claim 2 characterized by the above-mentioned. Monitoring device.   2. The mask image according to claim 1, wherein the attribute can be identified by changing a display element of at least one of a color, shading, a pattern, and a contour line according to the attribute. Item 4. The monitoring device according to any one of Items 3 to 4. The video output control unit generates the mask video in which the image area of the changed article is changed to a transparent mask image in order to display the change state of the article in the background image area in an identifiable manner. The monitoring device according to any one of claims 1 to 4. A background image processing unit for generating a background image from the video image ;
The monitoring apparatus according to claim 1, wherein the video output control unit generates the mask video in which the transparent mask image is superimposed on the background image. A monitoring system that generates and outputs a mask video in which an image area of a person detected from a video image capturing a monitoring area is changed to a mask image ,
A camera that captures the surveillance area;
A plurality of information processing devices;
Have
Any of the plurality of information processing devices
A portrait image analysis unit having the person detection section that acquires area information for each person to detect a person from the video image, and determines the attribute determination unit attributes of each detected person from said video image,
A background image analysis unit for acquiring article change information relating to a change situation of an article in a background image area in the video image ;
A mask image setting unit for setting the mask image for each attribute;
Based on the region information and the attributes for each person output from the person image analysis unit , a contour line for each person is generated, the inside of the contour line is changed to the mask image for each attribute, and the background image Based on the article change information output from the analysis unit, and a video output control unit that generates and outputs the mask video that displays the change state of the article in a background image region in an identifiable manner, and
A monitoring system characterized in that it is possible to grasp a behavior tendency of a person and a change state of the article by reproducing the mask video . A monitoring method for causing an information processing device to perform a process of generating and outputting a mask video in which a person's image area detected from a moving image captured in a monitoring area is changed to a mask image ,
A step of having the step, and the step of determining the attribute of each detected person from the moving image to obtain the area information of each person by detecting a person from the video image,
Obtaining article change information relating to a change situation of an article in a background image area in the video image ;
Setting the mask image for each attribute;
A contour line for each person is generated based on the region information and the attribute for each person, the inside of the contour line is changed to the mask image for each attribute, and in the background image region based on the article change information. Generating and outputting the mask image displaying the change state of the article in an identifiable manner , and
A monitoring method characterized in that it is possible to grasp a behavior tendency of a person and a change state of the article by reproducing the mask video .

Images