JP2023122673A - Image processing system, image processing method, and computer program - Google Patents

Abstract

To achieve an image processing system that can reduce a burden on a user who performs monitoring.SOLUTION: An image processing system comprises: image acquisition means that acquires an image; display control means that displays the image; object designation means that designates an object in the image; selection means that selects some notification means from a plurality of notification means; control means that displays a mark indicating the position of the object superimposed on the object in the image, and causes some of the notification means to perform notification; and storage means that, in association with performing the notification, stores the image, information on the mark or the object, and information on some of the notification means.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing system, an image processing method, and a computer program.

In recent years, shoplifting damage in various facilities such as retail stores and large-scale commercial stores has become serious, and there is a need on the store side to reduce the damage caused by shoplifting. In response to this need, technology to prevent shoplifting by analyzing video captured by a surveillance camera in real time, detecting suspicious behavior suspected of shoplifting, and automatically issuing voice alerts instead of store clerks and security guards. There is

Techniques for detecting suspicious behavior include, for example, the method described in Patent Document 1. In Patent Document 1, feature values such as the position, orientation, and line-of-sight direction of a person are extracted from image data obtained from video captured by a camera, and those feature values are analyzed to detect a person suspected of shoplifting. When a person who is suspected of shoplifting is detected, a store clerk or security guard is notified, and the shopkeeper or security guard calls out to the target person to suppress shoplifting.

Further, Patent Literature 2 discloses a method of determining a notification device from among a plurality of notification devices according to the position of a detected suspicious person in an image.

Further, Patent Document 3 discloses an example of notifying a store clerk when a person suspected of shoplifting is detected. In this example, detection information is notified to a mobile terminal held by a store clerk. In addition, when the clerk who received the notification on the mobile terminal completes the response to the suspicious person, the response completion information is transmitted to the management server.

JP 2017-076171 A Japanese Patent Application Laid-Open No. 2020-140520 JP 2015-187869 A

As in Patent Documents 1 to 3, when a person suspected of shoplifting is detected, it is common to notify the store clerk of the information. In order to prevent shoplifting, the shop clerk who has received the notification is required to approach the target suspicious person and see how he is doing, or to call out to him directly. On the other hand, it is desirable that the time and burden required to deal with the suspicious person should be as small as possible, because the store clerk is originally required to perform other tasks such as stocking products and dealing with the cash register.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing system capable of reducing the burden on a monitoring user.

The image processing system of the present invention is
an image acquisition means for acquiring an image;
display control means for displaying the image;
an object designation means for designating an object in the image;
A selection means for selecting a part of notification means from among a plurality of notification means;
a control means for superimposing and displaying a mark indicating the position of the object on the image of the object, and for performing notification by the part of the notification means;
It is characterized by comprising storage means for storing the image, the information of the mark or the object, and the information of the part of the notification means in accordance with the notification.

According to the present invention, it is possible to realize an image processing system that can reduce the burden on a monitoring user.

2 is a block diagram showing a hardware configuration example of an image processing apparatus according to Embodiment 1; FIG. 1 is a functional block diagram of an image processing system according to Example 1; FIG. 1 is a schematic diagram showing a layout example of a store monitored using the image processing apparatus 100 of Embodiment 1. FIG. 4A and 4B are diagrams showing examples of screens displayed on the display device 107 of the image processing apparatus 100 according to the first embodiment; FIG. 4 is a flow chart showing the flow of processing for determining an informing device by the image processing device in Embodiment 1. FIG. It is a figure which shows the example which selects a target person by user operation. FIG. 10 is a diagram showing an example of determining the size of a frame to be superimposed on image data based on setting of human body size and user's input coordinates; 8 is a diagram showing an example of saving an image 800 in which a frame 702 is superimposed on image data in Embodiment 1. FIG. It is a figure which shows the example of a setting of the alerting|reporting apparatus in an image. FIG. 10 is a diagram for explaining an example of image data saved in Example 2; FIG. FIG. 12 is a diagram showing an example of selecting a sound source from a plurality of sound sources registered in the notification device in Example 3; FIG. 11 is a diagram for explaining an example of image data saved in Example 3;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described using examples with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

(Example 1)
In the first embodiment, a mark such as a frame for specifying a target person specified by the user is superimposed (added) on the image and recorded. In the following description of the embodiments, an example of a frame will be described as a mark for specifying a designated target person. Also good.

FIG. 1 is a block diagram illustrating a hardware configuration example of an image processing apparatus according to the first embodiment. The image processing apparatus 100 has a CPU 101 as a computer, a ROM 102 and a RAM 103 as storage media, an HDD 104, a communication unit 105, an input device 106, a display device 107, and the like.

A communication unit 105 is connected to a wired or wireless network 108 and is capable of two-way communication with an external server or the like of the image processing apparatus. The CPU 101 reads computer programs recorded in the ROM 102 and executes various processes. A RAM 103 is used as a temporary storage area such as a main memory or a work area. A HDD (Hard Disk Drive) 104 is used for storing image data.

Note that the CPU 101 can acquire image data from a video acquisition device such as a surveillance camera via the network 108 . Image data can also be acquired by downloading image data stored in an external server. Image data can also be obtained by reading image data stored in HDD 104 .

The display device 107 is a display device such as a liquid crystal display that displays image data and suspicious person detection results. The input device 106 includes devices such as a keyboard, a mouse, and a touch panel for the user to manually input selection of image data, setting of a sound source position on the image data, various instructions, and the like. Note that each piece of hardware shown in FIG. 1 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

As described above, the hardware configuration of the image processing apparatus 100 has the same hardware components as those installed in a personal computer (PC). Therefore, various functions realized by the image processing apparatus 100 can be implemented as software operating on a PC. The image processing apparatus 100 can realize various functions of the image processing apparatus 100 as shown in FIG. 2 and processing shown in a flowchart of FIG. 8 described later by the CPU 101 executing a program.

FIG. 2 is a functional block diagram of the image processing system according to the first embodiment. Some of the functional blocks shown in FIG. 2 are realized by causing the CPU 101 as a computer included in the image processing apparatus 100 to execute a computer program stored in a memory such as the ROM 102 as a storage medium. there is However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

Also, when implementing the respective functional blocks shown in FIG. 2 by hardware, the respective hardware need not be built in the same housing, but separate devices connected to each other via signal paths. It may be configured as

The image processing apparatus 100 has an image acquisition unit 202, a display unit 203, a setting unit 204, an operation input unit 205, a user input reception unit 206, a notification instruction unit 207, a storage unit 208, and the like. The image processing apparatus 100 is also connected to a video acquisition device 201 such as a surveillance camera and a notification device 209 via a network 108 such as the Internet. Note that the number of video acquisition devices 201 and the number of notification devices 209 are not limited to the example in FIG.

Next, each part will be explained. The image acquisition unit 202 functions as an image acquisition unit that acquires an image, and executes an image acquisition step for acquiring image data obtained from the video sent from the video acquisition device 201 . Here, the video may be a video file in an image format such as an AVI file or MOV file, or may be a real-time video sent from a web camera. Also, the number of types may be one or plural. The video acquisition device 201 includes a plurality of surveillance cameras 303, 304, 305, etc., as shown in FIG. 3, for example.

FIG. 3 is a schematic diagram showing a layout example of a store monitored using the image processing apparatus 100 of the first embodiment. A store 300 has a terminal 301 installed in the backyard and a salesclerk 302 working in the backyard. A plurality of surveillance cameras 303, 304, 305 as image acquisition means, a plurality of notification devices 402-404, and product shelves 405-409 are arranged. Here, the notification device functions as notification means and includes, for example, a speaker. Also, the backyard may be provided at a location away from the store. It is assumed that the terminal 301 includes the image processing apparatus 100 of the embodiment.

A setting unit 204 specifies the human body size on the image data displayed by the display unit 203 . As for the method of specifying, the coordinates may be specified by receiving numerical input from the user, or the coordinates may be specified by the user using a UI. Note that the display unit 203 functions as display control means for performing a display control step for displaying an image.

FIG. 4 is a diagram showing an example of a screen displayed on the display device 107 of the image processing apparatus 100 according to the first embodiment, showing an example of setting the human body size on the image data using the human body size icons 422 and 423. It is a diagram. In FIG. 4 , 400 denotes a screen displayed on the display device 107 of the image processing apparatus 100 and includes an image 401 acquired by the image acquisition unit 202 .

The user scales the human body size icon according to the size of the human body included in the image 401 to set the maximum and minimum values of the human body size in consideration of the depth information. The human body size icon 422 designates a size according to the smallest human body in the image data. The human body size icon 423 designates a size according to the largest human body in the image data.

The operation input unit 205 is a block for inputting user operations using the input device 106 . In this embodiment, the operation result of the operation input unit 205 is used in the setting unit 204 and the user input reception unit 206 .
A user input reception unit 206 receives that the user has selected a notification target on the image data using the operation input unit 205 . Further, frame data for specifying the person to be notified selected by the user is generated and superimposed on the image data displayed on the display unit 203 .

The notification instruction unit 207 issues a notification instruction to the notification device 209 based on the selection of the notification target by the user input reception unit 206 .
The storage unit 208 stores image data superimposed with a mark such as a frame output from the user input reception unit 206 . Note that superimposition in the embodiment is not limited to, for example, synthesizing a mark such as a frame with an image, but includes adding information such as coordinates of a mark such as a frame to an image file as metadata.

FIG. 5 is a flowchart showing the flow of processing for determining a notification device by the image processing device according to the first embodiment. The CPU 101 as a computer executes a computer program stored in a memory such as the ROM 102, for example, to perform the operation of each step in the flow chart of FIG.

In step S501, the user uses the input device 106 to enlarge or reduce the human body size icons 422 and 423 according to the person appearing on the image data, thereby specifying the human body size. Then, the setting unit 204 sets the maximum and minimum values of the human body appearing on the image data.

Next, in step S502, the user uses the input device 106 to select a person to be notified from the persons in the image. Here, step S502 functions as an object designating step (object designating means) for designating the object in the image.
FIG. 6 is a diagram showing an example of selecting a target person by user operation. In FIG. 6, an image 401 includes a person 603, and a user operation 602 indicates that the person 603 is selected by touching the touch panel with a finger. After that, step S503 and step S505 are executed in parallel according to the selection of the notification target.

In step S<b>503 , the size of the frame for specifying the person as the notification target is determined from the human body size set in the setting unit 204 .
FIG. 7 is a diagram showing an example of determining the size of the frame superimposed on the image data from the human body size set in step S501 and the user's input coordinates. In FIG. 7, an auxiliary line 701 represents processing for determining the size of the frame at the coordinates input by the user from the human body size icons 422 and 423 . That is, the size of the frame as the mark is changed according to the position of the object.

Based on the change in the human body size along the Y-axis (vertical axis of the screen 400 in FIG. 7) direction of the frame image 401 circumscribing the human body size icons 422 and 423, the Y-axis coordinate of the screen 400 touched by the user is calculated. Estimate the frame size. A frame 702 indicates a frame for specifying a person 603 to be notified, which is estimated from the human body size icons 422 and 423 based on the Y-axis coordinates touched by the user operation 602 . The method for determining the shape and size of the frame is not limited to the above method. For example, the image of the person 603 may be recognized and the size of the frame may be determined according to its size.

Then, in step S504, the storage unit 208 superimposes the frame 702 determined in step S503 on the image data and stores the image data. Here, step S504 functions as a storage step (storage means) for storing the image, the position information of the mark (frame) or the target object, and the information of a part of the notification device along with the notification. .
FIG. 8 is a diagram illustrating an example of saving an image 800 in which a frame 702 is superimposed on image data in the first embodiment.

On the other hand, in step S505, a notification instruction is given to the person selected in step S502 as a notification target using the notification device 209 located near the person. Here, steps S504, S505, etc. are control steps (control means) for superimposing and displaying a mark (frame) indicating the position of the object on the image of the object, and for performing notification by a part of the notification means. functioning as

In the flow of FIG. 5, step S505 is executed in parallel with steps S503 and S504, but step S505 may be executed after step S504, for example.

Next, in step S506, it is determined whether or not the system has ended. If Yes in step S506, the flow of FIG. 5 ends.
As described above, according to the first embodiment, the image processing apparatus 100 can store a person selected by the user by adding a frame for specifying the person, and can notify the person. It can be carried out.

In the first embodiment, a mark such as a frame is superimposed on the image and saved. However, instead of the frame, the coordinates of the frame or the coordinates of the selected person may be added to the image file as metadata and saved. good. As described above, superimposition in the present embodiment includes not only image synthesis but also adding and saving as metadata, for example.

(Example 2)
In the first embodiment, the image data 800 in FIG. 8 is stored with a frame for identifying the person to be notified superimposed. On the other hand, as shown in FIG. 3, a plurality of notification devices 209 may be installed in a store 300 . Therefore, in the second embodiment, when a person to be notified is selected by a user operation, a notification device for notifying the target person is selected from among a plurality of notification devices. Information related to the selected notification device is also superimposed on the image data and stored.

In the second embodiment, the setting unit 204 presets the coordinates of each notification device on the image data displayed by the display unit 203 . As for the setting method, the user may numerically input the coordinates of the notification device on the screen, or the icon of the notification device may be dragged and dropped to a predetermined position (installation position) on the screen using the UI on the screen. By doing so, the coordinates of the installation position of the notification device may be set.

FIG. 9 is a diagram showing an example of settings for notification devices in an image. A setting screen 400 includes an image 401 acquired by the image acquisition unit 202 and a notification device list 908 . Note that the image 401 is an enlarged display of a part of the layout of the store 300 .
In the notification device list 908, a plurality of notification devices 402 to 404 installed in the store 300 and registered in advance by the user are displayed as SP1 to SP3, respectively.

In the second embodiment, it is assumed that the notification device 402 is installed on the product shelf 406 and the notification device 403 is installed on the product shelf 407 included in the image data, and the notification device icons 4021 and 4031 are arranged at the corresponding positions on the image data. and set the coordinates. When setting the coordinates of a plurality of notification devices, tags such as SP1 and SP2 may be displayed next to each notification device icon.

In addition, in the second embodiment, in order to select at least one notification device that notifies a target person from among a plurality of notification devices, the coordinates of the person specified on the screen by the user and a plurality of Based on the coordinates of the notification device, the notification device with coordinates relatively close to the designated person is selected.

That is, based on the position (coordinates) of the person selected by the user and the positions (coordinates) of the pre-registered notification devices 402, 403, etc., the notification instruction unit 207 determines the coordinates of the selected person and the distance to each notification device. Calculate the distances respectively. Then, some notification devices closest or relatively close to the coordinates of the selected person are selected as notification devices for notifying the target person. Here, the notification instruction unit 207 functions as selection means for executing a selection step of selecting a part of the notification means from among the plurality of notification means.

Furthermore, it is determined whether or not there is a shield between the position of the notification device and the position of the object, and depending on the result of the determination, a part of the notification devices is selected from among the plurality of notification devices. Also good.
That is, the notification instructing unit 207 selects the coordinates of a plurality of notification devices set in advance by the setting unit 204, the coordinates of a person specified by a user operation, and the coordinates of shields existing between them. The distance between the notification device and the person may be calculated considering the influence of

Then, from among the notification devices 402, 403, etc., at least one notification device in which the distance between the notification device considering the influence of the shield and the designated person is relatively short may be selected as the notification device for notification. . Note that the coordinates of the shielding object may be registered in advance by the user or the like.

Specifically, for example, when a shield exists between a straight line connecting the coordinates of the designated person and the notification device, the distance information is multiplied by a coefficient corresponding to the thickness of the shield in the straight line direction. For example, if the thickness of the shield in the straight line connecting the two points is 0m, the coefficient for multiplying the distance between the two points is 1, and if it is 0.5m, the coefficient for multiplying the distance between the two points is 1.5. For example, when the thickness is 1 m, the coefficient is set to 3. The greater the thickness, the larger the coefficient.

Then, after weighting the coordinate distance between each notification device and the person by multiplying it by a coefficient according to the thickness of the obstructing object, the coordinate distances between multiple notification devices and the person are compared to determine which is relatively close. All that is necessary is to determine the notification device. That is, it is only necessary to select a part of the notification devices among the plurality of notification devices according to the thickness of the shield existing between the position of the notification device and the target position.

Although the distance between the notification device and the person on the two-dimensional map has been described above, it is desirable to calculate the distance between the notification device and the person on the three-dimensional map. In that case, the height of the person's ears is set to, for example, 1.5 m, and the height of the shield is also registered in the three-dimensional map. Then, the thickness of the shield on a straight line between the coordinates of the notification device and the coordinates of the person's ear height on the three-dimensional map is obtained, and the distance is multiplied by a coefficient corresponding to the thickness.

In addition, the shielding object in this embodiment refers to an object having a positional relationship such as blocking in the three-dimensional space between the notification device and the target object. For example, if the notification device is installed on the ceiling and positioned sufficiently higher than the product shelf, it may not block the distance between the object and the notification device. Alternatively, the same applies when the product shelf or the like is located at a position lower than the height of the ear of the object. In such a case, those product shelves do not become shields in the three-dimensional map, so it is desirable to calculate the distance, the thickness of the shields, etc. in the embodiment in the three-dimensional map. Note that the method of selecting the notification device is not limited to the above method.

In addition, a notification area having a predetermined area or more is set for each notification device, and depending on which notification area contains the coordinate information of the person specified by the user operation, notification linked to the notification area Notification may be made from the device. That is, based on the relationship between a plurality of areas respectively set based on the positions of the plurality of notification means and the position of the object, some of the notification devices corresponding to the area containing the position of the object may be selected. good.

FIG. 10 is a diagram for explaining an example of image data stored in the second embodiment. In addition to a frame 702 specifying a person to be notified, coordinate information of the notification device selected for notification is displayed. image data superimposed with the indicated notification device icon 4021 is saved.
That is, on the image 401, in addition to the frame 702, the notification device icon 4021 indicating the notification device selected for notification is superimposed and saved.

As described above, in the second embodiment, by superimposing and saving the information of the notification device selected for notification, it becomes easy to confirm the notification device that notified the target person, the content of the notification, and the like. The information of the notification device selected for notification may be ID (identification information) of the notification device, position information of the selected notification device, information on the sound source of the notification device, etc. instead of the icon as described above. can be

(Example 3)
In the third embodiment, for each notification device, a plurality of sound sources and the like are prepared for notifying a person to be notified. Then, as information on the frame of the person selected by the user operation (coordinates of the person or the coordinates of the person) and information on the notification device selected for notification, the content notified by the notification device is also added to the image data. to save.

In the third embodiment, the notification device 402 registers sound sources A, B, and C that can be used for notification, and the notification device 403 registers sound sources D, E, F, is registered and selectable.

FIG. 11 is a diagram showing an example of selecting a sound source from a plurality of sound sources registered in the notification device according to the third embodiment. In FIG. 11, a selection icon 1101 for selecting a sound source for notification is displayed in a state in which the notification device 402 for notifying a target person is selected by user operation 602 . The selection icon 1101 can be selected by flick input or the like if it is a touch panel. It should be noted that the method of displaying the sound sources that can be used by the notification device and the method of selecting the sound sources are not limited to this.

In FIG. 11, after selecting the notification device 402 for notification, a plurality of selection icons 1101 are displayed by touching the position of the person 603 in the screen, and a desired selection icon 1101 is touched. By releasing , the sound source is selected. However, for example, by touching the notification device icon 4021 corresponding to the notification device 402, a plurality of selection icons 1101 are displayed, and by releasing the finger on the desired selection icon 1101, the sound source is selected. can be

FIG. 12 is a diagram for explaining an example of image data to be saved in the third embodiment, and includes a frame 702 for specifying a person to be notified and a notification indicating coordinate information of the notification device selected for notification. In addition to the device icon 4021, an example is shown in which the content of the notified voice is superimposed.
That is, in addition to the frame 702 and the notification device icon 4021, the image 401 is superimposed with an icon 1201 indicating sound source information (notified content, etc.), and in the third embodiment, such an image 401 is saved. .

Note that the sound source information of the notification device in this embodiment is not limited to the type of sound (for example, male voice, female voice, pitch of sound, etc.). pattern), content of notification (type of message), sound volume, number of times of notification, and the like. That is, the information about the sound source includes at least one of the type of sound, pattern of sound, content of notification, volume, and number of times of notification. Also, the sound source information may be identification information relating to the types of the sounds, or may be the sound itself.

As described above, according to the third embodiment, by superimposing sound source information such as the content notified by the notification device on an image and storing it as information of the notification device, it is possible to notify what kind of notification was made to the target person. Confirmation of contents becomes easy.
As described above, superimposition is not limited to superimposition by image synthesis in the third embodiment and the like, and includes superimposition or addition to an image as metadata or the like.

Although one notification device is selected from a plurality of notification devices in the above embodiment, the number of notification devices is not limited to one. For example, two or more notification devices relatively close to the object may be selected from a plurality of notification devices. That is, it is sufficient to select a part of the notification devices among the plurality of notification devices. Also, when selecting a part of the notification devices out of a plurality of notification devices, it is not necessary to completely turn off the other notification devices, and it is only necessary to relatively lower the volume.

In the above embodiment, an example in which the user designates a suspicious person or the like as a target object on the screen has been described, but the target object may be, for example, an animal instead of a human being. That is, the configuration of the above embodiment may be applied to a system that monitors or guides the behavior of domestic animals, for example.

Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention.
It should be noted that a computer program that implements the functions of the above-described embodiments for part or all of the control in this embodiment may be supplied to an image processing apparatus or the like via a network or various storage media. A computer (or CPU, MPU, etc.) in the image processing apparatus or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100 image processing device 201 video acquisition device 204 setting unit 207 notification instruction unit

Claims (13)

an image acquisition means for acquiring an image;
display control means for displaying the image;
an object designation means for designating an object in the image;
A selection means for selecting a part of notification means from among a plurality of notification means;
a control means for superimposing and displaying a mark indicating the position of the object on the image of the object, and for performing notification by the part of the notification means;
An image processing system, comprising: storage means for storing the image, the information of the mark or the object, and the information of the part of the notification means in accordance with the notification. 2. The image processing system according to claim 1, wherein said selection means selects said part of notification means based on the positions of said plurality of notification means and the position of said object. 2. The selection means selects the some of the notification devices based on a relationship between a plurality of areas respectively set based on the positions of the plurality of notification means and the position of the object. The image processing system described in . 4. The image processing system according to claim 3, wherein said selection means selects said part of notification devices corresponding to said region including the position of said object. The selection means determines whether or not a shield exists between the position of the notification means and the position of the object, and selects one of the plurality of notification devices according to the determination result. 5. The image processing system according to any one of claims 1 to 4, wherein the notification device of the department is selected. The selection means selects a part of the plurality of notification devices according to the thickness of the shield existing between the position of the notification device and the target position. 6. The image processing system according to claim 5. The control means is
2. The image processing system according to claim 1, wherein the size of said mark is changed according to the position of said object. 8. The image processing system according to any one of claims 1 to 7, wherein the information of the mark or the object includes position information of the mark or the object. 9. The information of any one of claims 1 to 8, wherein the information of the part of the notification means includes the position information of the part of the notification means or the information on the sound source of the part of the notification means. image processing system. 10. The image processing system according to claim 9, wherein the information about the sound source includes at least one of a type of sound, a pattern of sound, content of notification, volume, and number of times of notification. 11. The image processing system according to claim 1, wherein said mark includes a frame. an image acquisition step of acquiring an image;
a display control step of displaying the image;
an object designation step of designating an object in the image;
A selection step of selecting a part of the notification devices from among the plurality of notification devices;
a control step of superimposing and displaying a mark indicating the position of the object on the image of the object, and causing the predetermined notification device to perform notification;
An image processing method, comprising: a storage step of storing the image, the information of the mark or the object, and the information of the part of the informing devices in association with the informing. A computer program for controlling each means of the image processing system according to any one of claims 1 to 11 by a computer.

Images