JP2021072521A - Imaging device, imaging system, control method of imaging device, program, and recording medium - Google Patents

Abstract

To safely update a predetermined function in an imaging device that can communicate with a relay server capable of image analysis and improve the accuracy of the image analysis in the relay server.SOLUTION: An imaging device communicates with a relay server capable of image analysis via a network. The imaging device includes transmission means for transmitting an image to the relay server and first image processing means for performing predetermined image processing for a captured image. The transmission means transmits metadata on an image after application of image processing together with an image before the application of image processing by the first image processing means.SELECTED DRAWING: Figure 6

Description

The present invention relates to an imaging device, an imaging system, a control method of the imaging device, a program, and a storage medium.

With the development of communication systems, in addition to high-speed communication and large capacity, multiple connections and low latency (real time) are expected. For example, the 5th generation mobile communication system is characterized in that it provides an environment in which computing resources such as Multi-access Edge Computing (hereinafter referred to as MEC) can be used in a base station closer to the device side. Furthermore, it is said that it also has features such as a low latency of about 1 msec. On the other hand, in conventional surveillance cameras, a tracking technique for tracking an object being photographed is widely used. As for the tracking function of the surveillance camera, there is a demand to continuously use a new one because the function may be improved. However, there is also a request to avoid changing the firmware of the surveillance camera if possible because there is a concern that the device will not start.

Therefore, if the tracking function is realized by using the calculation resource of MEC, the function can be updated safely without changing the firmware. However, when tracking is performed by MEC, since tracking is performed on the delivered video, there is a problem that the target cannot be hidden and tracked by the mask if the mask is set. Patent Document 1 discloses a patent for synthesizing an image so as to avoid a face portion when synthesizing an image with the video, as a technique for switching the video processing under some conditions when processing the video.

Japanese Unexamined Patent Publication No. 2012-199743

However, if the technique disclosed in the above-mentioned patent document is applied to the tracking function of a surveillance camera, when tracking a plurality of human bodies, the human body to be tracked may be masked, which may make tracking difficult.

Therefore, an object of the present invention is to safely update a predetermined function in an imaging device capable of communicating with a relay server capable of image analysis, and to improve the accuracy of image analysis in the relay server.

In order to solve the above problems, the image pickup device of the present invention is an image pickup device that communicates with a relay server capable of image analysis via a network, and is used as a transmission means for transmitting an image to the relay server and a captured image. On the other hand, it has a first image processing means that performs predetermined image processing, and the transmitting means includes an image before the application of the image processing by the first image processing means and a post-application of the image processing. It is characterized by transmitting image metadata.

According to the present invention, in an imaging device capable of communicating with a relay server capable of image analysis, a predetermined function can be safely updated, and the accuracy of image analysis in the relay server can be improved.

It is a block diagram of the monitoring system which concerns on this embodiment. It is a hardware block diagram of the surveillance camera which concerns on this embodiment. It is a software configuration diagram of a surveillance camera. It is a software configuration diagram of a reverse proxy. It is a flowchart when the reverse proxy which concerns on Example 1 receives a request. It is a flowchart at the time of image generation in the surveillance camera which concerns on Example 1. It is a flowchart at the time of video reception in the reverse proxy which concerns on Example 1. FIG. It is a flowchart at the time of image generation in the surveillance camera which concerns on Example 2.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a configuration diagram of the monitoring system 1000 according to the present embodiment. The surveillance system 1000 includes a surveillance camera 100 and a reverse proxy 500. The surveillance camera 100 is connected to the reverse proxy 500 via the 5G network 400, and the reverse proxy 500 is connected to the client device 200 via the network 300. Here, the 5G network 400 is a wireless communication system represented by a 5th generation mobile communication system. Here, as an example, the 5G network 400 is used for the connection between the surveillance camera 100 and the reverse proxy 500, but the network is not limited to this as long as it is a network to which MEC can be applied.

The surveillance camera 100 includes a camera unit (imaging unit), and distributes an image (captured image) captured by the imaging unit via a network. The surveillance camera 100 is an imaging device capable of PTZ control for changing the positions of Panning, Til, and Zoom (PTZ). Further, the surveillance camera 100 has, for example, a mask function of hiding a predetermined area of a captured image with an image of a predetermined shape and size (hereinafter referred to as a mask) called a privacy mask. In addition, the surveillance camera 100 may perform video analysis such as motion detection (object recognition), face detection, face recognition, and person detection. It should be noted that, for example, when the surveillance camera 100 is installed, the surveillance camera 100 can communicate with the client device 200 without going through the reverse proxy 500, for example, in order to make settings such as administrator registration in a secure network. In this case, it is also possible to separately change the route to the network 300 such as Ethernet and communicate. That is, the surveillance camera 100 communicates with at least one of the reverse proxy 500 and the client device 200 via the network.

The reverse proxy 500 is installed in the MEC, which is an edge computing device installed in or near the base station of the 5G network 400. The MEC may be virtualized by a technique such as OpenStack. Regardless of the virtualization method, a configuration without virtualization can also be adopted. The reverse proxy 500 functions as a relay server that relays the request from the client device 200 to the surveillance camera 100. Further, the reverse proxy 500 receives the response from the surveillance camera 100 and performs content caching, video processing, and video analysis processing as needed. Further, the reverse proxy 500 also performs automatic tracking that tracks an object from PTZ control and video analysis results. When the surveillance camera 100 is connected to the 5G network 400, the reverse proxy 500 centrally manages the requests from the client device 200. As a result, the reverse proxy 500 and the surveillance camera 100 can be combined and shown to the client device 200 as a surveillance camera.

The client device 200 is a device typified by a personal computer, a smartphone, or the like, and as long as it communicates with the surveillance camera 100 and provides the user with a Graphical User Interface (GUI), the form thereof is not particularly limited. The client device 200 accesses the surveillance camera 100 or the reverse proxy 500 and acquires an image. Then, the user can change and control the setting of the surveillance camera 100 by using the client device 200.

In the present embodiment, the number of surveillance cameras is one for the sake of simplification of the explanation, but two or more cameras may be used. In addition to the client device 200, there may be other client devices that access the surveillance camera 100 to receive and store images.

The network 300 is a network that can be connected to a wide area such as the Internet. In this embodiment, the communication standard, scale, and configuration are not limited as long as communication between each surveillance camera and client can be performed without any trouble. Therefore, the network 300 can be applied from Ethernet to LAN (Local Area Network) and Wi-Fi.

FIG. 2 is a hardware configuration diagram of the surveillance camera 100 according to the present embodiment. The surveillance camera 100 includes a CPU 1101, a primary storage device 1102, a secondary storage device 1103, a video capture interface (hereinafter referred to as I / F) 1105, a network I / F1107, a camera control I / F1108, and a video analysis I / F1109. including. Also, they are connected to each other via the internal bus 1104.

Here, the primary storage device 1102 is a writable high-speed storage device typified by RAM, and is loaded with an OS, various programs, and various data, and is also used as a work area for the OS and various programs.

The secondary storage device 1103 is a non-volatile storage device typified by a flash memory, an HDD, an SD card, etc., and is used as a permanent storage area for an OS, various programs, and various data, as well as a short-term storage device. It is also used as a storage area for various data. Details of various programs and the like placed in the primary storage device 1102 and the secondary storage device 1103 of the surveillance camera 100 will be described later.

An image sensor such as a CCD or CMOS and a camera unit 1106 (imaging unit) including a lens are connected to the image capture I / F 1105. The video capture I / F 1105 converts and compresses the video data (captured image) acquired from the camera unit 1106 into a predetermined format and transfers it to the primary storage device 1102. It is also the role of the video capture I / F1105 to superimpose images such as masks.

The camera unit 1106 is also connected to the camera control I / F 1108 to control imaging parameters such as exposure and control Pan, Tilt, Zoom, etc., and the current parameters acquired from the camera unit 1106 are stored in the primary storage device 1102. save.

The image analysis I / F1109 performs image analysis processing such as motion detection, human body detection, face detection, and face recognition on the image on which the image such as the mask generated by the image capture I / F1105 is not superimposed, and the result is 1 It is stored in the next storage device 1102. In order to track a specific person from the result of the video analysis processing, the camera may be controlled via the camera control I / F1108.

The network I / F 1107 is an I / F for connecting to the above-mentioned network 300 or 5G network 400, and is responsible for communication with the client device 200 or the reverse proxy 500 via a communication medium such as a 5G network or Ethernet.

(Example 1)
Hereinafter, the tracking method of the monitoring system according to the first embodiment of the present invention will be described. FIG. 3 is a software configuration diagram of the surveillance camera 100. The OS 1000, the protocol processing unit 1001, the camera control unit 1002, the video generation unit 1003, the setting management unit 1004, and the video analysis unit 1005 are loaded into the primary storage device 1102.

OS1000 is a basic program that controls the entire surveillance camera 100. Typical functions of the OS1000 include management of a secondary storage device 1103 such as a file system, memory management such as virtual storage, process management such as multitasking, and management of network I / F1107.

The protocol processing unit 1001 distributes video data generated by the video generation unit 1003, acquires control information included in the request, and sets values included in the request in response to a request (request) from the client device 200 or the reverse proxy 500. And so on. The protocol processing unit 1001 relays the request to the video generation unit 1003, the camera control unit 1002, the setting management unit 1004, and the video analysis unit 1005. Then, based on the result from the relay destination, the corresponding response is transmitted to the client device 200 or the reverse proxy 500. The protocol processing unit 1001 communicates with an external device (here, the client device 200 or the reverse proxy 500) via the network I / F 1107. As a network, both 5G network 400 and network 300 can be applied. As a protocol, for example, Hypertext Transfer Protocol (HTTP) is used. The protocol processing unit 1001 also has an aspect such as an HTTP server, that is, a Web server, in order to return a response in response to a request from the client device 200. It does not depend on the version of HTTP, and can be applied such as HTTP / 1.0, HTTP / 1.1, HTTP / 2, HTTP / 3, etc., and may be applied via the Transport Layer Security (TLS) layer.

The camera control unit 1002 controls and adjusts the camera function based on the control information received from the protocol processing unit 1001 and the video analysis unit 1005. Typical examples are the control of the camera unit 1106 and the notification of the control result. The control may include imaging parameters such as Panning, Til, Zoom PTZ parameters and exposure. Since the protocol processing unit 1001 and the video analysis unit 1005 provide control information to the camera control unit 1002, the camera control unit 1002 also performs the adjustment function of the protocol processing unit 1001 and the video analysis unit 1005.

The video generation unit 1003 acquires the video data acquired by the camera unit 1106 via the video capture I / F 1105, encodes the acquired video data, and generates a video data file. That is, the image generation unit 1003 performs various image processing on the captured image. The encoding method may be jpg, and H.I. 264 / AVC, H. Any video compression coding method such as 265 / HEVC can be applied. In addition to the encoding method of this embodiment, another video compression method may be used. In addition, the video generation unit 1003 may have a function of generating a cut-out video by cutting out a part of the entire video generated by the video capture I / F 1105. In addition, the image generation unit 1003 performs mask processing for hiding a predetermined area of the captured image, processing for specifying the Q value of the specified area, and processing for superimposing (displaying) the time and an arbitrary character string on the captured image. May be good. These settings are notified from the setting management unit 1004 as image processing information, specifically, for example, metadata. In addition, in order to follow the mask area or the area of the specified Q value even when the PTZ moves, the PTZ position notified from the camera control unit 1002 is referred to, the area to be processed is calculated, and the area to be processed is superimposed. I do. As the video processed by the video analysis unit 1005, those that have not been subjected to the superimposition processing are used. By doing so, it becomes possible to analyze the video without extra processing, and the load of the analysis processing is reduced.

The setting management unit 1004 writes the parameters relayed from the protocol processing unit 1001 as set values to the secondary storage device 1103, and notifies the camera control unit 1002, the image generation unit 1003, and the image analysis unit 1005 that refer to the set values. Do. The setting values include distribution video, video analysis, tracking, mask setting, and the like. The mask can be set as long as it indicates the area, and examples thereof include the position, width, and height of the start point.

The image analysis unit 1005 performs image analysis processing such as motion detection, human body detection, face detection, and face recognition on the image generated by the image generation unit 1003. As the setting value of the video analysis, the one notified from the setting management unit 1004 is used. The result of the video analysis is transmitted to the client device 200 or the reverse proxy 500 via the protocol processing unit 1001. In addition, the image analysis unit 1005 also tracks the PTZ to move to the position of the detected object based on the result of the image analysis. In PTZ control, the camera control unit 1002 is controlled. Although the video analysis unit 1005 has been described in this embodiment, when the video analysis is performed by the reverse proxy 500, the tracking is possible by the reverse proxy, so that the video analysis unit in the surveillance camera 100 is indispensable. is not it.

FIG. 4 is a software configuration diagram of the reverse proxy 500. The reverse proxy 500 includes a protocol processing unit 5001, a camera control unit 5002, a video processing unit 5003, a setting management unit 5004, and a video analysis unit 5005, and relays a request from the client device 200 to the surveillance camera 100. .. Since the HTTP request is received from the client device 200, the reverse proxy serves as an HTTP server as in the surveillance camera 100. Therefore, it has an aspect of a Web server. In addition, since it sends an HTTP request to the surveillance camera 100, it also has an HTTP client aspect, and has both server and client functions. A reverse proxy can also be said to be a Web server in a broad sense.

The protocol processing unit 5001 relays the request to the surveillance camera 100 in response to the request of the client device 200. The set value may be passed to the setting management unit 5004 at the time of relaying, and the request received from the client device 200 may be partially changed and transmitted. Then, when the response from the surveillance camera 100 is received, processing is performed in the reverse proxy 500 as necessary, and the response is returned to the client device 200. The processing in the reverse proxy 500 as needed includes caching of the video and HTML files included in the response, providing the video to the video processing unit 5003, and the video analysis unit 5005.

The camera control unit 5002 is responsible for converting the PTZ control command received from the video analysis unit 5005 into a format accepted by the protocol processing unit 5001. It is also responsible for holding camera information including the PTZ position notified by the protocol processing unit 5001.

The video processing unit 5003 receives the video distributed from the surveillance camera 100 as an input, and processes the video based on the set value managed by the setting management unit 5004 and the metadata received from the surveillance camera 100 via the protocol processing unit 5001 ( Image processing). Video processing includes decompression processing (decoding) of video, compression processing (encoding), mask processing that hides a predetermined area, processing that specifies the Q value of a specified area, and processing that superimposes a time or an arbitrary character string on the video. Etc. are included. The encoding method may be jpg, and H.I. 264 / AVC, H. Any video compression coding method such as 265 / HEVC can be applied. In addition to the encoding method of this embodiment, another video compression method may be used.

The setting management unit 5004 receives parameters from the protocol processing unit 5001 and saves (stores) them as setting values. The saved set value is notified to the video processing unit 5003 and the video analysis unit 5005. The set values include video analysis, tracking, mask setting, and the like.

The image analysis unit 5005 performs image analysis processing such as motion detection processing (object recognition processing), human body detection processing, face detection processing, and face recognition processing on the image processed by the image processing unit 5003. As the setting value of the video analysis, the one notified from the setting management unit 5004 is used. The result of the video analysis is transmitted to the client device 200 via the protocol processing unit 1001. In addition, the image analysis unit 5005 also tracks the PTZ to move to the position of the detected object based on the result of the image analysis. For PTZ control, the camera control unit 1002 is controlled.

FIG. 5 is a flowchart when the reverse proxy 500 according to the first embodiment receives a request. S6000 starts when a request is received from the client device 200. In S6001, the protocol processing unit 5001 performs request analysis processing. Then, in S6002, branching is performed according to the type of request. When the request type is saving the mask setting value or saving the tracking setting, the process branches to S6003 and the setting value is saved in the setting management unit 5004. If the request type is other, branch to S6004 without saving the set value. After saving the set value of S6003, the header addition process of S6004 is performed. In the header addition process of S6004, the X-forwarded-for header, which is a header for indicating that the relay has been relayed from the reverse proxy 500, is added to the HTTP request header. In other words, it can be said that this process is a process of adding relay information. By referring to this X-forwarded-for by the surveillance camera 100, it becomes possible for the surveillance camera 100 to determine that the request is relayed by the reverse proxy 500. In this embodiment, it is clarified that relaying is performed by using the X-forwarded-for header. However, in addition to this, the setting value such as relaying may be transmitted, or the setting value such as performing image analysis outside the surveillance camera 100 or the HTTP header may be sent to the surveillance camera 100.

Next, in S6005, the protocol processing unit 5001 transmits an HTTP request to the surveillance camera 100, and relays the request from the client device 200. By saving the setting value in the reverse proxy 500 according to the type of request in S6003 at the time of relay, the same setting is made in both the surveillance camera 100 and the reverse proxy 500. Therefore, even when the reverse proxy 500 is relayed, the route between the surveillance camera 100 and the client device 200 changes, and even when the reverse proxy 500 is not used (does not go through), it is dynamic while maintaining both tracking processing and mask processing. Switching can be realized.

The flowchart finally ends at S6006. This is useful, for example, when the surveillance camera 100 is separated from the 5G network 400 for temporary maintenance and maintenance is performed on a local network, or when the surveillance camera 100 is installed on a moving body such as a car or a railroad. become.

FIG. 6 is a flowchart at the time of image generation in the surveillance camera 100 according to the first embodiment. S1500 is activated when the image generation unit 1003 acquires the image data before image compression. The request relayed by the reverse proxy 500 is processed by the protocol processing unit 1001 of the surveillance camera 100. When the acquired request includes the X-forwarded-for header in the protocol processing unit 1001, the information is transmitted to the video generation unit 1003 or the setting management unit 1004. That is, the protocol processing unit 1001 transmits information (relay information) that the acquired request is relayed by the reverse proxy 500 to the video generation unit 1003 or the setting management unit 1004. The video generation unit 1003 that has received the relay information refers to the value and determines whether or not the relay has been relayed by the reverse proxy 500 in S1501. That is, the video generation unit 1003 determines whether or not to communicate with the reverse proxy 500 using the relay information. If it is relayed by the reverse proxy 500 in S1501 (YES), it branches to S1505 and the video generation unit 1003 performs the encoding process. On the other hand, if it is not relayed (NO) in S1501, it branches to S1502, and the image generation unit 1003 determines whether or not the mask is set.

If the mask is not set in S1502 (NO), the process branches to S1505 without setting the mask. On the other hand, when the mask is set in S1502 (YES), the process branches to S1503, and the image generation unit 1003 performs mask processing. In the mask processing, in S1503, the image generation unit 1003 calculates the area coordinates on which the mask is superimposed from the current value of the PTZ notified from the camera control unit 1002 and the area information of the mask notified from the setting management unit 5004. Next, in S1504, the image generation unit 1003 superimposes the mask on the image and performs the encoding process of S1505. After the encoding process of S1505, the video generation notification is sent to the protocol processing unit 1001 in S1506. When the request is relayed by the reverse proxy 500, the protocol processing unit 1001 that has received the notification delivers the video to the reverse proxy 500. That is, the protocol processing unit 1001 transmits the metadata of the image after the application of the image processing to the reverse proxy 500 together with the image before the application of the image processing by the image generation unit 1003. On the other hand, if the request is not relayed by the reverse proxy 500, the video is delivered to the client device 200. In this embodiment, it is clarified that relaying is performed by using the X-forwarded-for header. However, in addition to this, information indicating relaying such as setting values such as relaying, setting values such as performing image analysis outside the surveillance camera 100, and video analysis using a relay server such as an HTTP header can be performed. You may use the information shown. In this case, the protocol processing unit 1001 transmits information indicating that the relay is to be performed or information indicating that the video analysis is performed by the relay server to the video generation unit 1003. Finally, the process ends in S1507.

By avoiding mask superposition in the surveillance camera 100 when relayed by the reverse proxy 500, it is possible to transmit a captured image in which the mask is not superposed on the reverse proxy 500. Then, the reverse proxy 500 can perform analysis processing on the video that has not been subjected to extra video processing.

Note that the video analysis unit 1005 may also switch whether to perform the tracking process depending on whether or not it has been relayed by the reverse proxy, as in the video generation unit 1003. As described in FIG. 6, when the image is relayed by the reverse proxy 500, the image without the mask superimposed can be sent to the reverse proxy 500 by not performing the mask superimposition on the surveillance camera 100, and the reverse proxy 500 can send the image. Image analysis processing becomes possible.

FIG. 7 is a flowchart at the time of receiving a video in the reverse proxy 500 according to the first embodiment. The S7000 is activated when the reverse proxy 500 receives an image from the surveillance camera 100. First, in S7001, the protocol processing unit 5001 receives an image from the surveillance camera 100. Next, the protocol processing unit 5001 relays the received video to the video analysis unit 5005, and the video analysis unit 5005 determines in S7002 whether or not the tracking setting is set. If the tracking setting is not set (NO), the video processing unit 5003 determines whether or not the mask is set without branching to S7004 and performing tracking. On the other hand, when the tracking setting is set (YES), the process branches to S7003, and the image analysis unit 5005 performs tracking. In the tracking process in S7003, the image analysis unit 5005 performs image analysis and detects the position of the object. Next, in order to perform PTZ to the detected position, the PTZ position is transmitted to the camera control unit 5002, and the camera control unit 5002 transmits the PTZ position to the protocol processing unit 5001. The protocol processing unit 5001 performs PTZ control on the surveillance camera 100. That is, the tracking process in S7003 performs PTZ control according to the image analysis result. In order to control the surveillance camera 100 according to the movement of the object (subject), the network connecting the surveillance camera 100 and the reverse proxy 500 needs to have a low latency like a 5G network.

In S7004, the video processing unit 5003 refers to the set value notified from the setting management unit 5004, and determines whether or not the mask is set. If the mask is not set in S7004 (NO), the mask is not superposed and branches to S7009, and the protocol processing unit 5001 transmits a video to the client device 200 and ends the processing. On the other hand, when the mask is set (YES), the process branches to S7005 and the decoding process is performed. In S7005, the video processing unit 5003 decodes the received video. After the decoding process, in S7006, the video processing unit 5003 acquires the mask coordinates. In the first embodiment, the mask coordinates are acquired by calculating the mask coordinates from the position of the PTZ received from the camera control unit 5002 and the mask setting notified from the setting management unit 5004. After that, in S7007, the image processing unit 5003 performs mask superimposition on the image decoded from the calculated mask coordinates. In S7008, the video processing unit 5003 performs encoding processing on the video on which the mask is superimposed. After that, in S7009, the protocol processing unit 5001 transmits the video to the client device 200, and finally ends in S7010.

In this embodiment, since the encoded video is distributed by the surveillance camera 100, the reverse proxy 500 has been described to perform re-encoding processing including decoding and encoding. However, when the surveillance camera 100 delivers without encoding, the re-encoding process is not necessary.

As described above, the reverse proxy 500 and the surveillance camera 100 negotiate the mask processing, so that the masks can be superimposed and the functions can be updated safely. An imaging system capable of tracking a plurality of human bodies. Can be provided.

(Example 2)
Hereinafter, the tracking method of the monitoring system according to the second embodiment of the present invention will be described. In the first embodiment, the flowchart of the image generation unit 1003 of the surveillance camera 100 has been described with reference to FIG. 6, but in the second embodiment, FIG. 8 will be used instead of FIG. Since the description of the remaining figures is the same as that of the first embodiment, the description thereof will be omitted.

FIG. 8 is a flowchart at the time of image generation in the surveillance camera 100 according to the second embodiment. The S8000 is activated when the video generation unit 1003 acquires the video data before image compression. The request relayed by the reverse proxy 500 is processed by the protocol processing unit 1001 of the surveillance camera 100. When the protocol processing unit 1001 includes the X-forwarded-for header, the information is transmitted to the video generation unit 1003 or the setting management unit 1004. That is, the protocol processing unit 1001 transmits information (relay information) that the acquired request is relayed by the reverse proxy 500 to the video generation unit 1003 or the setting management unit 1004. The video generation unit 1003 that has received the relay information refers to the value and determines whether or not the relay has been relayed by the reverse proxy 500 in S8001. When relayed by the reverse proxy 500 in S8001 (NO), it branches to S8006, and the image generation unit 1003 determines whether or not the mask is set.

On the other hand, if it is not relayed by the reverse proxy 500 in S8001 (NO), it branches to S8002 and the video generation unit 1003 determines whether or not the mask is set. If the mask is not set in S8002 (NO), the process branches to S8009 without setting the mask. On the other hand, in S8002, when the mask is set (YES), the process branches to S8003, and the image generation unit 1003 performs mask processing. In the mask processing, in S8003, the area coordinates on which the mask is superimposed are calculated from the current value of PTZ notified from the camera control unit 1002 and the area information of the mask notified from the setting management unit 1004. Next, in S8004, the image generation unit 1003 superimposes the mask on the image. Next, in S8005, the video generation unit 1003 notifies the protocol processing unit 1001 of the time information and the video frame number in association with each other in order to make the calculated mask coordinates correspond to the corresponding image. While the PTZ is moving, the image corresponding to the position of the PTZ that can be acquired may deviate. Therefore, it is essential to associate the mask coordinates with the image. After that, the encoding process is performed in S8009.

Next, the case where the relay is relayed by the reverse proxy 500 in S8001 (NO) will be described. In S8006, the image generation unit 1003 determines whether or not the mask is set. When the mask is set (YES), in S8007, the video generation unit 1003 calculates the mask coordinates as the information of the predetermined area for distribution to the reverse proxy 500. The method of calculating the mask coordinates is the same as that of S8003. Similarly, the mask coordinate notification in S8008 is the same process as in S8005. When the mask is not set in S8006 (NO), the calculation of the mask coordinates is not necessary, so the calculation of the mask coordinates is omitted and the process proceeds to the encoding process of S8009.

By avoiding mask superposition in the surveillance camera 100 when relayed by the reverse proxy 500, it is possible to transmit an image in which the mask is not superposed on the reverse proxy 500. Then, the reverse proxy 500 can perform analysis processing on the video that is not subjected to extra video processing.

In S8009, the video generation unit 1003 performs video encoding processing. After the encoding process of S8009, the video generation notification is sent to the protocol processing unit 1001 in S8010. Upon receiving the notification, the protocol processing unit 1001 delivers the video and the mask coordinates to the reverse proxy 500 when the request is relayed by the reverse proxy 500. On the other hand, if the request is not relayed by the reverse proxy 500, the video is delivered to the client device 200. In addition, as the information of the predetermined area, in order to associate the mask coordinates with the image, the time information and the frame number are associated with the mask coordinates and notified, but instead, the mask coordinates are inserted as metadata in the image instead. You may. Upon receiving the notification, the protocol processing unit 1001 distributes the video to the reverse proxy 500. In this embodiment, it is clarified that relaying is performed by using the X-forwarded-for header. However, in addition to this, the protocol processing unit 1001 may notify the setting value such as relaying, the setting value such as performing image analysis outside the surveillance camera 100, or the HTTP header. Finally, it ends in S8011.
Similarly to the video generation unit 1003, the video analysis unit 1005 may switch whether to perform the tracking process depending on whether or not the tracking process is relayed by the reverse proxy 500.

The video delivered to the reverse proxy 500 is relayed to the client device 200 by the reverse proxy 500 in the same manner as in FIG. 7 of the first embodiment. The processing at the time of relay, that is, the processing at the time of receiving video in the reverse proxy 500 is the same except for the acquisition of the mask coordinates of S7006. Therefore, the processing of S7006 different from that of the first embodiment will be described. In the acquisition of the mask coordinates of S7006, the video processing unit 5003 uses the mask coordinates transmitted from the surveillance camera 100 via the protocol processing unit 5001 and the camera control 5002. As a means for realizing this, for example, it may be acquired from the current position of the PTZ included in the video, or it may be obtained from the time information or the link information between the video frame number and the image.

(Other Examples)
Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to the above-described embodiment, and various modifications can be made based on the gist of the present invention. It is not excluded from the scope of the invention. For example, a Web server may be used instead of the surveillance camera, a mask area for hiding a predetermined area as metadata, a designated area for specifying a Q value, and an area for superimposing a time or an arbitrary character string on an image may be used.

Further, a computer program that realizes a part or all of the control in the present embodiment to realize the functions of the above-described embodiment may be supplied to the image pickup apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the image pickup apparatus may read and execute the program. In that case, the program and the storage medium that stores the program constitute the present invention.

100 Surveillance camera 200 Client device 500 Reverse proxy 1000 Surveillance system 1001 Protocol processing unit 1002 Camera control unit 1003 Video generation unit 1004 Setting management unit 1005 Video analysis unit 5001 Protocol processing unit 5002 Camera control unit 5003 Video processing unit 5004 Setting management unit 5005 Video Analysis department

Claims (11)

An imaging device that communicates with a relay server capable of image analysis via a network.
A transmission means for transmitting an image to the relay server and
It has a first image processing means for performing predetermined image processing on a captured image, and has
The transmitting means is an image pickup apparatus that transmits metadata of an image after the application of the image processing together with an image before the application of the image processing by the first image processing means. The metadata is characterized by including at least one of a mask setting for hiding a predetermined area of the captured image, a Q value designation of the designated area on the captured image, and a character string display setting. The imaging device according to claim 1. An imaging system including the imaging apparatus according to claim 1 or 2, and a relay server capable of communicating with the imaging apparatus via a network and performing image analysis. The imaging system according to claim 3, wherein the relay server is a relay server for MEC (Multi-access Edge Computing). The relay server has an analysis means that performs at least one of the image analysis processing of motion detection processing, face detection processing, object recognition processing, and tracking processing on the captured image delivered from the imaging device. 3. The imaging system according to claim 3 or 4. The relay server
A relay means for relaying a request from the client device to the image pickup device, and
It is provided with a management means for managing the setting value included in the request.
The imaging system according to any one of claims 3 to 5, wherein the management means stores the set value according to the type of the request. 6. The management means according to claim 6, wherein the management means stores the set value when at least one of the set value of the predetermined region of the captured image and the set value of the tracking process is included in the request. Imaging system. The relay server has the predetermined region of the captured image delivered from the imaging device based on the information of the predetermined region generated by the generation means for generating the information of the predetermined region of the captured image of the imaging device. The imaging system according to any one of claims 3 to 7, further comprising a second image processing means for superimposing an image on the image. It is a control method of an image pickup device that communicates with a relay server capable of image analysis via a network.
The transmission process of transmitting an image to the relay server and
Including a first image processing step of performing predetermined image processing on a captured image,
The transmission step is a control method comprising transmitting metadata of an image after application of the image processing together with an image before application of the image processing in the first image processing step. A program for causing a computer to execute the control method of the imaging device according to claim 9. A computer-readable storage medium for storing the program of claim 10.

Images