JP2019201344A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To provide an image processing apparatus capable of generating an image that makes it easy to monitor an object to be monitored, and an image processing method or the like.SOLUTION: An image processing apparatus 2 includes: an image acquisition unit 7 for acquiring an image; a face image extraction unit 9 for extracting a face image of a person 100 included in the acquired image acquired by the image acquisition unit 7; a monitoring image generator 10 for generating a monitoring image in which one or more face images extracted by the face image extraction unit 9 are arranged; and a determination and rearrangement unit 15 for arranging a face image of the same person in the same position included in each monitoring image generated using each acquired image, when an object image of the same target object is included in a plurality of acquired images.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for processing an image for monitoring.

  In recent years, surveillance cameras used in remote surveillance systems and the like have become widespread, but this surveillance camera reduces the amount of data of captured images (including video and moving images) to reduce the network load, and in a predetermined space. An image having a resolution capable of identifying an object to be monitored (hereinafter simply referred to as “object”) such as the face of the existing person 100 is required. For this reason, in many surveillance cameras, the captured image is reduced to an image with a resolution necessary for image recognition, and the reduced image is encoded using, for example, a codec such as H.264 or H.265. Send to image processing device. This image processing apparatus performs various processes for generating an image in which the amount of data is reduced and an object can be identified.

  Conventionally, in this image processing apparatus, the image information of the monitoring target region is separated from the image information of the monitoring target region that is important as the monitoring target and the image information of the non-monitoring target region that is not important as the monitoring target. There is known a technique for transmitting image information by lowering the bit rate of image information in a non-monitoring target area from the bit rate of the image (see, for example, Patent Document 1).

  Conventionally, the image processing apparatus includes a first acquisition unit that acquires an image from a camera via a network, and an analysis unit that performs an analysis process based on the image acquired by the first acquisition unit. According to the result of the analysis process by the analysis unit, a second acquisition unit that acquires an additional image from the camera via the network, and a reanalysis that performs the analysis process based on the image acquired by the second acquisition unit A technique including an analysis device including a means is known (for example, Patent Document 2).

  Further, conventionally, in an image processing apparatus, an object recognition unit recognizes an object from video data for a predetermined time generated by a camera, and a recognition result analysis unit analyzes the recognition result and receives it by a video reception unit. The minimum size of the face in the video recognized by the object recognition unit and the moving speed are obtained from the frame image, and the video processing control parameter determination unit determines the frame from the minimum size of the face and the moving speed obtained by the recognition result analysis unit. A technique for determining the lower limit value of the rate and resolution is known (for example, Patent Document 3).

JP 2013-70187 A JP 2017-212680 A JP 2010-263581 A

  Here, there are often a plurality of objects in the space where the surveillance camera captures images. For example, when the target is a person's face or the like, the positional relationship, size, direction, or the like of the target changes with time as the person moves. As a result, in the inventions described in Patent Documents 1 to 3, the time required for processing such as detection of each person's face position and extraction of each face image tends to change from frame to frame. . In that case, the face images are likely to be arranged in order from the image subjected to face recognition. For this reason, in the inventions described in Patent Documents 1 to 3, the image-processed image becomes easy to move on the image as time elapses, and monitoring of the object to be monitored is performed. There is a problem that it becomes difficult to do.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide an image processing apparatus, an image processing method, and the like that can generate an image that can easily monitor an object to be monitored.

  In order to solve such a problem, the invention according to claim 1 is an image processing apparatus, and includes an image acquisition unit that acquires an image, and an object included in the acquired image acquired by the image acquisition unit. Object image extracting means for extracting an object image, monitoring image generating means for generating a monitoring image in which one or more object images extracted by the object image extracting means are arranged, and a plurality of acquired images When the same target object image is included, the same image determination in which the same target object image is arranged at the same position included in each monitoring image generated using each acquired image An arrangement means is provided.

  According to a second aspect of the present invention, in addition to the configuration according to the first aspect, the same image determination and arrangement unit extracts feature amount information used for identifying an object from the acquired image, and the extracted feature amount information Is used to determine whether or not a plurality of object images are the same image.

  According to a third aspect of the present invention, in addition to the configuration according to the first or second aspect, the same image determination / arranging unit extracts the feature amount information using a neural network, and the plurality of object images are At least one of the determination of whether or not the images are the same is performed.

  According to a fourth aspect of the present invention, in addition to the configuration according to any one of the first to third aspects, a region image extracting unit that extracts a region image including the object image is provided, and the monitoring image generating unit Generates a monitoring image in which one or more of the region images extracted by the region image extraction unit are arranged.

  According to a fifth aspect of the present invention, in addition to the configuration according to any one of the first to fourth aspects, the number of pixels of the image is reduced to reduce at least one of the acquired image and the monitoring image. An image reduction means that performs the acquisition image or the acquisition image and the monitoring image, and the region of the acquisition image that includes the object image is the target in the monitoring image. It is characterized in that it is reduced so as to have a higher reduction ratio than the region including the object.

  According to a sixth aspect of the present invention, in addition to the configuration according to any one of the first to fifth aspects, a first stream transmission unit that transmits the acquired image and a second stream that transmits the monitoring image. Stream transmission means, and further comprising transmission means for transmitting at least one of the acquired image and the monitoring image.

  According to a seventh aspect of the present invention, in addition to the configuration according to any one of the first to fifth aspects, the monitoring image generation means includes a monitoring image including the acquired image and the object image. The image processing apparatus includes a transmission unit that generates an arrangement image arranged side by side and transmits at least one of the acquired image, the monitoring image, and the arrangement image.

  According to an eighth aspect of the present invention, in addition to the configuration according to any one of the first to fifth aspects, the monitoring image generation unit superimposes a monitoring image including an object image on the acquired image. The image processing apparatus includes a transmission unit that generates a superimposed image arranged together and transmits at least one of the acquired image, the monitoring image, and the superimposed image.

  The invention according to claim 9 is an image processing method executed by a computer, the object image of the object included in the acquired image acquired by the process of acquiring the image and the process of acquiring the image A process for generating a monitoring image in which one or more target images extracted by the process for extracting a target object image of the target object are arranged, and a target of the same target object in a plurality of acquired images And a process of arranging the object image of the same object at the same position included in each monitoring image generated using each acquired image when the object image is included. To do.

  The invention according to claim 10 is an image processing program, a process of acquiring an image, and a process of extracting a target object image included in the acquired image acquired by the process of acquiring the image. , A process of generating a monitoring image in which one or more object images extracted by the process of extracting the object image of the object are arranged, and object images of the same object are included in a plurality of acquired images In this case, the computer is caused to execute a process of arranging the object image of the same object at the same position included in each monitoring image generated using each acquired image.

  According to the first, ninth, and tenth aspects of the present invention, the monitoring image is generated by generating the monitoring image in which the object image extracted from the acquired image is arranged. An object to be monitored in the image is displayed. Since the position of the object in the monitoring image can be made independent of the position of the object in the acquired image, etc., the influence of changes in the position of the object in the acquired image over time Can be generated. As a result, it is possible to generate an image that makes it easy to monitor the target object.

  According to the first, ninth, and tenth inventions, when the same image exists in a plurality of acquired images, the same position included in the monitoring image generated using each acquired image By locating the same object image on the same object, the object image of the same object can be placed at the same position in the plurality of monitoring images regardless of the movement of the object. Accordingly, even if the object moves or the positional relationship of the plurality of objects is switched, the same object can be continuously displayed at the same position in the monitoring image. Accordingly, it is possible to prevent a situation in which the display position of the same object frequently changes in the monitoring image as time passes, and to provide a monitoring image that can easily monitor the object that is the monitoring target.

  According to the second aspect of the present invention, it is determined whether or not the objects are the same by determining whether or not the plurality of object images are the same image using the feature amount information extracted from the acquired image. It is possible to discriminate with accuracy, and it is possible to arrange the same object at the same position of the monitoring image with high accuracy. As a result, it is possible to provide a monitoring image that makes it easier to monitor an object that is a monitoring target.

  According to the third aspect of the present invention, by using a neural network for at least one of extraction of feature amount information and determination of whether or not a plurality of object images are the same image, the same based on the feature amount information It is possible to determine whether or not the image is an image of the monitoring image with high accuracy by learning of the neural network, and it is possible to place the same object at the same position of the monitoring image with high accuracy. As a result, it is possible to provide a monitoring image that makes it easier to monitor an object that is a monitoring target.

  According to the fourth aspect of the present invention, the region image including the object image can be extracted and used as the monitoring image. Therefore, the object image used for monitoring the object can be extracted with a simple procedure. And the processing efficiency can be improved.

  According to the invention described in claim 5, the acquired image or the acquired image and the monitoring image are included by the image reducing means, and the region including the target image in the acquired image includes the target image in the monitoring image. By reducing the image so that the reduction ratio is higher than that of the area, it is possible to generate an image necessary for monitoring the object as high-resolution image data while reducing the data amount of the entire image captured by the imaging unit. Accordingly, it is possible to reduce the data amount of the entire image data to reduce the network load, and to generate an image having a resolution that can easily identify the object to be monitored.

  According to the sixth aspect of the present invention, the acquired image is transmitted by the first stream transmitting unit, and the monitoring image is transmitted by the second stream transmitting unit. It is possible to transmit in real time in a good communication state by the stream type data communication while suppressing the communication state of one image from affecting the communication state of the other image.

  According to the seventh aspect of the present invention, an arrangement image in which the acquired image and the monitoring image including the object image are arranged and arranged is generated, and at least one of the acquired image, the monitoring image, and the arrangement image is generated. By transmitting, the entire image captured by the imaging unit and the image necessary for monitoring the object can be transmitted together without being separated. Thereby, it is possible to provide a monitoring image that can easily contrast the entire image captured by the imaging unit with the object that is the monitoring target included in the entire image.

  According to the eighth aspect of the present invention, a superimposed image is generated by superposing and arranging the monitoring image including the object image on the acquired image, and at least the acquired image, the monitoring image, and the superimposed image are generated. By transmitting one, it is possible to transmit the entire image captured by the imaging unit and the image arranged in a state in which it is easy to contrast the image necessary for monitoring the object. Thereby, it is possible to provide a monitoring image that can easily contrast the entire image captured by the imaging unit with the target object that is included in the entire image.

It is a functional block diagram which shows the image processing apparatus of this embodiment. (A) It is a schematic diagram which shows the state which has arrange | positioned the area | region image extracted from the acquired image of the specific time in the image for monitoring in an image processing apparatus same as the above, (b) An acquired image at the time after the said (a) It is a schematic diagram which shows the state which has arrange | positioned the area | region image extracted from the image for monitoring. It is a flowchart which shows the process sequence in an image processing apparatus same as the above. An example of a state in which an area image is extracted from an acquired image and a monitoring image is generated based on the extracted area image and the generated image is transmitted to the monitoring apparatus by stream type data communication in the image processing apparatus same as above. It is a figure shown typically. In the image processing apparatus as described above, another state in which an area image is extracted from an acquired image, and another state in which a monitoring image is generated based on the extracted area image and the generated image is transmitted to the monitoring apparatus by stream type data communication It is a figure which shows an example typically. In the image processing apparatus, the image processing apparatus generates a monitoring image based on the state in which the area image is extracted from the acquired image and the extracted area image, and the generated image is transmitted to the monitoring apparatus by stream type data communication. It is a figure which shows typically the other example of the state which transmits.

1 to 6 show an embodiment of the present invention.

[Basic configuration]
FIG. 1 is a functional block diagram showing the overall structure of an image processing system 1A according to this embodiment.

[Image processing system]
An image processing system 1A shown in FIG. 1 includes a monitoring camera 1 as “imaging means”, an image processing device 2 according to this embodiment, and a monitoring device 3 as “terminal”. The monitoring camera 1 and the image processing apparatus 2, and the image processing apparatus 2 and the monitoring apparatus 3 are connected by networks 4 and 5.

[Surveillance camera]
The monitoring camera 1 is a camera used for remote monitoring. The surveillance camera 1 includes an image sensor 6 such as a CCD image sensor or a CMOS image sensor. The monitoring camera 1 is installed in a predetermined space for monitoring, for example, a predetermined room or a predetermined outdoor area. The image sensor 6 of the monitoring camera 1 converts an image taken from a lens (not shown) of the monitoring camera 1 into an image (so-called 8K image) of a predetermined number of pixels, for example, 7680 (horizontal) × 4320 (vertical) per frame. ) An image of 4096 × (vertical) 2048 pixels per frame (horizontal) (so-called 4K image) or 1920 × 1080 pixels (so-called 2K image) per frame, etc., and a predetermined number of frames, for example, 30 frames per second As a video. However, the pixel size of the image captured by the monitoring camera 1 and the number of frames per unit time may be any. In this embodiment, the following description will be made assuming that the surveillance camera 1 captures an 8K moving image.

  An image captured by the monitoring camera 1 is transmitted to the image processing apparatus 2 as image data.

[Image processing device]
The image processing device 2 has a function of generating a monitoring image (described later) using an image captured by the monitoring camera 1 and sending it to the monitoring device 3. For example, the image processing apparatus 2 may be provided in the monitoring camera 1 or may be provided separately from the monitoring camera 1. The image processing apparatus 2 performs predetermined image processing by processing using a CPU, processing using hardware logic, or the like.

  As shown in the functional block diagram of FIG. 1, the image processing apparatus 2 includes an image acquisition unit 7 as an “image acquisition unit”, a face image detection unit 8 as an “object image extraction unit”, and an “object image detection unit”. , A face image extracting unit 9 as “object image extracting unit” and “region image extracting unit”, a monitoring image generating unit 10 as “monitoring image generating unit”, and an image reducing unit 11 as “image compressing unit” , An encoder 12, and a transmission unit 13 as "transmission means". The monitoring image generation unit 10 includes a feature amount extraction unit 14 as “feature amount information extraction unit” and a determination / sorting unit 15 as “same image determination / arrangement unit”. A part or all of these configurations may be configured as hardware, or may be configured as a program that functions by computing with a computer. The program may be stored in a computer-readable non-transitory recording medium. Examples of the recording medium include an SD memory card (SD Memory Card), an FD (Floppy Disk), a CD (Compact Disc), a DVD (Digital Versatile Disk), a BD (Blu-ray (registered trademark) Disk :), and a flash memory. It is a non-temporary recording medium.

  The image acquisition unit 7 acquires an image captured and transmitted by the monitoring camera 1. The acquired image is handled as an acquired image (hereinafter simply referred to as “acquired image”) as an image acquired by the image acquisition unit 7 in the image processing apparatus 2. The acquired image is generated for each frame image sent from the monitoring camera 1.

  The face image detection unit 8 detects an object image that is an image of a predetermined object included in the acquired image.

  The face image detection unit 8 has a function as an object detector using an algorithm of a neural network for object detection, for example, YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector), and is included in an acquired image. Detect objects. Here, the face image detection unit 8 can use an algorithm such as OpenCV without using a neural network.

  In this embodiment, as shown in FIG. 1, “object” is a face 101 of the person 100, and “object image” is a face image obtained by photographing the face 101 of the person 100. In this embodiment, the face image detection unit 8 detects a face image from the acquired image by extracting a feature amount from the acquired image using a neural network. In this embodiment, the detection of the face image by the face image detection unit 8 detects the face position of the person 100 in the acquired image, that is, the position where the face 101 of the person 100 is shown. The detection of the face image in the method may be any method as long as it detects the face image in the acquired image. For example, the face 101 of which person 100 appears in which direction in the acquired image. Information may be acquired.

  The face image extraction unit 9 extracts a region image (hereinafter simply referred to as “region image”) as a region including the face image detected by the face image detection unit 8.

  The monitoring image generation unit 10 is referred to as a monitoring image (hereinafter simply referred to as a “monitoring image”) in which one or a plurality of area images extracted by the face image extraction unit 9 are arranged in a predetermined arrangement area. Details will be described later. ) Is generated.

  The image reduction unit 11 reduces the number of pixels of the acquired image to reduce the acquired image. In this embodiment, the image reduction unit 11 reduces only the acquired image among the acquired image and the region image, but both the acquired image and the region image may be reduced.

  The encoder 12 encodes (encodes) the acquired image and the monitoring image. For encoding, for example, H.264. H.264 and H.264. An image compression protocol such as H.265 is used.

  The transmission unit 13 transmits the encoded acquired image and the monitoring image to the monitoring device 3. In this embodiment, the transmission unit 13 is a predetermined protocol such as H.264. H.264 and H.264. An image is encoded at 265 to perform stream-type data communication. Further, the transmission unit 13 can perform stream communication of a plurality of systems, convert the acquired image and the monitoring image into two streams, perform stream-type data communication with separate systems, and send each image to the monitoring device 3. Send. However, the transmission unit 13 may perform communication by using the acquired image and the monitoring image as the same data stream only by stream data communication of only one system.

  The feature amount extraction unit 14 extracts feature amount information, which is information of a feature amount necessary for identifying the face 101 of the person 100, from the acquired image. This feature amount information will be described later.

  In this embodiment, the feature amount extraction unit 14 has a feature detection function using a neural network algorithm for feature detection, for example, VGG16 or VGG-Face, and extracts feature amount information from the face image in the acquired image. To detect.

  The determination / sorting unit 15 is the same image (hereinafter, simply referred to as “same image”) that is a face image obtained by photographing the face 101 of the same person 100 among a plurality of acquired images acquired at different times. Is present, the same image is arranged at the same position (hereinafter simply referred to as “the same position”) of the respective monitoring images generated based on the respective acquired images. In this embodiment, the determination / sorting unit 15 determines whether or not the images are the same by comparing the feature amounts extracted by the feature amount extraction unit 14.

[Monitoring device]
The monitoring device 3 is a terminal that can transmit and receive data and signals to and from the image processing device 2 via the network 5 such as a personal computer, a smartphone, and a tablet. The monitoring device 3 includes an operation unit 31 such as a mouse and a keyboard, and a display unit 32 such as an LCD display. The operation unit 31 and the display unit 32 may be integrated like a touch panel display.

  The monitoring device 3 in this embodiment decodes (decodes) the image sent from the image processing device 2, displays the monitoring image on the display unit 32, and causes the monitoring person to visually recognize the monitoring image. It is comprised as an apparatus for performing monitoring by. However, the monitoring device 3 is provided with a person determination function based on image recognition using a neural network or the like, and the monitoring device 3 automatically identifies and monitors the person 100 in the image taken by the monitoring camera 1. May be.

[network]
The network 4 is a communication medium for connecting the monitoring camera 1 and the image processing apparatus 2 by wire. The network 5 is a communication medium for connecting the image processing apparatus 2 and the monitoring apparatus 3 by wire connection or wireless connection. As long as the networks 4 and 5 can transmit and receive image data and a signal for controlling the image processing system 1A, they may be any communication format or communication medium.

[Extract region image]
In this embodiment, the feature amount information extracted from the region image by the feature amount extraction unit 14 is vector information included in the image (that is, the image information includes parts of the face 101 such as “eyes” and “nose”). It is acquired as information representing image characteristics that is more abstract than information such as the position and size of the image). The feature amount information is feature amount information used for identifying the face 101 of the person 100. Specifically, it is determined whether or not the face 101 of the person 100 included in the plurality of region images is the face of the same person. It is information used to The feature amount extraction unit 14 inputs an acquired image to the feature amount detection function of the neural network provided, and extracts feature amount information used for identifying the face 101 of the person 100 from each region image. Then, the determination / rearrangement unit 15 determines whether the images included in the respective region images are the same image, that is, whether the face 101 is the same person, using the feature amount information.

  Specifically, the feature quantity extraction unit 14 uses a neural network or the like that appropriately captures image features such as a 4096-dimensional feature vector of VGG16, for example, and the face included in the region image is the face 101 of the same person. It is conceivable to determine whether or not. When using the feature amount of the neural network, for example, the face image of the previous frame (region image extracted from the acquired image) and the face image of the current frame (region image extracted from the acquired image) are 4096-dimensional. It may be possible to determine whether or not the face images being compared are face images of the same person by calculating feature vectors and looking at the value of the inner product or vector distance.

  Note that it is also possible to use a square error between images without using the feature amount information extracted by the feature amount extraction unit 14 in the determination / rearrangement unit 15 for determining whether or not they are face images of the same person. It is done.

  Then, the determination / sorting unit 15 has a high possibility that the area image extracted from the specific acquired image and the area image of the acquired image acquired next are the same image including the face image of the same person. The same image of the monitoring image generated using the next region image is displayed in the same arrangement as the same image included in the monitoring image generated using the previous region image.

[Placement of area image on monitoring image]
FIG. 2 is a diagram schematically showing the principle of processing when an area image is arranged on a monitoring image in the image processing apparatus 2 of the image processing system 1A of this embodiment.

As shown in (a) of FIG. 2, the face 101 ₁ of the person 100 ₁ to obtain an image 21 ₁ of a particular point in time, the face 101 ₂ of the person 100 _2, with the face 101 ₃ was captured person 100 _3, this from the obtained image 21, the area image 23 ₁ of the face 101 ₁ person 100 _1, a person 100 ₂ face 101 _{and second} area image 23 _2, surveillance area image 23 ₃ of the face 101 ₃ persons 100 ₃ is generated respectively Consider the case where an image 22 is generated. In this case, when each person 100 freely moves in the space, the face 101 of each person 100 of the image captured in each frame of the moving image captured by the surveillance camera 1 moves with time. The size changes. (B) in FIG. 2 is a state in which the acquired image 21 ₂ has been obtained after obtaining the image 21 ₁ is acquired as shown in FIG. 2 (a) shows schematically. In this state, the position in the lateral direction of the person 100 ₁ and the person 100 ₃ are swapped, is reflected a person 100 ₄ on behalf of the person 100 _2. In the actual photographing of the person 100 in the space, the positional relationship between the faces 101 of the person 100 in the acquired image 21 is changed over time as described above, or the person 100 appears or leaves every frame. It happens frequently.

  In such a situation, the order of face detection in the face image detection unit 8 is not constant due to the characteristics of the neural network. Therefore, when the plurality of area images 23 are arranged in the arrangement area of the monitoring image 22 in the order in which the face images are detected by the face image detection unit 8, the display order of the face 101 is changed for each monitoring image 22. It changes rapidly, and the visibility etc. of the face image which is a monitoring object, such as the monitoring person who monitors the display part 32 of the monitoring apparatus 3, will fall.

  Therefore, in this embodiment, as a result of the feature amount extraction by the feature amount extraction unit 14, a plurality of acquired images, for example, region images 23 extracted from a specific acquired image 21 and immediately after the acquired images 21 are generated. When the same image in which the same person appears is included in the region image 23 extracted from the generated acquired image 21, the determination / sorting unit 15 converts the same image into each of the monitoring images 22. Arrange them at the same position.

For example, in the monitoring image 22 of FIG. 2 (b), like the monitor image 22 of (a) 2, the face 101 ₁ of the person 100 ₁ to the leftmost area image 23 _1, most face 101 ₃ persons 100 ₃ to the right of the area image 23 _3, while being displayed respectively is maintained. On the other hand, in FIG. 2 (b), the area image 23 ₄ in the middle of the monitoring image 22 until the acquired image 21 ₂ of (in FIG. 2 from the acquired image 21 ₁ of FIG. 2 (a) (b) instead of the person 100 ₂ face 101 ₂ that is no longer displayed on the course in) the image of the time course of, and newly displayed person 100 ₄ face 101 ₄ appears. With this configuration, (for example in the FIG. 2 (a) (b), the face 101 ₁ of the person 100 _1, a person 100 ₃ face 101 ₃₎ the same person's face 101 displayed monitoring images 22 For example, the situation in which the position on the monitoring image 22 ₁ in FIG. 2A and the monitoring image 22 _{2 in} FIG. 2B rapidly changes with time is suppressed.

  In the following description, an acquired image 21 (that is, a specific acquired image 21 and an acquired image 21 acquired immediately thereafter) generated from adjacent frames of a moving image captured by the monitoring camera 1 is the same image. However, the present invention is not limited to this, and any acquired image 21 may be determined as long as it can be determined whether the images are the same. For example, the same image may be used when the face 101 of the same person is shown in common in the acquired image 21 composed of three or more adjacent frames, or acquired at every predetermined number of frames, for example, every 10 frames. A case where the face 101 of the same person is shown in common with the image 21 may be the same image.

  In addition, the determination / sorting unit 15 of this embodiment does not display the same image at the same position of the monitoring image 22 but assigns an ID to each person 100 and sets a display position for each person 100 set in advance. The monitoring image 22 may be generated by rearranging so that the area image 23 in which each person 100 is captured is displayed.

[Processing procedure]
FIG. 3 is a flowchart showing a processing procedure of the image processing system 1A according to this embodiment. The processing procedure of this embodiment will be described below based on this flowchart.

  First, when a monitor (hereinafter simply referred to as “monitorer”) who performs remote monitoring using the image processing system 1A activates the image processing system 1A, the monitoring camera 1 captures a predetermined space with a moving image (step S1). ). The surveillance camera 1 acquires a high-resolution image (here, an 8K image) as a moving image. An image taken by the monitoring camera 1 is sent to the image processing apparatus 2 via the network 4.

  In the image processing device 2, the image acquisition unit 7 acquires the image sent from the monitoring camera 1 (step S2). The acquired image 21 acquired by the image acquisition unit 7 is sent to the face image detection unit 8. The face image detection unit 8 performs a process for reducing the acquired image 21 into a predetermined format (for example, a process of reducing the number of pixels to change the 8K acquired image 21 to the acquired image 21 having a lower resolution). After this processing, the face image detection unit 8 inputs the acquired image 21 to the face detection function of the provided neural network, and detects the face position included in the acquired image 21 (step S3).

FIG. 4 schematically shows a state in which the face position is detected in step S2. Specifically, the face image detection unit 8 acquires four variables x, y, w, and h as information on the face position in the acquired image 21. Note that, x, y, w, 4 variables h, for example, the (x, y) is the coordinate position P of the upper left coordinates (original specific pixels of the acquired image 21 ₁ of the image (x, y) This coordinate position P corresponds to the upper left vertex of the area image 23 _{1 to be} extracted.), W represents the width of the image (size in the width direction of the area image 23 _{1 to be} extracted), and h There is shown the height of the image (size in the height direction of the area image 23 ₁ of extract).

  Next, the face image extraction unit 9 extracts the region image 23 by cutting out the face portion detected by the face image detection unit 8 (step S4).

FIG. 4 is a diagram schematically illustrating a state in which the region image 23 is extracted from the acquired image 21 and a state in which the monitoring image 22 is generated based on the extracted region image 23. As shown in the figure, for example, the region image 23 ₁ is obtained by extracting the vicinity of the face 101 ₁ of the person 100 ₁ shown in the original acquired image 21 and setting the x and y coordinate positions of the original acquired image 21 to the upper left vertex. And a rectangular image having a width w and a height h. The area image 23 ₁ is the number of fewer pixels than the original acquired image 21 of the image, has become a 2K image here.

  In this embodiment, the image reduction unit 11 performs a process of reducing the number of pixels of the original acquired image 21 (for example, a process of reducing the number of pixels of the 8K image to obtain the 2K acquired image 21). Later, encoding processing by the encoder 12 and transmission to the monitoring device 3 are performed, but the region image 23 is cut out from the original acquired image 21 to form a 2K image. By using a 2K image, only the portion of the acquired image 21 where the face 101 of the person 100 is photographed substantially retains the image quality of the 8K image, and the encoding process by the encoder 12 and the transmission to the monitoring device 3 are performed. It can be carried out. In other words, in this embodiment, even if the processing capability of the encoder 12 is low or the bandwidth of the network 5 is narrow, a situation such as an encoding delay or a communication delay is suppressed, and a high resolution is achieved. Monitoring using an image taken by the monitoring camera 1 can be performed.

  Next, the monitoring image generation unit 10 generates a monitoring image 22 by using one or a plurality of area images 23 extracted by the face image extraction unit 9 (step S5).

  The procedure of step S4 will be described in detail. First, the feature amount extraction unit 14 performs processing for reducing the area image 23 into a predetermined format (for example, reducing the number of pixels to make the 2K area image 23 an area having a lower resolution). (Process to make image 23). After this processing, the feature amount extraction unit 14 extracts feature amount information from the region image 23 (step S51).

The determination / sorting unit 15 compares the feature amount information for each region image 23 extracted by the feature amount extraction unit 14. Then, the neighboring acquired image 21 (that is, for example, shown in (a) of FIG. 2, an acquired image 21 ₁ generated by the particular frame monitoring camera 1 is taken, shown in FIG. 2 (b), the frame If the feature amount information of the face image acquired from the acquired image 21 ₂ ) generated by the next frame is the same or within a predetermined similar range, the determination / sorting unit 15 The face image of the same person, that is, the same image is determined.

  Then, when the plurality of region images 23 extracted from the adjacent acquired images 21 are the same image, the determination / sorting unit 15 rearranges each region image 23 generated based on each acquired image 21. Thus, the same image is arranged at the same position of the monitoring image 22 (step S52). Further, the region images 23 that are not the same image are arranged in an arrangement region where the same image of the monitoring image 22 is not arranged by a predetermined process of the determination / sorting unit 15.

  The monitoring image 22 is generated in the monitoring image generation unit 10 by the processing in step S5 described above. Each area image 23 arranged in the monitoring image 22 has the number of pixels of the 2K image.

  The image reduction unit 11 acquires the acquired image acquired by the image acquisition unit 7 from the monitoring camera 1 through a process different from the processes in the face image detection unit 8 to the face image extraction unit 9 to the monitoring image generation unit 10 described above. A reduction process is performed to reduce the number of pixels 21 (step S6). In this embodiment, the image reduction unit 11 performs a process of generating the acquired image 21 in which the number of pixels of the 8K image of the original acquired image 21 is reduced to the number of pixels of the 2K image. By performing such processing, the amount of image data can be reduced, so that even if the image of the number of pixels photographed by the monitoring camera 1 cannot be processed with the encoding processing capability of the encoder 12, encoding is performed properly. In addition, it is possible to suppress a situation where the load on the bandwidth of the network 5 when sending image data is excessive.

  The encoder 12 encodes (encodes) the monitoring image 22 generated in step S4 and the acquired image 21 generated in step S5 (step S7).

  Then, the transmission unit 13 transmits the encoded acquired image 21 and the monitoring image 22 to the monitoring device 3 (step S8). Here, as illustrated in FIG. 4, the transmission unit 13 includes two of an acquired image stream 24 as a “first stream transmission unit” and a monitoring image stream 25 as a “second stream transmission unit”. The image may be transmitted to the monitoring device 3 by a system data stream.

  Note that the transmission unit 13 may generate an image in a form in which the acquired image 21 and the monitoring image 22 are combined into one image, and may transmit the image to the monitoring device 3 using one data stream 28. For example, as in another example illustrated in FIG. 5, the transmission unit 13 generates an arrangement image 26 in which the upper half is arranged and arranged as the acquired image 21 and the lower half is the monitoring image 22 and is monitored by the data stream 28. You may transmit to the apparatus 3. Further, as in another example illustrated in FIG. 6, the transmission unit 13 generates a superimposed image 27 in which the monitoring image 22 is superimposed on the acquired image 21 in a window display form. The data stream 28 may be transmitted to the monitoring device 3. Further, an image transmitted from the image processing apparatus 2 to the monitoring apparatus 3 by remote operation from the monitoring apparatus 3 or the like is used as a specific monitoring image 22 (for example, a specific area image in which the face 101 of a specific person 100 is captured). 23) may be transmitted as a zoomed image, or transmission between the acquired image 21 and the monitoring image 22 may be switched.

  As described above, in this embodiment, by generating the monitoring image 22 in which the face image extracted from the acquired image 21 is arranged, the monitoring image 22 includes the face that is the monitoring target in the acquired image 21. An image is displayed. Since the position of the face image in the monitoring image 22 can be made independent of the position of the face 101 of the person 100 in the acquired image 21 or the like, the position of the person 100 in the acquired image 21 as time passes. The monitoring image 22 in which the influence of changes such as the position of the face 101 is suppressed can be generated. As a result, it is possible to generate an image that facilitates monitoring of the face 101 of the person 100 to be monitored.

  In this embodiment, when the same image obtained by photographing the face 101 of the same person exists in the plurality of acquired images 21, the same position included in the monitoring image 22 generated using each acquired image 21. By arranging the same person's face image in the same position, the same person's face image can be arranged at the same position in the plurality of monitoring images 22 regardless of the movement of the person 100, etc. Even if the face 101 of the person 100 moves or the positional relationship of the faces 101 of the plurality of persons 100 is switched, the face image of the same person can be continuously displayed at the same position of the monitoring image 22. As a result, it is easy to monitor the face 101 of the person 100 to be monitored by suppressing a situation in which the position where the face 101 of the same person is displayed in the monitoring image 22 is frequently changed over time. A monitoring image 22 can be provided.

  In this embodiment, by using the feature amount information extracted from the acquired image 21, it is determined whether or not the plurality of face images are the face images of the same person, thereby determining whether or not the photographed person 100 is the same. The face image of the same person can be arranged at the same position of the monitoring image 22 with high accuracy. As a result, it is possible to provide the monitoring image 22 that makes it easier to monitor the face 101 of the person 100 to be monitored.

  In this embodiment, a neural network is used for extracting feature amount information and determining whether or not a plurality of face images are the same image, thereby determining whether or not they are face images of the same person based on the feature amount information. Can be performed with high accuracy by learning a neural network, and face images of the same person can be arranged at the same position of the monitoring image 22 with high accuracy. As a result, it is possible to provide the monitoring image 22 that makes it easier to monitor the face 101 of the person 100 to be monitored.

  In this embodiment, since the area image 23 including the face image can be extracted and used as the monitoring image 22, the extraction of the face image used for monitoring the person 100 to be monitored is performed with a simple procedure. And the processing efficiency can be improved.

  In this embodiment, the image reduction unit 11 causes the acquired image 21 or the acquired image 21 and the monitoring image 22 to be included in the region including the face image in the acquired image 21. By reducing the image so that the reduction ratio is higher than that of the area, the amount of data of the entire image captured by the monitoring camera 1 is reduced, and the image necessary for monitoring the face 101 of the person 100 to be monitored has a high resolution. Image data. As a result, the data amount of the entire image data can be reduced to reduce the network load, and an image having a resolution that can easily identify the face 101 of the person 100 to be monitored can be generated.

  In this embodiment, the acquired image 21 is transmitted by the first stream transmitting unit, and the monitoring image 22 is transmitted by the second stream transmitting unit. It is possible to transmit in real time in a good communication state by the stream type communication while suppressing the communication state of one image from affecting the communication state of the other image.

  In this embodiment, an arrangement image 26 in which the acquired image 21 and the monitoring image 22 including the face image are arranged side by side is generated, and at least one of the acquired image 21, the monitoring image 22, and the arrangement image 26 is generated. By transmitting one, it is possible to transmit the entire image captured by the monitoring camera 1 and the image necessary for monitoring the person 100 to be monitored together without being separated. Thereby, it is possible to provide the monitoring image 22 that can easily contrast the entire image captured by the monitoring camera 1 with the face 101 of the person 100 that is the monitoring target included in the entire image.

  In this embodiment, a superimposed image 27 is generated by superimposing a monitoring image 22 including a face image on the acquired image 21, and the acquired image 21, the monitoring image 22, and the superimposed image 27 are among the acquired images 21, superposed images 27. By transmitting at least one of them, it is possible to transmit an entire image captured by the monitoring camera 1 and an image arranged in a state where it is easy to contrast the image necessary for monitoring the person 100 to be monitored. Accordingly, it is possible to provide the monitoring image 22 that can easily contrast the entire image captured by the imaging unit with the face 101 of the person 100 that is the monitoring target included in the entire image.

  In this embodiment, the face image extraction unit 9 extracts a region image 23 as a region including the face image detected by the face image detection unit 8 from the acquired image 21, and displays the monitoring image 22. However, instead of this, the face image extraction unit 9 extracts only the face image detected by the face image detection unit 8 from the acquired image 21, and generates the monitoring image 22 using only this face image. It is good also as a structure.

  In this embodiment, the object to be monitored is the face 101 of the person 100. However, the object is not limited to this, and the object other than the face 101 of the person 100, for example, a full body image, a hand, a leg, or the like. Other than the person 100, for example, an animal, an insect, a plant, or the like, or an automobile, an airplane, a train, or the like may be used.

  The above embodiment is an exemplification of the present invention, and it is needless to say that the present invention is not limited to the above embodiment.

1 ... Surveillance camera (photographing means)
2 ... Image processing device 7 ... Image acquisition unit (image acquisition means)
8. Face image detection unit (object image extraction means, object image detection means)
9 ... Face image extraction unit (object image extraction means, area image extraction means)
10: Monitoring image generation unit (monitoring image generation means)
11: Image reduction section (image reduction means)
13: Transmitter (transmitter)
15 ... Judgment / rearrangement unit (same image judgment arrangement means)
21, 21 ₁ , 21 ₂ ... Acquired image 22, 22 ₁ , 22 ₂ ... Monitoring image 23, 23 ₁ , 23 ₂ , 23 ₃ . One stream transmission means)
25 ... Image stream for monitoring (second stream transmission means)
26 ... arranged image 27 ... superimposed image 100, 100 _{1, 100 2,} 100 3, ₁₀₀ 4.. People 101, 101 _{1, 101 2,} 101 3, ₁₀₁ 4.. Face

Claims (10)

Image acquisition means for acquiring images;
Object image extraction means for extracting the object image of the object included in the acquired image acquired by the image acquisition means;
Monitoring image generation means for generating a monitoring image in which one or more object images extracted by the object image extraction means are arranged;
When object images of the same object are included in a plurality of acquired images, the object image of the same object is placed at the same position included in each monitoring image generated using each acquired image. An image processing apparatus comprising: an identical image determination / arranging means for arranging. The same image determination / arrangement unit extracts feature amount information used for identifying an object from the acquired image, and determines whether or not a plurality of object images are the same image using the extracted feature amount information. The image processing apparatus according to claim 1. The said same image determination arrangement | positioning means performs at least one of the extraction of said feature-value information using a neural network, and the determination whether these several target object images are the same images. An image processing apparatus according to 1. The object image extraction means includes
A region image extracting means for extracting a region image of a region including the object image;
4. The monitoring image generation unit generates a monitoring image in which one or more region images extracted by the region image extraction unit are arranged. 5. Image processing device. Image reduction means for reducing the number of pixels of the image and reducing at least one of the acquired image and the monitoring image;
The image reduction means is configured to reduce the acquired image or the acquired image and the monitoring image in a region including the object image in the acquired image compared to a region including the object image in the monitoring image. The image processing apparatus according to claim 1, wherein the image processing apparatus is reduced so as to have a high reduction ratio.   Transmission means for transmitting at least one of the acquired image and the monitoring image, comprising: a first stream transmitting means for transmitting the acquired image; and a second stream transmitting means for transmitting the monitoring image. The image processing apparatus according to claim 1, further comprising: an image processing apparatus according to claim 1. The monitoring image generation means includes
Generating an arrangement image in which the acquired image and the monitoring image including the object image are arranged side by side;
The image processing apparatus according to claim 1, further comprising a transmission unit configured to transmit at least one of the acquired image, the monitoring image, and the arrangement image. The monitoring image generation means includes
Generating a superimposed image in which the monitoring image including the object image is superimposed on the acquired image,
The image processing apparatus according to claim 1, further comprising: a transmission unit that transmits at least one of the acquired image, the monitoring image, and the superimposed image. An image processing method executed by a computer,
Processing to acquire images,
A process of extracting a target object image included in the acquired image acquired by the process of acquiring the image;
A process of generating a monitoring image in which one or more object images extracted by the process of extracting the object image of the object are arranged;
When object images of the same object are included in a plurality of acquired images, the object image of the same object is placed at the same position included in each monitoring image generated using each acquired image. An image processing method comprising: arranging processing. Processing to acquire images,
A process of extracting a target object image included in the acquired image acquired by the process of acquiring the image;
A process of generating a monitoring image in which one or more object images extracted by the process of extracting the object image of the object are arranged;
When object images of the same object are included in a plurality of acquired images, the object image of the same object is placed at the same position included in each monitoring image generated using each acquired image. An image processing program that causes a computer to execute a process of arranging.

Images