JP2017158067A - Monitoring system, monitoring method, and monitoring program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a monitoring system capable of monitoring more accurately than before.SOLUTION: The monitoring system includes an imaging unit, a storage unit, an image analysis unit and a display unit. The storage unit stores dictionary data including coefficients for realizing a plurality of mutually different super-resolution processing. The image analysis unit performs super-resolution processing on the image captured by the imaging unit, using one of a plurality of pieces of dictionary data stored in the storage unit. The display unit displays the image on which the super-resolution processing has been performed by the image analysis unit.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to a monitoring system, a monitoring method, and a monitoring program.

  A technique for sharpening an image by super-resolution processing is known. The conventional technique, when applied to a system for monitoring a peripheral object, sharpens an image without limiting the type of target or environmental conditions.

“Image Super-Resolution Using Deep Convolutional Networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume38, Issue2, Pages 295-307, Jun.7, 2015

  The problem to be solved by the present invention is to provide a monitoring system capable of performing more accurate monitoring in a scene where a plurality of types of objects may appear.

  The monitoring system according to the embodiment includes an imaging unit, a storage unit, an image analysis unit, and a display unit. The storage unit stores dictionary data including coefficients for realizing a plurality of different super-resolution processes. The image analysis unit performs super-resolution processing on the image captured by the imaging unit using any of a plurality of dictionary data stored in the storage unit. The display unit displays the image on which the super-resolution processing has been performed by the image analysis unit.

It is a figure which shows an example of a structure of the monitoring system. It is a figure which shows an example of a function structure of the monitoring system. 4 is a flowchart showing a flow of processing executed by the monitoring system 1. It is a figure for demonstrating determination of the window W. FIG. It is a figure which shows an example of the dictionary data. It is a figure which shows an example of a mode that the image IM (1) to (4) based on each dictionary data 64 is displayed by the display part. 6 is a diagram illustrating another example in which an image is displayed on the display unit 40. FIG. 10 is a flowchart illustrating a part of a flow of processing executed by the image processing apparatus 50 according to a modification of the first embodiment.

  Hereinafter, a monitoring system, a monitoring method, and a monitoring program according to embodiments will be described with reference to the drawings. Moreover, it demonstrates using a XYZ coordinate as needed.

  FIG. 1 is a diagram illustrating an example of the configuration of the monitoring system 1. FIG. 2 is a diagram illustrating an example of a functional configuration of the monitoring system 1. As shown in FIG. 1, the monitoring system 1 is mounted on a moving body such as a vehicle, a ship, or an aircraft. Moreover, the monitoring system 1 may be fixed on the ground or the sea. The monitoring system 1 monitors surrounding objects. In the illustrated example, the monitoring system 1 is mounted on a vehicle S that is a moving body. The monitoring result of the monitoring system 1 is displayed on the display unit 40 and is visually recognized by the supervisor SB.

  The monitoring system 1 includes, for example, an imaging unit 10, a radar device 20, a sensor 30, a display unit 40, and an image processing device 50. The imaging unit 10 includes a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 10 repeatedly captures the periphery of the monitoring system 1 at a predetermined cycle, and outputs captured image data to the image processing device 50.

  The radar apparatus 20 radiates radio waves toward an object such as an object and receives a reflected wave reflected by the object. The radar apparatus 20 specifies the distance from the radar apparatus 20 to the target object based on the time from radiating the radio wave to receiving the reflected wave, the phase of the reflected wave, and the like. The radar apparatus 20 is a radar that radiates microwaves, for example. The radar apparatus 20 may be another type of radar apparatus such as a millimeter wave radar or a laser radar. Further, the radar apparatus 20 may be one that can change the directivity angle electronically like a phased array radar, or may be one that is mechanically rotationally driven in the horizontal direction.

  The sensor 30 measures the momentum of the moving body on which the monitoring system 1 is mounted. The sensor 30 is, for example, an acceleration sensor, a gyro sensor, a direction sensor, or the like.

  The display unit 40 is a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence). The display unit 40 displays the processing result of the monitoring system 1 and the like. In addition to the display unit 40, an image such as a keyboard, a mouse, a microphone, and other input devices 42 that accept user operations, and other input devices 42 are provided. When the display unit 40 is a tablet terminal, the input device 42 may be a touch panel included in the tablet terminal.

  The image processing apparatus 50 may include a front image processing unit 52, a rear image processing unit 54, and a storage unit 60, but is not limited thereto. The pre-stage image processing unit 52 and the post-stage image processing unit 54 may be realized by a processor such as a CPU (Central Processing Unit) executing a program. Further, the pre-stage image processing unit 52 and the post-stage image processing unit 54 may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). The storage unit 60 is realized by a storage device such as a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a flash memory. The storage unit 60 stores image data 62 and dictionary data 64 to be described later. The image data 62 may include image data captured by the imaging unit 10 and image data processed by the image processing device 50.

  The pre-stage image processing unit 52 performs super-resolution processing by reconstruction on the time-series images captured by the imaging unit 10. The super-resolution processing by reconstruction is processing for improving resolution using information obtained by accurately estimating the movement amount between a plurality of imaging frames. The pre-stage image processing unit 52 performs super-resolution processing by reconstruction based on the position change of the imaging unit 10 detected by the sensor 30.

  The post-stage image processing unit 54 performs super-resolution processing on the image that has been subjected to super-resolution processing by reconstruction by the pre-stage image processing unit 52 using any of a plurality of dictionary data 64. The post-stage image processing unit 54 performs super-resolution processing using the dictionary data 64 corresponding to the type of object according to the operation received by the input device 42.

  FIG. 3 is a flowchart showing a flow of processing executed by the monitoring system 1. In this embodiment, the monitoring system 1 demonstrates the example which performs each process with respect to one target object. When the monitoring system 1 uses a plurality of objects as objects, the monitoring system 1 performs the processing described later on each object in parallel.

  First, the image processing apparatus 50 acquires an image captured by the imaging unit 10 (step S100). Next, the pre-stage image processing unit 52 executes super-resolution processing by reconstruction (step S102). In the super-resolution processing by reconstruction, the pre-stage image processing unit 52 determines the position of the window that is estimated to include the same object in the plurality of frame images. The window is a partial area that is considered to include an object in the image captured by the imaging unit 10.

  For example, the pre-stage image processing unit 52 determines the position of the window based on feature points in the image, information acquired from the radar device 20, and part or all of the information acquired from the sensor 30.

  FIG. 4 is a diagram for explaining the determination of the window W. (1), (2), and (3) in FIG. 4 (A) and FIG. 4 (B) correspond to each imaging frame, and each imaging frame corresponds to an image IM (shown in FIG. 4C). 1), IM (2) and IM (3).

  The pre-stage image processing unit 52 determines the window W based on the information acquired by the radar device 20. When acquiring the amount of movement of the object OB, the pre-stage image processing unit 52 determines the position of the window W so that the object OB is located at the same position in the window W in consideration of the amount of movement. At this time, the pre-stage image processing unit 52 performs conversion from coordinates on the real plane to coordinates on the image plane.

  Further, the pre-stage image processing unit 52 derives the azimuth (direction), movement amount, movement direction, and the like of the imaging unit 10 (or the vehicle S) from the sensor 30. The pre-stage image processing unit 52 converts the imaging direction, movement amount, and movement direction of the imaging unit 10 into the movement amount and movement direction of the window W in the image, and changes in the window W due to the azimuth and movement of the imaging unit 10. The position of the window W is determined so as to be canceled out.

  Further, the pre-stage image processing unit 52 may determine the reference target acquired from the imaging unit 10 and determine the window W based on a change in position or size of the determined reference target image. The reference target may be a vanishing line such as a horizon, or may be a known stationary object. For example, the pre-stage image processing unit 52 detects an annihilation line from the acquired image by a technique such as Hough transform, and estimates the movement of the erasure line in the image, thereby taking an imaging direction unrelated to the movement of the object OB. Detect changes in Then, the pre-stage image processing unit 52 determines the position of the window W so as to cancel the change.

  The pre-stage image processing unit 52 determines the position of the window W by combining the above methods. For example, the pre-stage image processing unit 52 changes the position of the window W based on the change in the position of the object OB, changes in the position of the window W based on changes in the orientation of the imaging unit 10, and the window based on the position in the reference target image. The position of the window W is determined by assigning a weight to each change in the position of W and obtaining a weighted sum.

  After determining the position of the window W, the pre-stage image processing unit 52 performs super-resolution processing by reconstruction based on the images in the windows W of the plurality of frame images. The pre-stage image processing unit 52 detects the movement (or orientation) of the object OB in the image in the window W, and generates a high-definition image in consideration of the detected movement. The pre-stage image processing unit 52 determines a reference frame image as a reference from images in a predetermined number of windows W. The pre-stage image processing unit 52 performs non-rigid deformation processing for each non-reference frame image that is a frame image that is not selected as the reference frame image. For example, the pre-stage image processing unit 52 performs processing so that the orientation of the object OB in the non-reference frame image matches the orientation of the object OB in the reference frame image. The pre-stage image processing unit 52 performs non-rigid deformation processing using, for example, a known FFD (Free Form Deformation) or other known methods.

  For example, the pre-stage image processing unit 52 divides the region in the image in a grid pattern in the vertical direction and the horizontal direction so that the object OB has a predetermined number of pixels with respect to the reference frame image and the non-reference frame image. A desired area centering on a lattice point where a line divided in the vertical direction intersects with a line divided in the horizontal direction corresponds to one pixel. The pre-stage image processing unit 52 sets a plurality of feature points for the object OB of the image to be processed. The pre-stage image processing unit 52 gives each grid point set on the reference frame image a coefficient in the horizontal direction and the vertical direction. The pre-stage image processing unit 52 calculates the positions (coefficients) of lattice points that match the positions of the plurality of feature points in the non-reference frame image with the positions on the reference frame image.

  The pre-stage image processing unit 52 moves each grid point of the non-reference frame image based on the coefficient calculated for the grid point. At this time, the image of the region surrounded by the four adjacent grid points is deformed in accordance with the movement of these grid points, so that the orientation of the target object OB in each non-reference frame image is the target object in the reference frame image. It is matched to the direction of OB. Then, the pre-stage image processing unit 52 calculates an average value, a weighted average value, and the like of the corresponding pixels in the reference frame image and the non-reference frame image subjected to the above-described processing, and determines the average value or the like as the pixel value. To do. Thereby, the pre-stage image processing unit 52 acquires an image that has been subjected to super-resolution processing by reconstruction. Note that the pre-stage image processing unit 52 may acquire an enlarged image using an intra-frame reconstruction type super-resolution process using self-congruity.

  Next, the post-stage image processing unit 54 executes super-resolution processing using the dictionary data (1) (step S104). For example, the post-stage image processing unit 54 performs super-resolution processing on the window W including the object OB. The dictionary data 64 stored in the storage unit 60 includes dictionary data (1) to (n).

  The dictionary data 64 stores, for example, dictionary data for each type of object that can be imaged by the imaging unit 10. The dictionary data 64 may store dictionary data for each imaging condition such as sunny or rainy. FIG. 5 is a diagram illustrating an example of the dictionary data 64. In the example shown in the figure, a coefficient that is a substantial part of the dictionary data is associated with the type of the object OB and the imaging condition.

  Further, the dictionary data 64 includes coefficients necessary for the subsequent image processing unit 54 to execute a convolution operation in order to acquire an image subjected to super-resolution processing. The dictionary data 64 is generated, for example, by learning a combination of high resolution data and low resolution data, which are a large number of correct answer data, using a technique such as deep learning. The post-stage image processing unit 54 performs a convolution operation on the actually acquired image using the dictionary data 64 generated by this learning, and acquires a high-resolution image (enlarged image). There are various learning type super-resolution processes, and the monitoring device 50 may be realized by using other known methods.

  Next, the post-stage image processing unit 54 causes the display unit 40 to display the super-resolution image acquired in step S104 (step S106).

  Next, the post-stage image processing unit 54 performs super-resolution processing using the dictionary data (2) (step S108). The post-stage image processing unit 54 performs super-resolution processing using the dictionary data (2), similarly to the processing in step S102 described above. Next, the post-stage image processing unit 54 causes the display unit 40 to display the super-resolution image acquired in step S108 (step S110).

  In this way, super-resolution images are displayed while sequentially changing the dictionary data. The post-stage image processing unit 54 executes super-resolution processing using the dictionary data (n) (step S112), and displays the super-resolution image acquired in step S112 on the display unit 40 (step S114). FIG. 6 is a diagram illustrating an example of a state in which super-resolution processing is performed while sequentially changing the dictionary data 64, and images IM (1) to (4) based on each dictionary data 64 are displayed on the display unit 40. is there.

  Next, the post-stage image processing unit 54 determines whether or not an image selection operation has been performed by the user (step S116). The user visually recognizes images sequentially displayed on the display unit 40, for example, and selects an image (for example, IM (3) in FIG. 6) that the object OB can clearly recognize. On the other hand, when there is no image that can be clearly recognized by the object OB, and the user has not performed an image selection operation, one routine of this flowchart ends, and the processing from step S100 is executed again. The

  When an image selection operation is performed by the user, the post-stage image processing unit 54 acquires an image captured by the imaging unit 10 (step S118). Next, the post-stage image processing unit 54 performs super-resolution processing by reconstruction on the image acquired in step S118 in the same manner as described above (step S120), and corresponds to the image selected in step S116. Super-resolution processing is performed using the dictionary data (k) to be performed (step S122).

  Next, the post-stage image processing unit 54 causes the display unit 40 to display the image acquired in step S122 (step S124). Next, the post-stage image processing unit 54 determines whether or not a reset operation has been performed by the user (step S126). If the reset operation has not been performed by the user, the process returns to the process of step S118 of this flowchart, and the process of displaying the super-resolution processed image on the display unit 40 is repeated. When the reset operation is performed by the user, the process of one routine of this flowchart is completed.

  Through the above-described processing, the monitoring system 1 can acquire an image of the object OB that can be clearly seen, and can monitor a plurality of objects OB in parallel. In addition, the monitoring system 1 tracks the target object with higher accuracy by adding the information acquired from the radar device 20 or the sensor 30 to the determination of the window W in the image captured by the imaging unit 10, and the tracked target A window W containing the object can be determined.

  FIG. 7 is a diagram illustrating another example in which an image is displayed on the display unit 40. The post-stage image processing unit 54 may collectively display images acquired by the super-resolution processing on the display unit 40. The post-stage image processing unit 54 sets the priority of a high-definition image to a high level, arranges the images subjected to the super-resolution processing based on the set priority, or sequentially displays the images on the display unit 40. May be.

  In the present embodiment, the monitoring system 1 performs super-resolution processing on each object OB using the dictionary data 64. However, the monitoring system 1 performs super-resolution processing using the dictionary data 64 or the super-structure obtained by reconstruction in step S102. The resolved image may be displayed on the display unit 40 and the user may select the dictionary data 64 used for the super-resolution process.

  In the present embodiment, the monitoring system 1 executes super-resolution processing using the dictionary data 64 corresponding to the selected image according to the user's selection in step S116. However, the index such as the sharpness of the image is used. Based on the above, the dictionary data 64 used for the super-resolution processing may be automatically selected.

  The monitoring system 1 determines the type of the object associated with the dictionary data 64 used when the image with the highest definition is acquired as the type of the object in the image, and displays the determination result on the display unit. 40 may be displayed.

  According to the first embodiment described above, the post-stage image processing unit 54 performs super-resolution processing on the image captured by the imaging unit 10 using any of the plurality of dictionary data 64. Is displayed on the display unit 40, it is possible to monitor more accurately in a scene where a plurality of types of objects OB may appear.

(Modification of the first embodiment)
Hereinafter, modifications of the first embodiment will be described. In the monitoring system 1 according to the modified example of the first embodiment, pattern matching is performed on an image acquired by super-resolution processing using the dictionary data 64. Hereinafter, a description will be given focusing on differences from the first embodiment.

  In addition to the image data 62 and the dictionary data 64, the storage unit 60 according to the modification of the first embodiment further corresponds to a plurality of types of objects OB in advance (which may also correspond to imaging conditions). A device dictionary is stored. The latter-stage image processing unit 54 functioning as a discriminator includes feature quantities such as edge strength and direction acquired by the Sobel filter processing on the image subjected to the super-resolution processing, and a dictionary used in the super-resolution processing. The similarity with the pattern discriminator dictionary corresponding to the type of the data 64 is calculated, and the probability of the type of the dictionary data 64 is determined.

  FIG. 8 is a flowchart illustrating a part of the flow of processing executed by the image processing apparatus 50 according to the modification of the first embodiment. The processing from step S200 to step S206 in this flowchart is processing executed after one dictionary data 64 is selected by the image processing apparatus 50 and super-resolution processing is executed. Therefore, the process of the flowchart of FIG. 8 is executed for each type of dictionary data 64 in accordance with the sequential selection of the dictionary data 64.

  The post-stage image processing unit 54 acquires an image that has been super-resolved using the selected dictionary data 64, and displays the acquired image on the display unit 40 (step S200).

  Next, the latter-stage image processing unit 54 includes a feature amount (described above) obtained from the image in the window among the images displayed on the display unit 40, and a pattern classifier dictionary corresponding to the type of the selected dictionary data 64. (Step S202), and it is determined whether or not the type of the object OB included in the window matches the type of the object OB corresponding to the selected dictionary data 64 (step S204). . Specifically, the latter-stage image processing unit 54 determines that these match when the similarity calculated in step S202 is high. If they do not match, step S206 is skipped.

  If the type of the object OB included in the window matches the type of the object OB corresponding to the selected dictionary data 64, the subsequent image processing unit 54 fixes the dictionary data 64 (step S206). Thereby, the image processing apparatus 50 can automatically select the dictionary data 64 suitable for the super-resolution processing of the object OB.

  The monitoring system 1 according to the modified example of the first embodiment described above includes an image on which super-resolution processing has been performed and a pattern classifier dictionary corresponding to the type of dictionary data 64 selected in the super-resolution processing. Matching is automatically performed, and dictionary data 64 suitable for the object OB is automatically selected based on the result of matching. As a result, it is possible to provide the image processing apparatus 50 that has the same effects as those of the first embodiment and is highly convenient for the user.

  According to at least one embodiment described above, the imaging unit 10, the storage unit 60 that stores a plurality of different dictionary data 64, and the image captured by the imaging unit 10 are stored in the storage unit 60. The post-stage image processing unit 54 that performs super-resolution processing using any one of the dictionary data 64 including coefficients for realizing a plurality of super-resolution processes, and the post-stage image processing unit 54 performs the super-resolution processing. By having the display unit 40 that displays the displayed image, it is possible to monitor more accurately in a scene where a plurality of types of objects OB may appear.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Monitoring system, 10 ... Imaging part, 20 ... Radar apparatus, 30 ... Sensor, 40 ... Display part, 42 ... Input device, 50 ... Image processing apparatus, 52 ... Pre-stage image processing part, 54 ... Post-stage image processing part, 60 ... Storage unit, 62 ... Image data, 64 ... Dictionary data

Claims (10)

An imaging unit;
A storage unit for storing dictionary data including coefficients for realizing a plurality of different super-resolution processes;
An image analysis unit that performs super-resolution processing on the image captured by the imaging unit using any of a plurality of dictionary data stored in the storage unit;
A display unit for displaying an image subjected to super-resolution processing by the image analysis unit;
A monitoring system comprising: The plurality of dictionary data are dictionary data for each type of object that can be imaged by the imaging unit.
The monitoring system according to claim 1. A reception unit that receives an operation of selecting a type of the object by a user;
The image analysis unit performs super-resolution processing using dictionary data corresponding to the type of object according to the operation received by the reception unit.
The monitoring system according to claim 2. The plurality of dictionary data is dictionary data for each imaging condition.
The monitoring system according to any one of claims 1 to 3. The image analysis unit sequentially selects the plurality of dictionary data, and sequentially displays the images subjected to super-resolution processing using the selected dictionary data on the display unit.
The monitoring system according to any one of claims 1 to 4. A pre-stage image processing unit that performs super-resolution processing by reconstruction on the time-series images captured by the imaging unit;
The image analysis unit performs super-resolution processing using the plurality of dictionary data on the image that has been subjected to super-resolution processing by reconstruction by the pre-stage image processing unit.
The monitoring system according to any one of claims 1 to 5. A position change detection unit for detecting a position change of the imaging unit;
The pre-stage image processing unit performs super-resolution processing by reconstruction based on the position change of the imaging unit detected by the position change detection unit.
The monitoring system according to claim 6. A radar unit that detects the position of an object is provided.
The pre-stage image processing unit performs super-resolution processing by reconstruction based on the position change of the object detected by the radar unit.
The monitoring system according to claim 6 or 7. Computer
Acquire the image acquired by the imaging unit,
Super-resolution processing is performed on the acquired image using any of a plurality of dictionary data stored in a storage unit that stores dictionary data including coefficients for realizing a plurality of different super-resolution processing. Done
Displaying the image subjected to the super-resolution processing on a display unit;
Monitoring method. On the computer,
The image acquired by the imaging unit is acquired,
Super-resolution processing is performed on the acquired image using any of a plurality of dictionary data stored in a storage unit that stores dictionary data including coefficients for realizing a plurality of different super-resolution processing. Let
Causing the display unit to display the image subjected to the super-resolution processing;
Monitoring program.

Images