KR20160072295A - Device and method for generating compound image - Google Patents

Abstract

The present invention relates to a device and a method for generating a compound image. Disclosed is the device which includes a sonar image object detecting part which detects an object in a sonar image; and a compound image generating part which generates a compound image by synthesizing the object detected in the sonar image with an optical image. The compound image generating part synthesizes the object in the image region of the optical image corresponding to the position information of the object and generates a compound image.

Description

TECHNICAL FIELD [0001] The present invention relates to a composite image generation apparatus and a composite image generation method,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for generating a composite image, and more particularly, to an apparatus and method for generating a compound image by synthesizing a SONAR image with an optical image .

Underwater Sonar is an apparatus that generates ultrasonic waves and visualizes the front using the time taken for the sound waves reflected from the object to come back and the intensity of the reflected sound waves. Because of the short visibility distance in the case of optical cameras, underwater ROV (remote operated vehicle) or autonomous underwater vehicle (AUV) are mainly detecting imaging obstacles using imaging SONAR. An imaging sonar emits ultrasonic waves of different frequencies at the same time, receives reflected waves reflected from the object, and generates a sonar image. The imaging sonar has the advantage of being able to quickly generate sonar images for the measurement area. However, sonar images have disadvantages in that they can not provide high resolution images compared with optical images.

The underwater optical camera has an advantage that it can provide color information and can provide a high-resolution image required for close-range precision work, but has a disadvantage in that the visibility deteriorates remarkably in the water than in the ground. Moreover, the viewing distance of an optical camera is about 10 m in a clean underwater condition, but it is often not secured in a 1 m in an environment in which floating matters such as the ocean are present. Therefore, it is difficult to utilize underwater optical camera for ROV or AUV obstacle detection. Further, since the underwater optical camera has lost the distance information in the direction of the optical axis, unlike the sonar image, it has a problem that it is difficult to know the distance information of the object.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a composite image generating apparatus and method which can obtain an effect corresponding to the extension of the visual range of an optical imaging apparatus.

Another object to be solved by the present invention is to provide a composite image generation method capable of easily grasping the existence of an object such as a work object or an obstacle through a composite image (composite image) without confusion due to the recognition of the optical image and the sonar image separately And to provide a method and apparatus.

The problems to be solved by the present invention are not limited to the above-mentioned problems. Other technical subjects not mentioned will be apparent to those skilled in the art from the description below.

According to one aspect of the present invention, there is provided a composite image generation apparatus comprising: a sonar image object detection unit for detecting an object in a sonar image; And a composite image generation unit for generating a composite image by synthesizing an object detected from the sonar image with an optical image, wherein the composite image generation unit synthesizes the object in an image region of an optical image corresponding to the positional information of the object, And generates the composite image.

The composite image generation unit may generate the composite image so that the object detected in the sonar image is displayed in a translucent manner on the optical image.

Wherein the composite image generation apparatus comprises: an optical imaging apparatus for acquiring the optical image on a first region; An imaging unit for acquiring the sonar image on a second area including at least a portion of the first area; And a coordinate system matching unit for matching the coordinate system of the optical imaging apparatus and the imaging sonar.

Wherein the sonar image object detection unit comprises: an object extraction unit for extracting an object region based on brightness information of a pixel in the sonar image; A distance calculating unit for calculating distance information to the object; A width calculating unit for calculating horizontal direction width information of the object from the sonar image; And a height calculation unit for detecting a shadow area for the object area in the sonar image and calculating height information of the object using the shadow area information.

The composite image generation unit may generate the composite image by transforming the width and height of the object according to the distance information of the object detected from the sonar image and the internal parameter information of the optical imaging apparatus and projecting the object on the optical image .

An object located in an area outside the visible range of the optical imaging apparatus may be displayed on the optical image.

According to another aspect of the present invention, there is provided a method of detecting an object in a sonar image, And generating a composite image by synthesizing an object detected in the sonar image with an optical image, wherein the step of generating the composite image includes synthesizing the object in an image region of an optical image corresponding to the positional information of the object, Thereby generating the composite image.

The detecting the object may include extracting an object region based on pixel brightness information in the sonar image; Calculating the distance information to the extracted object and the width information of the object in the horizontal direction; And detecting a shadow area for the object area in the sonar image and calculating height information of the object using the shadow area information.

The generating of the composite image may include converting the width and height of the object according to distance information from the sonar image to the detected object and internal parameter information of the optical imaging apparatus that has obtained the optical image, The composite image can be generated.

According to an embodiment of the present invention, an object located in an area beyond the visible range of the optical imaging apparatus can be synthesized into an optical image and displayed as a composite image. Therefore, it is possible to obtain an effect corresponding to the extension of the visual distance of the optical image, and it is possible to easily grasp the position of the object through one composite image (composite image) without confusion by recognizing the optical image and the sonar image separately Control of an underwater robot or the like can be performed.

The effects of the present invention are not limited to the effects described above. Unless stated, the effects will be apparent to those skilled in the art from the description and the accompanying drawings.

1 is a block diagram of a composite image generating apparatus according to an embodiment of the present invention.
FIG. 2 is a block diagram of a sonar image object detecting unit included in the composite image generating apparatus according to an embodiment of the present invention. Referring to FIG.
3 is a flowchart of a composite image generation method according to an embodiment of the present invention.
4 is a diagram schematically illustrating acquisition of an optical image and a sonar image using an optical imaging apparatus and an imaging sonar according to a composite image generating method according to an embodiment of the present invention.
5 is a diagram illustrating an example of a sonar image for explaining an object detection in a sonar image according to a composite image generating method according to an embodiment of the present invention.
6 is a side view schematically showing acquisition of a two-dimensional sonar image for objects using an imaging sonar;
FIG. 7 is an exemplary diagram of two-dimensional sonar image data for explaining step S30 shown in FIG. 3; FIG.
8 is an exemplary view showing an optical image obtained by an optical imaging apparatus.
9 is a diagram illustrating an example of a composite image generated by a composite image generation method according to an embodiment of the present invention.

Other advantages and features of the present invention and methods for accomplishing the same will be apparent from the following detailed description of embodiments thereof taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and the present invention is only defined by the scope of the claims. Although not defined, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by the generic art in the prior art to which this invention belongs. A general description of known configurations may be omitted so as not to obscure the gist of the present invention. In the drawings of the present invention, the same reference numerals are used as many as possible for the same or corresponding configurations. To facilitate understanding of the present invention, some configurations in the figures may be shown somewhat exaggerated or reduced.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises", "having", or "having" are intended to specify the presence of stated features, integers, steps, operations, components, Steps, operations, elements, parts, or combinations thereof, whether or not explicitly described or implied by the accompanying claims.

Used throughout this specification may refer to a hardware component such as, for example, software, FPGA or ASIC, as a unit for processing at least one function or operation. However, "to" is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors.

As an example, the term '~' includes components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, procedures, Routines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided by the components and components may be performed separately from the plurality of components and components, or may be integrated with other additional components.

A composite image generation apparatus according to an embodiment of the present invention generates a composite image (fusion image) by detecting an object in a sonar image and synthesizing the detected object in the sonar image to be fused with the optical image. The object may be synthesized on the area of the optical image corresponding to the position of the object and displayed as a composite image. The width and height of the object in the horizontal direction can be transformed according to the distance information of the object calculated by the sonar image and the internal parameters of the optical imaging apparatus obtained from the optical image and synthesized into the optical image.

According to the embodiment of the present invention, an object that is out of the visible range of the optical image is detected by the sonar image and displayed on the optical image, thereby obtaining the effect of expanding the visual distance of the optical image. In addition, according to the embodiment of the present invention, it is possible to easily grasp the position of an object through one composite image (composite image) without disturbing the optical image and the sonar image separately, and control the underwater robot or the like .

1 is a configuration diagram of a composite image generation apparatus 100 according to an embodiment of the present invention. Referring to FIG. 1, a composite image generation apparatus 100 includes a coordinate system matching unit 110, a sonar image object detection unit 120, a composite image generation unit 130, and an output unit 140. The composite image generation apparatus 100 generates a composite image from the optical image acquired by the optical imaging apparatus 10 and the sonar image acquired by the imaging sonar 20.

The optical imaging device 10 can acquire an optical image on the first area. The imaging sonar 20 may acquire a sonar image on a second area including at least a part of the first area. In one embodiment, the optical imaging device 10 and the imaging sonar 20 may be combined to obtain an optical image and a sonar image in the same direction. The optical imaging device 10 may include, for example, a CCD camera or a CMOS camera.

In one embodiment, the imaging sonar 20 can simultaneously emit ultrasonic waves of different frequencies and receive reflected waves that are reflected back from the object to generate a two-dimensional sonar image. The imaging sonar 20 can generate a two-dimensional sonar image using the time taken for the ultrasonic waves to be reflected from the object and returned, and the intensity of the reflected ultrasonic waves. Two-dimensional sonar images contain distance and azimuthal information to objects (objects)

The optical imaging device 10 acquires a two-dimensional optical image, and the imaging sonar 20 acquires a two-dimensional sonar image. However, the composite image generation apparatus according to the embodiment of the present invention can also generate a composite image by synthesizing a three-dimensional optical image with a three-dimensional sonar image. For example, the three-dimensional optical image can be generated by matching a plurality of optical images obtained using a plurality of cameras. For example, a three-dimensional sonar image can be obtained by measuring the height of an object detected in a two-dimensional sonar image and measuring three-dimensional information of the object.

The coordinate system matching unit 110 matches the coordinate system of the optical imaging device 10 and the imaging sonar 20. Coordinate system matching can be performed through calibration. For example, an optical image and a sonar image for common targets are acquired by the optical imaging device 10 and the imaging sonar 20, and the image positions of common targets are compared between the optical image and the sonar image, Relationship can be obtained. At least one of the optical image obtained by the optical imaging device 10 and the imaging sonar 20 and the sonar image can be transformed to have a common coordinate system by coordinate system matching.

Before detecting an object in a sonar image, the noise of the sonar image can be removed by using an image processing technique such as a Gaussian filter. The sonar image object detection unit 120 detects an object in the sonar image. 2 is a configuration diagram of a sonar image object detection unit 120 constituting the composite image generation apparatus 100 according to an embodiment of the present invention. 2, the sonar image object detection unit 120 includes an object extraction unit 121, a distance calculation unit 122, a width calculation unit 123, and a height calculation unit 124.

The object extracting unit 121 extracts an object based on pixel brightness information in a sonar image. That is, it is possible to extract an object (object) by extracting a pixel having a predetermined brightness or more from the sonar image, and to divide the area of the object using a segmentation image processing technique or the like. The distance calculator 122 analyzes the sonar image data and calculates distance information to the object. The width calculating unit 123 calculates the horizontal width information of the object from the sonar image. The height calculating unit 124 detects a shadow area behind the object area corresponding to the object in the sonar image and calculates the height of the shadow area using the detected distance information of the imaging sonar 20 and the distance to the object And calculates the height information of the object.

The composite image generation unit 130 generates a composite image by combining the objects detected in the sonar image by the sonar image object detection unit 120 with a specific image region of the optical image. The object detected in the sonar image can be converted to have the same shape as that directly photographed by the optical imaging device 10 and synthesized into the optical image. The composite image generation unit 130 may generate a composite image such that the object detected from the sonar image is displayed in a translucent manner on the optical image.

The composite image generation unit 130 converts the width and height of the object according to the internal parameter information such as the distance information of the object detected from the sonar image and the focal length of the camera lens of the optical imaging apparatus 10, ) To generate a composite image. The output unit 140 outputs the composite image generated by the composite image generation unit 130 through a display screen or the like.

According to the embodiment of the present invention, an object located in an area outside the visible range of the optical imaging apparatus 10 can be synthesized and displayed on an optical image. Therefore, an effect corresponding to the extension of the visual distance of the optical image can be obtained. In addition, according to the embodiment of the present invention, it is possible to easily grasp the position of an object through one composite image (composite image) without disturbing the optical image and the sonar image separately, and control the underwater robot or the like .

3 is a flowchart of a composite image generation method according to an embodiment of the present invention. 1 and 3, the coordinate system matching unit 110 first aligns the coordinate system of the optical imaging apparatus 10 and the imaging sonar 20 through calibration (S10) , A two-dimensional optical image and a two-dimensional sonar image are acquired using the optical imaging device 10 and the imaging sonar 20 (S20).

FIG. 4 is a view schematically showing acquisition of an optical image and a sonar image using the optical imaging device 10 and the imaging sonar 20 according to the composite image generating method according to the embodiment of the present invention. The optical imaging device 10 and the imaging sonar 20 may be arranged to be close to each other. The relative positions of the optical imaging device 10 and the imaging sonar 20 can be fixed. The optical imaging device 10 may have a shorter viewing distance than the imaging sonar 20. The optical imaging device 10 can acquire an optical image for the first region 11. The imaging sonar 20 may be disposed in a position and direction capable of acquiring the sonar image with respect to the second area 21 including at least a part of the first area 11. [

Referring back to FIG. 3, after the sonar image is acquired, the sonar image object detection unit 120 detects the objects 30 to 34 in the sonar image (S30). 5 is a diagram illustrating an example of a sonar image for explaining an object detection in a sonar image according to a composite image generating method according to an embodiment of the present invention. The sonar image object detection unit 120 extracts the objects 31 to 34 based on the brightness information of the pixels in the sonar image, analyzes the sonar image data, and calculates the distance information D to the object , Calculates the horizontal width information W of the object from the sonar image, detects the shadow area behind the object area corresponding to the object in the sonar image, calculates the length information H of the shadow area and the distance information D ), The inclination angle of the imaging sonar 20, and the like.

FIG. 6 is a side view schematically showing acquisition of a two-dimensional sonar image for the objects 32 to 33 using the imaging sonar 20, and FIG. 7 is a side view for explaining the step S30 shown in FIG. And Fig. 1, 3, 6, and 7, the sonar image object detection unit 120 detects

An area of the two-dimensional sonar image data whose brightness is equal to or greater than the first reference value T ₀ is detected as the object area OA and the first reference value T ₀ is set adjacent to the object area OA on the rear side of the object area OA, ) it is set to be no greater than is possible to detect a region having a brightness of less than second reference value (T _S) to the shadow area (SA) for the object area (OA) (S20).

The sonar image object detection unit 120 can detect the shadow area SA by the progress direction (horizontal azimuth angle) of the sound waves of the imaging sonar 20. An area other than the object area (OA) and the shadow area (SA) is detected as the bottom area (BA). 7, the distance between the end point Send of the shadow area SA from the starting point of the shadow area SA is calculated as the distance information of the shadow area SA. The starting point of the shadow area (SA) can coincide with the end point (Oend) of the object area (OA).

When the shadow area SA is detected in the two-dimensional sonar image, the sonar image object detection unit 120 detects the length of the detected shadow area SA, the inclination angle φ of the imaging sonar 20, The height information of the objects 32 and 33 is calculated using information such as the distance to the object. Since the length of the shadow area SA is proportional to the height of the objects 32 and 33, height information of the objects 32 and 33 can be calculated from the length information of the shadow area SA. For example, the sonar image object detection unit 120 may calculate the object height information z ₀ according to Equation 1 below.

[Formula 1]

Where _o is the height information of the object,? Is the vertical tilt angle of the imaging sonar 20,? Is the horizontal sonic angle of the imaging sonar 20, k is the proportional constant, Ostart is the starting point of the object area , Ostart is the starting point of the object region, that is, the point at which the intensity of the 2D image data is higher than the first reference value (T ₀ ), Intensity _i is the brightness of the object region, Oend is 2 The end point of the object region where the brightness is lower than the first reference value T ₀ in the dimension sonar image data, Send is the point where the brightness is higher than the second reference value T _S in the two-dimensional sonar image data after the end point Oend of the object region, , k is a proportional constant, and R is the distance to the object, which represents the height measurement point for the object.

As the inclination angle (height increase rate) of the object is increased, the intensity of the ultrasonic wave reflected from the object is increased and the brightness value is increased in the two-dimensional ultrasonic image data. On the contrary, The intensity of the reflected ultrasound waves is reduced, so that the brightness value is reduced in the two-dimensional sonar image data. Therefore, the height of the object with respect to the measurement point can be expressed by the integral value of the intensity value of the two-dimensional sonar image data from the starting point (Ostart) of the object region to the height measurement point (R) That is, the cumulative sum of the height increasing rate of each object position).

The composite image generation unit 130 generates a composite image by combining objects in an optical image region corresponding to the position information of the object. The composite image generation unit 130 converts the size of the object according to the width, height, and distance information of the object detected from the sonar image by the sonar image object detection unit 120, and synthesizes the image with the optical image, (S40).

The composite image generation unit 130 may generate a composite image such that the object detected from the sonar image is displayed in a translucent manner on the optical image. The composite image generation unit 130 converts the width and the height of the object according to the distance information of the object detected from the sonar image and the internal parameter information of the optical imaging apparatus 10 So that a composite image can be generated.

In an embodiment, the composite image generation unit 130 synthesizes only the object of the sonar image that is out of the visible range of the optical imaging device 10 into an optical image, and for the object located within the visible range of the optical imaging device 10, It is possible not to combine the object information of the object with the optical image. This is because the object located within the visible range of the optical imaging apparatus 10 already appears in the optical image, and therefore, when the object detected from the sonar image is further synthesized into the optical image, the sense of confusion may be increased.

FIG. 8 is an exemplary view showing an optical image 12 obtained by the optical imaging apparatus 10. As shown in FIG. 4 and 8, a forward object located within the visible range of the optical imaging device 10 is shown in the optical image 12, but the backward objects 31 to 34, which deviate from the viewing distance of the optical imaging device 10, Is not displayed in the optical image 12. FIG. 9 is an exemplary view showing a composite image 13 generated by the composite image generation method according to the embodiment of the present invention.

4 and 9, the rearward objects 31 to 34 can be detected from the sonar image acquired by the imaging sonar 20 having a large visual distance, and the rearward objects 31 to 34 detected from the sonar image can be detected. 34 are synthesized to the optical image 12 to generate the composite image 13 so that the rear objects 31 to 34 that can not be photographed by the optical imaging device 10 can be confirmed through the composite image 13 have. In the composite image 13, the rear objects 31 to 34 may be displayed in a semi-transparent form or in a color different from the objects of the optical image. The composite image 130 may additionally display information displays 41 to 44 indicating information L1 to L4 such as distance values from the sonar image to the rear objects 31 to 34 that can be detected.

According to the embodiment of the present invention, an object located in an area outside the visible range of the optical imaging apparatus 10 can be synthesized and displayed on an optical image. Therefore, it is possible to obtain an effect corresponding to the extension of the visual distance of the optical image, and it is possible to obtain an object such as a work object or an obstacle through a composite image (composite image) without confusion by recognizing the optical image and the sonar image individually Can be easily grasped and the underwater robot can be adjusted.

The method according to an embodiment of the present invention can be realized in a general-purpose digital computer that can be created as a program that can be executed by a computer and operates the program using a computer-readable recording medium. The computer readable recording medium may be a volatile memory such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Electrically Programmable ROM) Non-volatile memory such as EEPROM (Electrically Erasable and Programmable ROM), flash memory device, Phase-change RAM (PRAM), Magnetic RAM (MRAM), Resistive RAM (RRAM), Ferroelectric RAM But are not limited to, optical storage media such as CD ROMs, DVDs, and the like.

It is to be understood that the above-described embodiments are provided to facilitate understanding of the present invention, and do not limit the scope of the present invention, and it is to be understood that various modifications are possible within the scope of the present invention. It is to be understood that the technical scope of the present invention should be determined by the technical idea of the claims and the technical scope of protection of the present invention is not limited to the literary description of the claims, To the invention of the invention.

10: optical imaging device 20: imaging sonar
30, 31, 32, 33, 34: object (object) 100:
110: coordinate system matching unit 120: sonar image object detection unit
121: object extracting unit 122: distance calculating unit
123: width calculating section 124: height calculating section
130: Composite image generator 140:

Claims (7)

A sonar image object detector for detecting an object in a sonar image; And
And a composite image generation unit for generating a composite image by synthesizing the object detected from the sonar image with an optical image,
Wherein the composite image generation unit generates the composite image by synthesizing the object in an image region of an optical image corresponding to the position information of the object. The method according to claim 1,
Wherein the composite image generation unit generates the composite image so that the object detected in the sonar image is displayed in a translucent manner on the optical image. The method according to claim 1,
An optical imaging device for acquiring the optical image on a first area;
An imaging unit for acquiring the sonar image on a second area including at least a portion of the first area; And
And a coordinate system matching unit for matching the coordinate system of the optical imaging apparatus and the imaging sonar. The method of claim 3,
Wherein the sonar image object detecting unit comprises:
An object extraction unit for extracting an object region based on pixel brightness information in the sonar image;
A distance calculating unit for calculating distance information to the object;
A width calculating unit for calculating horizontal direction width information of the object from the sonar image; And
And a height calculation unit for detecting a shadow area for the object area in the sonar image and calculating height information of the object using the information of the shadow area. 5. The method of claim 4,
Wherein the composite image generation unit comprises:
And converting the width and height of the object according to the distance information to the object detected from the sonar image and the internal parameter information of the optical imaging device and projecting the object on the optical image. 6. The method of claim 5,
Wherein an object located in an area out of the visible range of the optical imaging device is synthesized and displayed on the optical image. Detecting an object in the sonar image; And
And synthesizing the object detected in the sonar image with an optical image to generate a composite image,
Wherein the step of generating the composite image comprises: synthesizing the object in an image region of an optical image corresponding to the position information of the object to generate the composite image.

Images