KR20210137845A - Method and apparatus for stabilization of drone-projected image - Google Patents

Abstract

The present invention relates to a method and device for stabilizing a projection image of a drone-mounted projector. The method for stabilizing a projection image of a drone-mounted projector in the present invention includes: a stabilization model estimation unit estimating a stabilization model for correcting the distortion of a projection image based on flight information, when the flight information of the drone on which the projector is mounted is received; an image conversion unit converting the projection image based on the estimated stabilization model; and an image transmission unit transmitting the converted image toward the projection target.

Description

Method and apparatus for stabilizing the projection image of a drone-mounted projector

The present invention relates to a method and apparatus for stabilizing a projection image of a drone-mounted projector.

A drone-mounted projector refers to a system in which a beam projector is mounted on a drone without a camera. A drone-mounted projector mainly projects an image on the outer wall of a high-rise building, thus serving as a movable digital signage display.

However, unlike conventional fixed projectors, drone-mounted projectors can distort the projected image even when the drone is in stationary flight due to vibrations from the drone's motor or changes in the position and attitude of the drone due to drone flight control errors.

In order to solve this problem, research on stabilizing the projection image using a camera has been conducted. For example, stabilization of the projected image may be performed by attaching a camera to a drone, capturing an undistorted projected image and a distorted projected image with a camera to obtain a coordinate pair. However, since this method requires a coded visual marker that is the standard for image correction, stabilization of the projected image can be performed only in a limited space where the visual marker is attached, and the projected image must be acquired by shooting with a camera. It can increase complexity.

In the present invention, in order to perform stabilization of the projected image, it is necessary to acquire attitude information or location information of the drone-mounted projector. The attitude information of the drone-mounted projector can be obtained from the Inertial Measurement Unit (IMU) (or the Attitude and Heading Reference (AHRS) System)) can be obtained. However, when the location information of the drone-mounted projector is extracted through the accelerometer of the inertial measurement device (or posture and orientation device), it may be difficult to extract accurate location information because integration errors are accumulated.

In addition, when an image of a drone-mounted projector in a three-dimensional space is projected as a two-dimensional image, an image tilt, keystone phenomenon, or image movement may occur depending on the posture of the drone-mounted projector. Similarly, scaling and movement of the image may occur depending on the location of the drone-mounted projector.

Therefore, research on an image stabilization apparatus and method for a drone-mounted projector other than the above-described method is in progress.

The present invention discloses a method and apparatus for stabilizing a projection image in consideration of distortion of a projection image of a drone-mounted projector based on flight information of a drone.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and another problem(s) not mentioned will be clearly understood by those skilled in the art from the following description.

The present invention relates to a method and apparatus for stabilizing a projection image of a drone-mounted projector, and the method for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention includes the steps of receiving flight information of a drone equipped with a projector , estimating a stabilization model for correcting the distortion of the projected image based on the received flight information, converting the projected image based on the estimated stabilization model, and transmitting the converted image toward the projection target. can do.

Preferably, the step of receiving the flight information of the drone may be receiving the flight information of the drone from a sensor driving unit inside the drone.

Preferably, the method of stabilizing the projection image of the drone-mounted projector further comprises the step of sensing flight information of the drone by a sensor driving unit, and the step of receiving the flight information of the drone is to receive the sensed flight information of the drone can

Preferably, the flight information of the drone may include at least one of a distance between the drone and a projection target, an altitude of the drone, and a rotation angle of the drone.

Preferably, the resolution of the transmitted image may be lower than the maximum image resolution that the projector can transmit.

Preferably, the stabilization model may be decomposed into at least one of an internal parameter matrix, a distortion image correction matrix according to the movement of the drone, and a distortion image correction matrix according to the rotation of the drone.

Preferably, the internal parameter matrix may vary according to at least one of the focal length of the projector and the coordinates of the center point of the transmitted image.

Preferably, when it is determined through the received flight information that the drone has moved on the X-axis, which is the direction of travel, the distortion image correction matrix according to the movement of the drone is the projection angle, the initial set distance between the drone and the projection target, and the distance between the drone and the projection target. It may depend on at least one of the current distances.

Preferably, when it is determined through the received flight information that the drone has moved on the Z-axis in the height direction, the distortion image correction matrix according to the movement of the drone may vary depending on the altitude information of the drone.

Preferably, when it is determined through the received flight information that the drone rotates on the roll axis in the traveling direction, the distortion image correction matrix according to the rotation of the drone is at least one of the roll axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. may vary depending on

Preferably, when it is determined that the drone rotates on the pitch axis in the height direction through the received flight information, the distortion image correction matrix according to the rotation of the drone is one of the pitch axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. It may depend on at least one.

Preferably, when it is determined through the received flight information that the drone rotates on the yaw axis in the left and right direction, the distortion image correction matrix according to the rotation of the drone varies depending on at least one of the yaw axis rotation angle and the initial set distance between the drone and the projection target. can

Preferably, the step of transforming the image includes performing a Hadamard product operation of the estimated stabilization model and a unit conversion matrix, and the operation result of the Hadamard product is applied to the projected image in units of pixels. , the unit transformation matrix may vary according to at least one of a projection angle, a current distance between the drone and a projection target, and a vertical resolution of the transmitted image.

In accordance with another embodiment of the present invention, the projection image stabilization apparatus of a drone-mounted projector is stabilizing for estimating a stabilization model for correcting the distortion of the projection image based on the flight information when flight information of the drone equipped with the projector is received It may include a model estimator, an image converter that converts a projected image based on the estimated stabilization model, and an image transmitter that transmits the converted image toward a projection target.

Preferably, it further comprises a sensor driver for transmitting the flight information of the drone to the stabilization model estimator, and the flight information of the drone may include at least one of a distance between the drone and a projection target, an altitude of the drone, and a rotation angle of the drone. .

Preferably, the resolution of the image transmitted by the image transmitting unit may be lower than the maximum image resolution that the projector can transmit.

The present invention provides a computer-readable storage medium storing a computer program in which the above-described method is performed when the computer program is executed by a processor.

Since the projection image stabilization apparatus of the drone-mounted projector according to an embodiment of the present invention corrects the distortion of the projection image based on the flight information of the drone, there is an effect of improving the correction performance of the projection image.

Since the apparatus for stabilizing the projection image of a drone-mounted projector according to an embodiment of the present invention does not require a separate camera, it is compared with the apparatus for stabilizing the projection image using a camera. can be solved, and space complexity can be reduced.

On the other hand, the effects of the present invention are not limited to the above-mentioned effects, and various effects may be included within the range apparent to those skilled in the art from the contents to be described below.

1 is a view for explaining an apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.
2 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.
3 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to another embodiment of the present invention.
4 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to another embodiment of the present invention.
5 is a diagram for explaining the structure of a projection image of a drone-mounted projector according to an embodiment of the present invention.
6 is a view for explaining a coordinate system used in a method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.
7 is a diagram for explaining a distortion type of a projected image according to an embodiment of the present invention.
8 is a view for explaining a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.
9 is a diagram for explaining a method of correcting distortion of a projected image due to movement of a drone-mounted projector according to an embodiment of the present invention.
10 is a view for explaining the movement coefficient according to the roll axis rotation of the projection image stabilization model of the drone-mounted projector according to an embodiment of the present invention.
11 is a view for explaining the movement coefficient according to the pitch axis rotation of the projection image stabilization model of the drone-mounted projector according to an embodiment of the present invention.
12 is a view for explaining a movement coefficient according to a yaw axis rotation of a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.
13 is a diagram for explaining a unit transformation matrix of a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.
14 is a view for explaining a performance test environment of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.
15 and 16 are diagrams for explaining the performance of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.
17 is a diagram for explaining the performance of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.
18 is a flowchart illustrating a method of stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining an apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.

The drone-mounted projector system according to an embodiment of the present invention refers to a system including the drone 110 and the projector 120 mounted on the drone 110 . Since the drone-mounted projector system can serve as a digital signage display capable of flying by projecting the image 130 on the outer wall of a high-rise building, it can be used as a display for information transfer or publicity.

However, the drone-mounted projector system may cause distortion of the projected image 130 due to vibrations generated from the motor of the drone 110 or a change in the position and posture of the drone 110 due to a drone 110 flight control error. can

In order to solve this problem, the method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention corrects the distortion of the projection image according to the flight pattern of the drone through flight information that can be obtained during the flight of the drone. can do. Therefore, using the method and apparatus disclosed in the present invention, it is possible to stabilize the projected image without using a separate equipment such as a camera.

2 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.

The apparatus 200 for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention may include a stabilization model estimator 210 , an image converter 220 , and an image transmitter 230 .

When the flight information of the drone equipped with the projector is received, the stabilization model estimator 210 may estimate a stabilization model for correcting the distortion of the projection image based on the flight information. Here, the flight information of the drone may include at least one of a distance between the drone and a projection target, an altitude of the drone, and a rotation angle of the drone.

Meanwhile, the stabilization model may be decomposed into at least one of an internal parameter matrix, that is, a matrix composed of constant parameters determined by the projector, a distortion image correction matrix according to the movement of the drone, and a distortion image correction matrix according to the rotation of the drone.

The internal parameter matrix may vary depending on at least one of the focal length of the projector and the coordinates of the center point of the transmitted image. Here, the projection angle is an internal parameter that is not included in the internal parameter matrix but is determined by the projector.

The movement and rotation of the drone is defined by three axes, and the movement of the drone is defined as the X-axis, the right direction relative to the drone, the Y-axis, and the lower side as the Z-axis. In the rotation of the drone, the X, Y, and Z axes of movement become the Roll, Pitch, and Yaw axes, and the direction in which the right-hand rule is applied based on the axis direction is the positive direction of the rotation angle.

In the present invention, in order to reduce additional calculations, the definition of the coordinate system of the drone is also used in the projection image.

When it is determined through the received flight information that the drone has moved on the X axis, which is the direction of travel, the distortion image correction matrix according to the movement of the drone is the projection angle, the initial set distance between the drone and the projection target, and the current distance between the drone and the projection target. It may depend on at least one.

When it is determined through the received flight information that the drone has moved on the Z-axis in the height direction, the distortion image correction matrix according to the movement of the drone may vary depending on the altitude information of the drone.

When it is determined through the received flight information that the drone has rotated on the roll axis in the direction of travel, the distortion image correction matrix according to the rotation of the drone will vary depending on at least one of the roll axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. can

When it is determined that the drone rotates on the pitch axis in the height direction through the received flight information, the distortion image correction matrix according to the rotation of the drone is at least one of the pitch axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. may vary depending on

When it is determined through the received flight information that the drone rotates on the yaw axis in the left and right direction, the distortion image correction matrix according to the rotation of the drone may vary depending on at least one of the yaw axis rotation angle and the initial set distance between the drone and the projection target.

Since the image stabilization model by the movement and rotation of the drone is defined independently of each other, when movement and rotation about each axis occur in combination, it can be expressed through matrix multiplication.

The image converter 220 may convert the projection image based on the estimated stabilization model.

In addition, the image converter 220 performs a Hadamard product of the unit transformation matrix on the stabilization model in order to convert the parameters of the stabilization model in pixel units. Here, the unit transformation matrix may vary according to at least one of a projection angle, a current distance between the drone and a projection target, and a vertical resolution of the transmitted image.

The image transmitting unit 230 may transmit the converted image toward the projection target. In this case, the resolution of the image transmitted by the image transmitting unit 230 may be lower than the maximum image resolution that the projector can transmit.

In the present invention, the higher the resolution of the converted image to stabilize the image, the longer the converted image transmission time. Therefore, the resolution of the image may be lowered in order to ensure the real-time of the present invention.

3 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to another embodiment of the present invention.

The apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention may receive flight information from a drone.

Referring to FIG. 3 , the projection image stabilization device 300 of the drone-mounted projector may communicate with the drone 350 , and the drone 350 includes a flight control device 360 and a sensor driver 370 . can do. Meanwhile, the stabilization model estimator 310 , the image converter 320 and the image transmitter 330 of FIG. 3 include the stabilization model estimator 210 , the image converter 220 and the image transmitter 230 of FIG. 2 . ), so a detailed description will be omitted.

The sensor driving unit 370 of the drone 350 may include a lidar sensor 371 , an acceleration sensor 372 , an ultrasonic sensor 373 , a geomagnetic sensor 374 and an angular velocity sensor 375 , but in FIG. It is apparent to those skilled in the art that the description is merely an example, and the sensor driving unit 370 may include some of the above-described sensors, and may include other sensors in addition to the above-described sensors.

For example, the sensor driver 370 may sense the distance between the drone and the projection area through the lidar sensor 371 or the ultrasonic sensor 373 , and an inertial measuring device (not shown) or an attitude and orientation device (not shown). ), the drone's attitude information (roll, pitch, yaw axis rotation angle) can be sensed.

Accordingly, the projection image stabilization apparatus 300 of the drone-mounted projector may receive the distance from the sensor driver 370 to the drone and the projection target, the altitude of the drone, the rotation angle of the drone, and the like.

4 is a block diagram of an apparatus for stabilizing a projection image of a drone-mounted projector according to another embodiment of the present invention.

The apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention may further include a sensor driver for sensing flight information of the drone. Meanwhile, the stabilization model estimator 410 , the image converter 420 and the image transmitter 430 of FIG. 4 include the stabilization model estimator 210 , the image converter 220 and the image transmitter 230 of FIG. 2 . ), so a detailed description will be omitted.

The sensor driver 440 of FIG. 4 may transmit flight information of the drone to the stabilization model estimator 410 . At this time, since the flight information of the drone sensed by the sensor driver 440 corresponds to the flight information of the drone sensed by the sensor driver 370 of FIG. 3 , a detailed description thereof will be omitted.

5 is a diagram for explaining the structure of a projection image of a drone-mounted projector according to an embodiment of the present invention.

The resolution of the projected image of the drone-mounted projector according to an embodiment of the present invention may be lower than the maximum resolution that the projector can transmit.

Referring to FIG. 5 , only the W/3XH/3 size projection image area 520 is used among the maximum images 510 that the projector supporting the maximum WXH resolution can transmit. Meanwhile, although the size of the projected image area 520 of FIG. 5 corresponds to 1/9 of the maximum image 510 that can be transmitted, this is only an example, and it is common in the art that other ratios other than 1/9 may be applied. It is obvious to the engineer of

Accordingly, in the image of the projection of the drone-mounted projector according to an embodiment of the present invention, the area of the projected image is out of the maximum image that can be transmitted by the projector due to scaling, movement of the center point, and inclination (and keystone) of the image. can prevent

6 is a view for explaining a coordinate system used in a method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.

In the method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention, the movement of the drone may be divided into translation and rotation of the drone.

Referring to FIG. 6 , the movement of the drone can be defined on the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis represent the drone's progress (front and back) direction, left-right direction, and height (up and down) directions, respectively. it means. And the rotation of the drone can be defined on the roll axis, the pitch axis, and the yaw axis. do.

And the 3D projector image plane 620 means an undistorted image. When the drone-mounted projector projects an image on the three-dimensional projector image plane 620, distortion due to vibrations occurring in the motor of the drone or changes in the position and posture of the drone due to drone flight control error occurs. A distorted image may be projected on the 2D projection plane 630 .

Here, it can be seen that the X value of the 3D projection image plane 620 does not appear any longer while being projected onto the 2D projection plane 630 , and only Y and Z values exist. In the present invention, it can be seen that the coordinates defined in the coordinate system of the drone-mounted projector 610 are followed in the three-dimensional projection image plane 620 and the two-dimensional projection plane 630 . This is also shown in the following equation for stabilizing the projected image.

Hereinafter, the present invention will be described in detail on the premise that the position and posture of the drone are the same as the position and posture of the drone-mounted projector. However, even if the position and posture of the drone and the drone-mounted projector are different, it is obvious to those skilled in the art that the projection image stabilization method and apparatus of the present invention can be applied.

7 is a diagram for explaining a distortion type of a projected image according to an embodiment of the present invention.

A method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention can perform stabilization of a projected image in consideration of a distance variation between a drone and a projection area, a height variation of the drone, and a rotation variation of the drone. Each of the three transitions causes scaling of the projected image, shifting the center point of the projected image, and tilting (and keystone) of the projected image, respectively.

Fig. 7(a) shows the types of projection image distortion according to the movement of the drone, and Fig. 7(b) shows the projection image distortion types according to the rotation of the drone.

Referring to FIG. 7A , image distortion may occur on the X-axis, Y-axis, and Z-axis according to the movement of the drone. Image distortion on the X-axis causes scale distortion of the projected image and vertical movement of the image. Image distortion on the Y-axis causes horizontal movement of the projected image, and image distortion on the Z-axis causes vertical movement of the projected image.

Referring to FIG. 7B , image distortion may occur on a roll axis, a pitch axis, and a yaw axis according to the rotation of the drone. Image distortion on the roll axis may cause inclination and vertical and horizontal movement of the projected image, and image distortion on the pitch axis may cause vertical keystone and vertical movement of the projected image. Also, image distortion on the yaw axis may cause horizontal keystone and horizontal shift of the projected image.

A method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention include distortion of the projection image on the X-axis and Z-axis according to the movement of the drone, and on the roll axis, the pitch axis, and the yaw axis according to the rotation of the drone. can correct the distortion of the projected image.

8 is a view for explaining a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.

It is apparent to those skilled in the art that FIG. 8 does not mean a specific rotation of the projected image, but is merely a diagram for explaining a projection image stabilization model.

Referring to FIG. 8 , an image 825 on the 3D projector image plane 820 means an undistorted image. When the drone-mounted projector projects the image 825 , distortion occurs due to vibrations generated in the motor of the drone or changes in the position and posture of the drone due to drone flight control errors. The distorted image may be displayed as an image 835 on the virtual plane 830 . A method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention, based on flight information of a drone, convert an image 835 on a distorted virtual plane 830 to an image on the projection plane 840 ( 845) discloses a stabilization method.

Referring to Equation 1 below, the method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention converts coordinates X _i on a distorted two-dimensional projection plane into coordinates X _w on a three-dimensional projector image plane. A stabilization model can be used for stabilization.

[Equation 1]

은 수학식 2와 같이 정의할 수 있다.And, the stabilization model of Equation 1

can be defined as in Equation (2).

[Equation 2]

Here, H _intrinsic represents an internal parameter matrix, and may include a projector focal length that is a constant value, a coordinate value of a center point of an image during projection, and the like, but the internal parameter included in H _{intrinsic is not limited thereto.}

And, H _translation means a distortion image correction matrix according to the movement of the drone-mounted projector, and can be decomposed into a distortion image correction matrix H _x according to the X-axis movement and a distortion image correction matrix H _{z according to the Z-axis movement.} H _rotation means a distortion image correction matrix according to the rotation of the drone-mounted projector, a distortion image correction matrix H _Roll according to the roll axis rotation, a distortion image correction matrix H _Pitch according to the pitch axis rotation, and a distortion image correction matrix according to the yaw axis rotation It can be decomposed into H _Yaw.

In addition, H _unit means a parameter matrix for converting parameters estimated through flight information into pixel units, and can be obtained using the distance between the drone and the projection area and the projection angle. Finally, the о operation means a Hadamard product operation.

Meanwhile, in the method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention, the internal parameter matrix H _intrinsic of Equation 2 may be defined as Equation 3 below.

[Equation 3]

Here, f denotes the focal length of the projector, and c _x , c _z denote the coordinates of the center point of the projected image. f, c _x , and c _z may vary depending on the characteristics of the projector used in manufacturing the drone-mounted projector, and since the values are determined when the projector to be used is determined, they can be viewed as constant values.

9 is a diagram for explaining a method of correcting distortion of a projected image due to movement of a drone-mounted projector according to an embodiment of the present invention.

In addition, since the present invention considers the movement of the drone on the X-axis and the Z-axis, a distortion image correction matrix according to the movement of the drone can be defined as in Equation 4 below.

[Equation 4]

Here, H _x is a distortion image correction matrix according to the X-axis movement, and H _z is a distortion image correction matrix according to the Z-axis movement. H _x and H _z in Equation 4 may be defined as in Equation 5 below.

[Equation 5]

Here, S is the scale value, S _x is the vertical movement coefficient according to the X-axis movement, and S _z is the vertical movement coefficient according to the Z-axis movement. Among them, the scale value S can be obtained through the distance ratio between the drone and the projection area.

Specifically, as shown in FIG. 9 , the distance from the setting position 910 of the drone-mounted projector to the projection area 920 is defined as d ₀ , and when a motion occurs in the drone-mounted projector, the current sensed by the lidar sensor If the distance from the position 930 of the drone-mounted projector to the projection area 920 is d _c , S is d ₀ /d _c .

On the other hand, when the drone-mounted projector projects an image on the projection area 920, such as the outer wall of a building, since there is a vertical offset of the projector lens used in the drone-mounted projector, a shift in the position of the center point of the projected image may occur. have. Here, the vertical offset of the projector lens refers to the degree to which the center of the projected image is moved up and down with respect to the center line of the projector lens. For example, 0% offset means that the center of the projected image is aligned with the centerline of the lens, and 100% offset means that the center of the projected image is above the centerline of the lens, so the bottom of the image is aligned with the centerline of the lens do.

The movement coefficient S _x can be expressed as Equation 6 below through geometric modeling.

[Equation 6]

는 프로젝션 각도, d ₀ 는 드론과 프로젝션 영역의 초기 설정 거리, d _c 은 측정된 드론과 프로젝션 영역의 현재 거리를 의미한다.In the above formula,

is the projection angle, d ₀ is the initial set distance between the drone and the projection area, and d _c is the measured current distance between the drone and the projection area.

That is, the distortion image correction matrix according to the X-axis movement of the drone-mounted projector can be inferred as in Equation 7 below based on Equations 5 and 6.

[Equation 7]

And the movement coefficient S _z can be obtained as in Equation 8 based on the current altitude value a _c and the set altitude value a _{0 of the drone-mounted projector.}

[Equation 8]

Therefore, the distortion image correction matrix according to the Z-axis movement of the drone-mounted projector can be inferred as follows.

[Equation 9]

10 is a view for explaining the movement coefficient according to the roll axis rotation of the projection image stabilization model of the drone-mounted projector according to an embodiment of the present invention.

In the present invention, since the rotation on the roll axis, the pitch axis, and the yaw axis of the drone are considered, a distortion image correction matrix according to the rotation of the drone can be defined as in Equation 10 below.

[Equation 10]

Here, H _Roll is a distortion image correction matrix according to roll axis rotation, and H _Pitch means a distortion image correction matrix according to pitch axis rotation. Also, H _Yaw is It means a distortion image correction matrix according to the rotation of the yaw axis. On the other hand, H _Roll , H _Pitch , H _Yaw are rotation coefficients ( r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ , r ₃₁ , r ₃₂ ) and movement coefficients ( t _x , t _y , t _z , respectively, as in Equation 11) ) is composed of

[Equation 11]

Here, the rotation coefficients ( r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ , r ₃₁ , r ₃₂ ) can be obtained through a well-known 3D rotation matrix, and the movement coefficients ( t _x , t _y , t _z ) are geometrically modeled. necessary. Therefore, H _Roll , H _Pitch , and H _Yaw can be expressed as in Equation 12 below.

[Equation 12]

은 각각 롤축 회전각도, 피치축 회전각도, 요축 회전각도를 의미하며,

은 각각 롤축 회전에 의한 수평/수직 이동계수를 의미한다. 또한,

은 각각 피치축 회전에 의한 수평/수직 이동계수를 의미하고,

은 각각 요축 회전에 의한 수평/수직 이동계수를 의미한다. 이하에서는 각 축에 따른 이동계수 모델링에 관하여 설명하기로 한다.here,

means the roll axis rotation angle, the pitch axis rotation angle, and the yaw axis rotation angle, respectively.

are the horizontal/vertical movement coefficients by the rotation of the roll shaft, respectively. In addition,

is the horizontal/vertical movement coefficient by the rotation of the pitch axis, respectively,

are the horizontal/vertical movement coefficients by rotation of the yaw axis, respectively. Hereinafter, motion coefficient modeling along each axis will be described.

는 도 10과 같은 기하학적 모델링으로 유추할 수 있다.First, the movement coefficient in the image distortion correction model according to the rotation of the drone's roll axis

can be inferred by geometric modeling as shown in FIG. 10 .

를 이용하여 표현할 수 있으며,

는 각각 기하학적 모델링에 의해 수학식 13과 같이 정의할 수 있다.Meanwhile, since the projector lens offset according to an embodiment of the present invention is 100%, rotation occurs based on the lower line of the projected image. Accordingly, referring to FIG. 10 , the position of the midpoint of the projected image is shifted from 1010 to 1020. That mutation is

can be expressed using

can be defined as in Equation 13 by geometric modeling, respectively.

[Equation 13]

이다.here

am.

11 is a view for explaining the movement coefficient according to the pitch axis rotation of the projection image stabilization model of the drone-mounted projector according to an embodiment of the present invention.

는 다음과 같은 기하학적 모델링으로 유추할 수 있다.On the other hand, the movement coefficient in the image distortion correction model according to the rotation of the drone's pitch axis

can be inferred from the following geometrical modeling.

로 변이가 발생하게 된다. 해당 변이는

으로 표현할 수 있는데, Y축으로는 이동이 발생하지 않고, Z축으로만 이동이 발생하기 때문에,

의 값은 0이고,

의 값만 존재하게 된다. Since the rotational characteristic of the pitch axis causes movement of the center point of the projected image, a movement coefficient exists. _Referring to FIG. 11 , the location of the midpoint of the projected image is at C z_pitch .

mutation will occur. That mutation is

It can be expressed as

has a value of 0,

only values of .

Therefore, the movement coefficient according to the rotation of the pitch axis can be defined as in Equation 14 below through the geometric modeling of FIG. 11 .

[Equation 14]

=

=

,

이다.here,

,

am.

12 is a view for explaining a movement coefficient according to a yaw axis rotation of a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.

는 다음과 같은 기하학적 모델링으로 유추할 수 있다.On the other hand, the movement coefficient in the image distortion correction model according to the rotation of the yaw axis of the drone

can be inferred from the following geometrical modeling.

으로 표현할 수 있는데, Z축으로는 이동이 발생하지 않고, Y축으로만 이동이 발생하기 때문에,

의 값은 0이고

의 값만 존재하게 된다. Since the yaw axis rotation characteristic causes a movement of the center point of the projected image, a movement coefficient exists during the yaw axis rotation. Referring to FIG. 12 , the position of the midpoint of the projected image is shifted from 1210 to 1220. That mutation is

It can be expressed as

has a value of 0

only values of .

Therefore, the movement coefficient according to the rotation of the yaw axis can be defined as in Equation 15 below through the geometric modeling of FIG. 12 .

[Equation 15]

=

=

13 is a diagram for explaining a unit transformation matrix of a projection image stabilization model of a drone-mounted projector according to an embodiment of the present invention.

과 단위변환 행렬

은 아다마르 곱 (Hardamard product)로 계산되며, 이때 단위변환 행렬

은 아래 수학식 16에 의해 정의될 수 있다. Meanwhile, the method and apparatus for stabilizing a projection image of a drone-mounted projector according to an embodiment may change a unit of values estimated through flight information of a drone from a distance unit (cm) to a pixel unit (pixel). As defined in Equation 2, the stabilization model

and unit conversion matrix

is calculated as a Hardamard product, where the unit conversion matrix

may be defined by Equation 16 below.

[Equation 16]

는 도 13에서 현재 드론의 위치(1310)에서 프로젝션 영역(1320)까지의 거리이고,

는 프로젝션 각도를 의미한다.here,

is the distance from the current drone position 1310 to the projection area 1320 in FIG. 13,

is the projection angle.

14 is a view for explaining a hum environment for quantitative performance evaluation of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.

In order to verify the quantitative performance of the apparatus for stabilizing the projection image of the drone-mounted projector according to an embodiment of the present invention, the stabilization process of the projection image was photographed with a camera. Referring to FIG. 14 , in the performance verification experiment, the distance from the drone 1410 to the projection target 1420 was 3.05 m, and the distance from the drone 1410 to the camera 1430 was 3.60 m. In addition, to simulate the motion of the drone-mounted projector, a ball joint, a camera slider, and a tripod were used. Specifically, a ball joint was used to simulate 3-axis rotation, and a camera slider and tripod were used to simulate distance and elevation changes, respectively.

15 and 16 are diagrams for explaining the qualitative performance of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.

15 is a view showing the degree of movement of the projected image when the projection image stabilization apparatus of the drone-mounted projector according to an embodiment of the present invention is used and when not used.

Referring to FIG. 15, when the projection image stabilization device of the drone-mounted projector is not used (stabilization is OFF), it can be seen that the position and size of the projected image in the 0th, 350th, 700th, and 1050th frames change significantly. can However, when the projection image stabilization device of the drone-mounted projector was used (stabilization ON), the position and size of the projected image in the 0th, 350th, 700th, and 1050th frames were relatively stable, and the attitude of the drone-mounted projector was It can be seen that image tilt and keystone do not occur according to the change.

16 is a diagram illustrating a projection image in the case of using and not using the projection image stabilization apparatus of the drone-mounted projector according to an embodiment of the present invention.

16 (a) is a sample image used in an experimental environment used to quantitatively verify the performance of the projection image stabilization apparatus of the drone-mounted projector described in FIG. 14 . This was projected using a drone-mounted projector using a sample image in which five dots (top left, top right, center, bottom left, bottom right) of different colors were displayed. Then, five points were tracked using different colors for quantitative evaluation.

Referring to (b) of FIG. 16 , when the projection image stabilization device of the drone-mounted projector is not used, it can be seen that the positions of the five dots of different colors change significantly.

However, referring to (c) of FIG. 16 , when the projection image stabilization apparatus of the drone-mounted projector is used, it can be seen that the positions of the five dots of different colors are relatively stabilized.

17 is a diagram for explaining the performance of a projection image stabilization apparatus of a drone-mounted projector according to an embodiment of the present invention.

Referring to FIG. 17, when the projection image stabilization device of the drone-mounted projector is not used (OFF), the standard deviations of the upper left, upper right, center, lower left, and lower right points of the sample image are (12, 65, 14.31, respectively). ), (13.66, 16.91), (14.44, 17.15), (13.74, 15.94), (14.28, 16.95).

However, when the projection image stabilization device of the drone-mounted projector is used (ON), the standard deviations of the upper-left, upper-right, center, lower-left, and lower-right points of the sample image are (4.33, 4.73), (4.51, 5.06), respectively, (4.35, 5.00), (4.60, 5.28), (4.03, 4.93), it can be seen that the standard deviation is reduced compared to the case where the stabilization device is not used.

18 is a flowchart illustrating a method of stabilizing a projection image of a drone-mounted projector according to an embodiment of the present invention.

In operation 1810, flight information of a drone equipped with a projector may be received. In this case, the flight information of the drone may include at least one of a distance between the drone and a projection target, an altitude of the drone, and a rotation angle of the drone.

Meanwhile, step 1810 of the projection image stabilization method according to an embodiment may be to receive flight information of the drone from a sensor driver inside the drone.

In addition, the projection image stabilization method according to another embodiment may further include sensing flight information of the drone by a sensor driving unit, and in this case, step 1810 may be to receive the sensed flight information of the drone.

In operation 1820, a stabilization model for correcting the distortion of the projection image may be estimated based on the received flight information.

Here, the stabilization model may be decomposed into at least one of an internal parameter matrix, that is, a matrix composed of constant parameters determined by the projector, a distortion image correction matrix according to the movement of the drone, and a distortion image correction matrix according to the rotation of the drone.

In this case, the internal parameter matrix may vary depending on at least one of the focal length of the projector and the coordinates of the center point of the transmitted image. Here, the projection angle is an internal parameter that is not included in the internal parameter matrix but is determined by the projector.

The movement and rotation of the drone is defined by three axes, and the movement of the drone is defined as the X-axis, the right direction relative to the drone, the Y-axis, and the lower side as the Z-axis. In the rotation of the drone, the X, Y, and Z axes of movement become the Roll, Pitch, and Yaw axes, and the direction in which the right-hand rule is applied based on the axis direction is the positive direction of the rotation angle.

In the present invention, in order to reduce additional calculations, the definition of the coordinate system of the drone can be equally used in the projection image.

If it is determined through the received flight information that the drone has moved on the X axis, which is the direction of travel, the distortion image correction matrix according to the movement of the drone is the projection angle, the initial set distance between the drone and the projection target, and the current distance between the drone and the projection target. may vary depending on at least one of

If it is determined through the received flight information that the drone has moved on the Z-axis in the height direction, the distortion image correction matrix according to the movement of the drone may vary depending on the altitude information of the drone.

If it is determined through the received flight information that the drone has rotated on the roll axis in the direction of travel, the distortion image correction matrix according to the rotation of the drone is based on at least one of the roll axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. may vary.

If it is determined that the drone rotates on the pitch axis in the height direction through the received flight information, the distortion image correction matrix according to the rotation of the drone is at least one of the pitch axis rotation angle, the projection angle, and the initial set distance between the drone and the projection target. may vary depending on

If it is determined through the received flight information that the drone rotates on the yaw axis in the left and right direction, the distortion image correction matrix according to the rotation of the drone may vary depending on at least one of the yaw axis rotation angle and the initial set distance between the drone and the projection target. .

Since the image stabilization model by the movement and rotation of the drone is defined independently of each other, when movement and rotation about each axis occur in combination, it can be expressed through matrix multiplication.

In operation 1830, the projected image may be transformed based on the estimated stabilization model.

Specifically, in step 1830, the estimated stabilization model is applied in units of pixels by performing a Hardamard product operation of the estimated stabilization model and the unit transformation matrix. Here, the unit transformation matrix may vary according to at least one of a projection angle, a current distance between the drone and a projection target, and a vertical resolution of the transmitted image.

In operation 1840, the converted image may be transmitted toward the projection target. In this case, the resolution of the transmitted image may be lower than the maximum resolution that the projector can transmit.

In the present invention, the higher the resolution of the converted image to stabilize the image, the longer the converted image transmission time. Accordingly, the resolution of an image may be lowered to ensure real-time performance of the present invention. So far, preferred embodiments of the present invention have been mainly described. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

200: Projection image stabilization device of a drone-mounted projector
210: transformation model estimator
220: video conversion unit
230: video transmitter

Claims (17)

A method of stabilizing a projection image of a drone-mounted projector, the method comprising:
Receiving flight information of a drone equipped with a projector;
estimating a stabilization model for correcting the distortion of the projected image based on the received flight information;
converting the projected image based on the estimated stabilization model; and
A method of stabilizing a projection image of a drone-mounted projector, comprising the step of transmitting the converted image toward a projection target. According to claim 1,
Receiving the flight information of the drone comprises:
A method of stabilizing a projection image of a drone-mounted projector by receiving the flight information of the drone from the sensor driving unit inside the drone. According to claim 1,
Further comprising the step of sensing the flight information of the drone by a sensor driving unit,
The step of receiving the flight information of the drone is to receive the sensed flight information of the drone, the projection image stabilization method of the drone mounted projector. According to claim 1,
The flight information of the drone,
A method for stabilizing a projection image of a drone-mounted projector comprising at least one of a distance between the drone and the projection target, an altitude of the drone, and a rotation angle of the drone. According to claim 1,
The resolution of the transmitted image is lower than the maximum resolution that the projector can transmit, the drone-mounted projector projection image stabilization method. According to claim 1,
The stabilization model is to be decomposed into at least one of an internal parameter matrix, a distortion image correction matrix according to the movement of the drone, and a distortion image correction matrix according to the rotation of the drone. 7. The method of claim 6,
The internal parameter matrix is a method for stabilizing a projection image of a drone-mounted projector that varies depending on at least one of a focal length of the projector and a coordinate of a center point of the transmitted image. 7. The method of claim 6,
When it is determined through the received flight information that the drone has moved on the X-axis, which is the direction of travel,
The distortion image correction matrix according to the movement of the drone depends on at least one of a projection angle, an initial set distance between the drone and the projection target, and a current distance between the drone and the projection target. Projection image of a drone mounted projector stabilization method. 7. The method of claim 6,
When it is determined that the drone has moved on the Z-axis in the height direction through the received flight information,
A method for stabilizing a projection image of a drone mounted projector, wherein the distortion image correction matrix according to the movement of the drone is changed according to the altitude information of the drone. 7. The method of claim 6,
When it is determined through the received flight information that the drone rotates on the roll axis in the traveling direction,
The distortion image correction matrix according to the rotation of the drone is changed according to at least one of a roll axis rotation angle, a projection angle, and an initial set distance between the drone and the projection target. 7. The method of claim 6,
When it is determined that the drone rotates on the pitch axis in the left and right directions through the received flight information,
The distortion image correction matrix according to the rotation of the drone is changed according to at least one of a pitch axis rotation angle, a projection angle, and an initial set distance between the drone and the projection target. 7. The method of claim 6,
When it is determined that the drone rotates on the yaw axis in the height direction through the received flight information,
A method for stabilizing a projection image of a drone mounted projector, wherein the distortion image correction matrix according to the rotation of the drone varies according to at least one of a yaw axis rotation angle and an initial set distance between the drone and the projection target. According to claim 1,
Converting the image comprises:
Comprising the step of performing a Hadamard product (Hardamard product) operation of the estimated stabilization model and the unit transformation matrix,
The operation result of the Hadamard product is applied to the projected image in units of pixels,
The unit transformation matrix is a projection angle, the current distance between the drone and the projection target, the method of stabilizing a projection image of a drone mounted projector that varies depending on at least one of the vertical resolution of the transmitted image. A computer-readable storage medium storing a computer program in which the method according to any one of claims 1 to 13 is performed when the computer program is executed by a processor. In the projection image stabilization device of the drone mounted projector,
When flight information of a drone equipped with a projector is received, a stabilization model estimator for estimating a stabilization model for correcting distortion of a projected image based on the flight information;
an image conversion unit that converts the projected image based on the estimated stabilization model;
Projection image stabilization apparatus of a drone-mounted projector, comprising an image transmitting unit, which transmits the converted image toward a projection target. 16. The method of claim 15,
Further comprising a sensor driver for transmitting the flight information of the drone to the stabilization model estimator,
The flight information of the drone,
A projection image stabilization apparatus for a drone mounted projector comprising at least one of a distance between the drone and the projection target, an altitude of the drone, and a rotation angle of the drone. 16. The method of claim 15,
The resolution of the image transmitted by the image transmitting unit is lower than the maximum resolution that the projector can transmit, the projection image stabilization device of the drone mounted projector.

Images