KR20230066752A - Method and apparatus for collecting video information - Google Patents

Abstract

The present invention relates to a method and apparatus for collecting image information. A method according to an embodiment of the present invention is a method for collecting image information performed by an electronic device, and affects the position of each first training image for a plurality of first training images having different frame positions in an image sequence. Generating n pieces of position data for learning to which noise is added (where n is a natural number of 2 or more); Performing learning according to a machine learning technique on a deep learning neural network each having an input layer inputting the training location data and an output layer outputting original location data for the original location of the corresponding first training image; generating n object position data to which noise affecting the position of each target image is added for a plurality of target images having positions of different frames in the image sequence; and inputting each object position data to the first model of the learned neural network, extracting the estimated position data for the estimated positions of a plurality of target images, and correcting the position of each target image using the extracted estimated position data. doing; and performing depth estimation or 3D shape restoration using each corrected target image.

Description

Video information collection method and apparatus {METHOD AND APPARATUS FOR COLLECTING VIDEO INFORMATION}

The present invention relates to a method and apparatus for collecting image information, and more particularly, when image information is collected in a rain control system or the like, image information for optimal depth estimation is collected by removing jitter noise included in the image. It relates to a method and apparatus for doing so.

In computer graphics-based applications such as virtual reality, games, and animations, skilled designers manually create 3D models, which take a lot of time and have many problems in quality depending on the designer's skill level. As an alternative to these problems, 3D shape restoration technology is being used. That is, the 3D shape restoration technology is a technology for restoring information about a 3D shape of an object by analyzing an image.

Among these 3D shape restoration techniques, a frequently used depth estimation technique based on depth from focus (DFF) is simple and precise technique to implement. In this case, in order to perform DFF-based depth estimation, the frame position (ie, order) of each image in the image sequence must be accurately identified. However, jitter noise tends to occur during shooting with a camera, and such jitter noise may adversely affect frame positioning of each image in an image sequence.

Accordingly, in order to remove such jitter noise, a method using a Kalman filter and a Bayes filter has been developed. However, since this method restricts jitter noise to Gaussian noise and removes it, there is a limitation in that it is difficult to apply to actual images.

In addition, as another method for removing jitter noise, there is a method using a modified Kalman filter and an adaptive neural network filter. However, since this method uses only the variance of jitter noise, it cannot effectively remove jitter noise having outliers that deviate greatly from the average value.

In addition, as another method for removing jitter noise, there is a method using a Kalman filter based on maximum corentropy. However, this method cannot effectively remove jitter noise due to numerical instability and computational complexity in the covariance matrix and Kalman gain for the frame position of the image, so the accuracy of the depth estimation technology through the image sensor in the unmanned control system is lowered. Occurs.

In order to solve the problems of the prior art as described above, the present invention effectively removes jitter noise included in the image when image information is collected in a rain control system, etc., and image information with improved accuracy for depth estimation. Its purpose is to provide a technology for collecting

However, the problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

To solve the above problem, a method for collecting image information according to an embodiment of the present invention is a method for collecting image information performed by an electronic device, and includes a plurality of first learning images having different frame positions in an image sequence. generating n (n is a natural number of 2 or more) position data for learning to which noise affecting the position is added for each first training image; Performing learning according to a machine learning technique on a deep learning neural network each having an input layer inputting the training location data and an output layer outputting original location data for the original location of the corresponding first training image; generating n object position data to which noise affecting the position of each target image is added for a plurality of target images having positions of different frames in the image sequence; and inputting each object position data to the first model of the learned neural network, extracting the estimated position data for the estimated positions of a plurality of target images, and correcting the position of each target image using the extracted estimated position data. doing; and performing depth estimation or 3D shape restoration using each corrected target image.

The deep learning neural network may be a long short-term memory neural network.

In the step of performing depth estimation or 3D shape restoration, a focus value of each target image is derived based on the corrected position of each target image, and the depth estimation or 3D shape restoration is performed using the derived focus value. It may include the step of performing.

The step of performing depth estimation or 3D shape restoration is based on the focus value of each pixel derived as a result of the first focus measurement operation on a plurality of pixels in the second training image and the corresponding focus value. The depth estimation or the 3D shape reconstruction may be performed using a second model in which machine learning of Gaussian Process Regression has been performed based on the fitting function of the estimated focus curve.

The performing of depth estimation or 3D shape restoration may include performing a second focus measurement operation on a plurality of pixels in the corrected target image; fitting a plurality of focus curves by inputting focus values of a plurality of pixels derived as a result of the second focus measurement operation to the second model; extracting a pixel position in the corrected target image having a maximum focus value from a plurality of fitted focus curves; and performing the depth estimation or 3D shape restoration based on the extracted pixel position.

The first and second focus measurement operations may use a sum of modified laplacian (SML).

The SML may use the following formula.

(However, I(x, y) is the gray level brightness at the pixel of (x, y), W is the video window size)

The learning method for the second model may include selecting a kernel function of a probability distribution for a fitting function as k(i, i′), which is a square exponential kernel;

(단, i와 I'는 제곱 지수 커널 함수의 입력, | | | |는 유클리디안 거리)

(However, i and I' are the inputs of the square exponential kernel function, | | | | is the Euclidean distance)

For the initial probability distribution for the fitting function, setting the mean to 0 and setting the kernel function to k(x ₀ , x ₀ '), where x ₀ and x ₀ ' are axes for pixel locations in the target data a set of values on the x-axis); and setting the mean to m _g (x ₀ ) and the kernel function to k _g (x ₀ , x ₀ ') for the updated probability distribution for the fitting function.

(However, x _t is a set of x-axis values, which is an axis for pixel position values in training data, and x _t 'is a set of y-axis values, which is an axis for focus values in training data)

The performing of the second focus measurement operation may include randomly sampling and extracting a plurality of pixels from the corrected target image, and performing a second focus measurement operation on the plurality of extracted pixels.

The noise may have a non-Gaussian distribution.

An apparatus according to an embodiment of the present invention includes a control unit for controlling depth estimation or 3D shape restoration using information stored in a memory.

The control unit controls n (n is a natural number greater than or equal to 2) to which noise affecting the position of each first training image is added to a plurality of first training images having positions of different frames in the image sequence. For a deep learning neural network having an input layer that controls the generation of location data and inputs each of the location data for learning, and an output layer that outputs the original location data for the original location of the corresponding first learning image, according to machine learning techniques Controls learning, controls generation of n object position data to which noise affecting the position of each target image is added for a plurality of target images having positions of different frames in an image sequence, and controls the learning of the neural network Each object location data is input to the first model of , the estimated location data for the estimated locations of a plurality of target images is extracted, and position correction of each target image is controlled using the extracted estimated location data, and the corrected Depth estimation or 3D shape restoration may be controlled using each target image.

The present invention configured as described above can collect image information according to optimal depth estimation or optimal 3D shape restoration by effectively removing jitter noise included in images when image information is collected in a rain control system or the like. There are advantages that can be

In addition, the present invention can improve the accuracy of depth estimation or 3D shape restoration by more accurately correcting the frame position of the target image changed according to jitter noise by applying long and short term memory technology, which is one of deep learning technologies. There is an advantage.

In addition, the present invention has an advantage in that image information according to optimal depth estimation or optimal 3D shape restoration can be collected with only a small amount of data by using a Gaussian process regression technique for depth estimation or 3D shape restoration.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows a block configuration diagram of an image information collection device 100 according to an embodiment of the present invention.
FIG. 2 shows a block diagram of a controller 150 for controlling frame position correction of an image in the image information collection device 100 according to an embodiment of the present invention.
3 is a block diagram of a controller 150 for controlling depth estimation according to a DFF technique in the image information collecting apparatus 100 according to an embodiment of the present invention.
4 shows a conceptual diagram for depth estimation according to a depth from focus (DFF) technique.
5 is a flowchart of a method for collecting image information according to an embodiment of the present invention.
6 shows a detailed flowchart of S210.
7 shows a detailed flowchart of S220.
8 is a conceptual diagram of a technique for minimizing jitter noise with respect to a position of an image frame using a first model according to the present invention.
9 shows an example of the structure of a long-short-term memory neural network.
FIG. 10 shows an example of the structure of a cell at time t included in the long and short term memory neural network of FIG. 9 .
11 shows a detailed flowchart of S230.
12 shows a detailed flowchart of S240.
13 illustrates an example of a principle of forming a depth estimation or a 3D reconstructed image by depth from focus (DFF) or shape from focus (SFF).
14 shows a comparison result graph according to the prior art and the present invention.

The above objects and means of the present invention and the effects thereof will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily understand the technical idea of the present invention. will be able to carry out. In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "have", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "or B" and "at least one of B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, and tolerances, measurement errors, limits of measurement accuracy and other commonly known factors It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In the present specification, when an element is described as being 'on' or 'in contact with' another element, it may be in direct contact with or connected to the other element, but another element may be present in the middle. It should be understood that On the other hand, if an element is described as being 'directly on' or 'directly in contact with' another element, it may be understood that another element in the middle does not exist. Other expressions describing the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows a block configuration diagram of an image information collection device 100 according to an embodiment of the present invention.

The image information collection apparatus 100 according to an embodiment of the present invention performs depth estimation on a plurality of target images having different frame positions in an image sequence, and the image according to the depth estimation (or 3D shape restoration). As a device for collecting information, it may be an electronic device capable of computing or a computing network.

In this case, depth estimation or 3D shape restoration is based on depth from focus (DFF) technique or shape from focus (SFF) technique for a plurality of target images, and depth information (ie, depth image, etc.) or 3D It corresponds to an image processing technology that derives a restored shape. Also, depth estimation or 3D shape reconstruction according to the DFF or SFF technique may be performed based on Gaussian Process Regression.

In particular, the image information collection device 100 may be a device applied to various control systems such as a rain control system. That is, the image information collecting device 100 uses a depth from focus (DFF) technique for an image of a target area (ie, a target image) obtained from an image capture device (eg, a camera) to manage the target area of the control system. Alternatively, depth estimation or 3D shape restoration may be performed by applying the SFF technique.

For example, the electronic device includes a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, and a personal digital assistant (PDA). , a smartphone, a smart pad, or a mobile phone, or may be a separately implemented device for collecting image information, but is not limited thereto.

As shown in FIG. 1 , the image information collecting device 100 may include an input unit 110, a communication unit 120, a display 130, a memory 140, and a control unit 150.

The input unit 110 generates input data in response to a user's input, and may include various input means. For example, the input unit 110 includes a keyboard, a key pad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), menu button (menu button), etc. may be included, but is not limited thereto.

The communication unit 120 is a component that communicates with other devices. For example, the communication unit 120 may perform 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication or wired communication such as cable communication may be performed, but is not limited thereto.

For example, the communication unit 120 provides image information, a deep learning-based model for correcting the position of each image frame in an image sequence (hereinafter, referred to as a "first model"), and a focus value (focus value) of each pixel of the image. A Gaussian process regression model (hereinafter referred to as “second model”) for fitting a focus curve for value) may be received from another device, and depth estimation results may be transmitted to other devices. . In this case, when the first model or the second model is received from another device, it may be a case in which another device performs learning on the first model or the second model in an image information collection method described later.

The display 130 displays various image data on a screen, and may be composed of a non-emissive panel or a light-emitting panel. For example, the display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or electronic paper (electronic paper) display, etc. may be included, but is not limited thereto. Also, the display 130 may be combined with the input unit 110 and implemented as a touch screen or the like.

The memory 140 stores various types of information necessary for the operation of the image information collection device 100 . The stored information may include image information, a first model, a second model, and program information related to a method for collecting image information to be described later, but is not limited thereto. For example, the memory 140 may be a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), and an optical media type according to its type. ), Magneto-optical media type, Multimedia card micro type, flash memory type, read only memory type, or RAM type (random access memory type), etc., but is not limited thereto. In addition, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The controller 150 may perform various control operations of the image information collection device 100 . That is, the control unit 150 may control the execution of an image information collection method to be described later, and the remaining components of the image information collection device 100, that is, the input unit 110, the communication unit 120, the display 130, the memory ( 140) can be controlled. For example, the control unit 150 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in the corresponding processor.

FIG. 2 shows a block diagram of a controller 150 for controlling frame position correction of an image in the image information collection device 100 according to an embodiment of the present invention. 3 shows a block diagram of the controller 150 for controlling depth estimation according to the DFF technique in the image information collecting device 100 according to an embodiment of the present invention.

The control unit 150 controls the execution of the image information collection method according to an embodiment of the present invention, and as shown in FIGS. 2 and 3, the data generator 151, the first learning unit 152, the first A first application unit 153, a focus measurement operation unit 154, a second learning unit 155, a second application unit 156, an extraction unit 157, and a collection unit 158 may be included. For example, the data generating unit 151, the first learning unit 152, the first application unit 153, the focus measurement operation unit 154, the second learning unit 155, the second application unit 156, The extraction unit 157 and the collection unit 158 may be hardware components of the controller 150 or software processes executed by the controller 150, but are not limited thereto.

Meanwhile, the first and second models may be models learned according to machine learning. In particular, the first model may be a model learned using a long short-term memory neural network, which is a deep learning neural network, through the first training data (first training data), and the second model is the second training data. It may be a model learned according to the machine learning technique of Gaussian process regression through

For example, the first and second training data may include input data and output data pairs (data sets). In this case, the first and second models include a plurality of layers, and a function for a relationship between input data of an input layer and output data of an output layer is included in a hidden layer. When input data is input to the input layer of the first and second models, output data according to the corresponding function may be output to the output layer.

That is, the first and second models express the relationship between input data and output data in multiple layers (ie, layers), and these multiple expression layers are also referred to as “neural networks”. Each layer in the artificial neural network consists of at least one filter, and each filter has a matrix of weights. That is, each element in the matrix of the corresponding filter may correspond to a weight value. A detailed description of the learning and use of the first and second models will be described later.

4 shows a conceptual diagram for depth estimation according to a depth from focus (DFF) technique.

Referring to FIG. 4 , depth estimation according to a depth from focus (DFF) technique is a technique of estimating depth information of a target region using a plurality of 2D images having different focal degrees of the target region. In this case, each 2D image may be obtained according to an image sequence from an image capture device that captures the target region, and a frame position of each 2D image obtained according to the image sequence is determined, and each 2D image obtained according to the determined frame position A focus value of the image may be derived, and depth information of the target region may be estimated based on the derived focus value according to a DFF technique.

However, when acquiring each 2D image, jitter noise may occur due to mechanical vibration of the image capture device in the direction of the optical axis. When such jitter noise occurs, the frame positions of 2D images determined in accordance with the order of image sequences change, and accordingly, depth information estimated using these 2D images inevitably deteriorates in accuracy. In order to solve this problem, in the present invention, frame positions of 2D images may be corrected by removing jitter noise using the first model. As a result, the accuracy of the depth information estimated using the corrected 2D image can be further improved.

Hereinafter, a method for collecting image information according to the present invention will be described in more detail.

5 is a flowchart of a method for collecting image information according to an embodiment of the present invention. 6 shows a detailed flow chart of S210, and FIG. 7 shows a detailed flow chart of S220.

That is, referring to FIG. 5 , the image information collection method according to an embodiment of the present invention includes learning a first model (S210), and correcting a frame position of a target image using the learned first model. Step S220, step S230 of learning a second model, and step S240 of performing depth estimation using the learned second model. At this time, referring to FIG. 6 , S210 may include S211 and S212. Also, referring to FIG. 7 , S220 may include S221 and S222.

8 is a conceptual diagram of a technique for minimizing jitter noise with respect to a position of an image frame using a first model according to the present invention. 9 shows an example of the structure of the long and short-term memory neural network, and FIG. 10 shows an example of the structure of a cell at time t included in the long and short-term memory neural network of FIG. 9 .

First, in S211, the location data generation unit 151 of the control unit 150 generates noise-added information (hereinafter, referred to as “location data for learning”) about frame positions of a plurality of first training images. In this case, each first learning image may be an image frame having a different frame position in an image sequence for a specific region. Also, the noise is noise simulating jitter noise, and may have a non-Gaussian distribution. That is, reality can be reflected in the corresponding noise through such a non-Gaussian distribution.

The location data generation unit 151 is configured with n (provided that n is 2 or more natural number) to generate location data for learning.

For example, n pieces of location data for learning are generated for the first training image at the location of the first frame of the image sequence, and n pieces of location data for learning are generated for the first training image at the location of the second frame of the image sequence. In this way, n pieces of location data for learning (where n is a natural number greater than or equal to 2) may be generated for each first learning image.

In addition, n pieces of training location data generated for any one first training image have the same location value as information on the original frame location of the first training image (hereinafter referred to as "in-situ data") according to the added noise. or may have other positional values.

For example, n pieces of training location data generated for the first training image at the m-th (where m is a natural number greater than or equal to 1) frame position of the image sequence are values of original location data of the first training image according to the added noise. (that is, the 'mth' related value). At this time, as the value of the added noise increases, the generated location data for learning has a value that deviate more from the 'mth' related value as the value of the location data for learning. Of course, if the value of the added noise is '0', the generated position data for learning has the same value as the original position data as the value of the position data for learning.

That is, the n pieces of noise applied to the n pieces of training location data generated for the first training image at the mth frame location are randomly generated and may have a non-Gaussian distribution to simulate actual jitter noise.

Thereafter, the data generator 151 may generate first training data by using the learning position data generated for each first training image and the original position data of the corresponding first training image. At this time, the first learning data includes a pair of input data and output data. That is, the input data of the first training data is data input to the input layer when learning the first model, and includes position data for learning to which noise is added. In addition, output data of the first training data is data input to the output layer when learning the first model, and includes in-situ data.

For example, the first training data for the first training image at the m-th frame location includes n pieces of generated training location data as input data, and the value of the original location data of the first training image (that is, 'm-th ' related values) can be included as output data. Such first training data may be generated for first training images at all frame positions.

Next, in S212, the first learning unit 152 performs learning on the first model according to a machine learning technique using the location data for learning (in particular, the first training data) generated in S211.

In this case, the first model is a deep learning neural network, and may be a short-term memory neural network. That is, the first model may include an input layer inputting position data for learning, an output layer outputting original position data of the corresponding first training image, and a hidden layer having a function between the input layer and the output layer.

Referring to FIGS. 9 and 10 , the short and long term memory neural network operates by deleting unnecessary memories by adding an input gate, a forget gate, and an output gate to long and short term memory cells of the hidden layer and determining what to remember. That is, the long-short-term memory layer in the long-short-term memory neural network generates the cell state C _t with respect to the time step through four components: an input gate (i), a forget gate (f), a cell candidate (g), and an output gate (σ). And forgetting, updating, and outputting operations can be performed on the hidden output value h _t .

In addition, equations related to the long and short term memory cells may be defined as shown in Table 1 below.

Element ceremony σ _c State activation function (hyperbolic tangent function) σ _g Gate activation function (sigmoid function) i _t σ _g (W _t x _t + R _i h _t-1 + b _i ) f _t σ _g (W _f x _t + R _f h _t-1 + b _f ) g _t σ _c (W _g x _t + R _g h _t-1 + b _g ) o _t σ _g (W _o x _t + R _o h _t-1 + b _o ) c _t f _t × c _t-1 + i _t × g _t h _t o _t × σ _c (c _t )

In Table 1, input weight (W), cyclic weight (R), and bias (b), which are combinations of four components, are learnable weights of each short-term memory cell and may be defined according to the following equation.

Such a long-term short-term memory neural network can solve the problem of long-term dependencies of a recurrent neural network (RNN), in which influence on initial information is reduced in a situation where the viewpoint is sufficiently long. That is, compared to RNN, long-short-term memory neural networks show excellent performance in processing long sequences of inputs. Accordingly, the long-term and short-term memory neural network may be advantageous for learning about n pieces of training location data of a non-Gaussian distribution, which require effects on long time points.

For example, when learning is performed using the first training data for the first training image at the m-th frame position, the first learning unit 152 has n training data generated for the first training image at the m-th frame position. Position data is input to the input layer of the first model as input data. That is, referring to FIG. 9 , the first learning unit 152 stores m-th frames in the input regions (ie, n input regions) of the input layer respectively corresponding to the cells (ie, n cells) of the long and short-term memory layer. n pieces of location data for learning generated for the first learning image of the location are input one by one. In addition, the first learning unit 152 inputs the value of the original position data of the first training image at the position of the mth frame (ie, the 'mth' related value) to the output layer of the first model as output data. In this way, learning may be performed on all first training images for each frame position.

However, the above-described S210 may be performed in another device. In this case, the first model according to S210 performed by the other device may be received by the image information collection device 100 of the present invention through the communication unit 120 and used in S220 to be described later. In addition, the first model, which is the result of performing the above-described S210 in the image information collection device 100 of the present invention, may be transmitted to another device through the communication unit 120 . In this case, the image information collecting device 100 of the present invention may operate as a server that performs learning on the long-term and short-term memory neural network and transmits the resultant first model.

Next, in S221, the data generation unit 151 of the control unit 150 generates noise-added information (hereinafter, referred to as “target location data”) about frame positions of a plurality of target images. In this case, each target image may be an image frame having a different frame position in the image sequence of the target region. Also, the noise may be the same type of noise used in S211. That is, the noise is noise simulating jitter noise and may have a non-Gaussian distribution. Of course, S221 may be performed before S212.

For each target image having a different frame position, the position data generation unit 151 adds n random noises of a non-Gaussian distribution that affect the frame position (where n is a natural number greater than or equal to 2). Create target location data of

For example, n object location data are generated for the target image at the first frame location, and n object location data are generated for the target image at the second frame location. In this way, n object location data (where n is a natural number of 2 or more) can be generated for each target image.

In addition, n pieces of location data generated for any one target image may have the same location value as information on the original frame location (ie, original location data) of the target image or may have a different location value depending on the added noise. .

For example, n pieces of object location data generated for the target image at the m-th frame location may be the same as or different from values of the original location data (ie, 'm-th' related value) of the target image according to added noise. . At this time, as the value of the added noise increases, the generated target location data has a value that deviates from the 'mth' related value as a value of the target location data. Of course, if the value of the added noise is '0', the generated target location data has the same value as the original location data as the value of the target location data.

That is, the n pieces of noise applied to the n pieces of object location data generated for the object image at the mth frame location are randomly generated and may have a non-Gaussian distribution to simulate actual jitter noise.

Next, in S222, the first application unit 153 of the control unit 150 applies each target location data to the first model learned in S212. That is, the first application unit 153 inputs each target location data to the input layer of the first model, and data on the estimated locations of a plurality of target images output from the output layer of the first model (hereinafter referred to as “estimated location data”). ”) is extracted. Then, the first application unit 153 corrects the position of each target image using the extracted estimated position data.

For example, when the object location data for the target image at the m-th frame location is applied to the first model, the first application unit 153 applies n pieces of object location data generated for the target image at the m-th frame location. Input data to the input layer of the first model. That is, the first application unit 153 applies the target image at the m-th frame position to the input regions (ie, n input regions) of the input layer respectively corresponding to the cells (ie, n cells) of the long and short-term memory layer. Input the generated n target location data one by one. As a result, estimated position data similar to or equal to the value of the original position data of the target image at the m-th frame position (ie, 'm-th' related value) is output as output data from the output layer of the first model. Then, the first application unit 153 corrects the estimated position data output from the output layer of the first model to the frame position of the target image at the m-th frame position. In this way, the first application unit 153 may correct frame positions (ie, original position data) of target images of all frame positions.

11 shows a detailed flow chart of S230, and FIG. 12 shows a detailed flow chart of S240.

Next, S230 and S240 are performed. That is, in S230, a second model is learned based on Gaussian process regression, and in S240, a depth from focus (DFF) technique or shape from focus based on the second model learned in S230 using each target image corrected in S222 ), depth estimation or 3D shape restoration is performed. At this time, referring to FIG. 11 , S230 may include S231 and S232. Also, referring to FIG. 12 , S240 may include S241 to S244.

In S231, the focus measurement operation unit 154 of the control unit 150 performs a focus measurement operation (hereinafter, referred to as “first focus measurement operation”) on a plurality of pixels selected from the second training image. In this case, the focus measurement operation refers to calculating focus values for each of a plurality of pixels of the second training image. In this case, the plurality of pixels selected may be randomly sampled from the second training image, and may be pixels that are not adjacent to each other and are spaced apart from each other with at least one pixel interposed therebetween.

For example, a technique for calculating focus may include, but is not limited to, a Laplacian technique, a sum of modified Laplacian (SML) technique, a Tenengrad focus measurement technique, or a gray level variance technique. However, in the case of the Laplacian technique, as a technique based on spatial frequency, it can operate normally only when many textures exist on the surface of an object. Accordingly, it may be desirable to use an SML technique capable of measuring a focus value even within a small window in order to compensate for such a texture problem.

At this time, the SML technique may calculate the focus value using the following equation.

However, SML(x, y) represents the result value of the SML operation (i.e., the focus value) for the pixel at the (x, y) coordinate point of the image, and ML(x, y) represents the (x, y) value of the image Indicates the result value of ML operation on the pixel of the coordinate point. In addition, I(x, y) represents the gray level brightness of a pixel at the (x, y) coordinate point of the image, and W represents the image window size.

Next, in S232, the second learning unit 155 of the control unit 150 uses the focus value of each pixel calculated in S241 to calculate the second model using a machine learning technique based on Gaussian Process Regression. carry out learning That is, the second learning unit 155 performs machine learning of Gaussian process regression based on the focus value of each pixel derived as a result of the first focus measurement operation and the fitting function of the focus curve estimated based on the corresponding focus value. can be done

In this case, the input data may include information about a focus value of the image. In addition, the output data may include information about a pixel position of the corresponding image corresponding to the corresponding focus value. Accordingly, the second model is a model learned by a machine learning technique of Gaussian process regression to output a pixel position corresponding to a focus value of a certain image as an input.

For SFF, which will be described later, the second learning data (second training data) may include focus values obtained through the first focus measurement operation as input data, and the regression function may include a focus curve estimated based on the training data ( The x-axis may be defined as a focus position, and the y-axis may be defined as a curve on a coordinate system representing a focus value. Accordingly, the hidden layer of the second model has a function for a relationship between a focus value (input data) and a pixel position (output data) in an image. In this case, the function may reflect characteristics of multiple focus curves estimated according to focus values.

Specifically, S232 may include selecting a kernel function of the probability distribution for the fitting function as k(i, i′), which is a square exponential kernel.

In this case, k(i, i'), which is a square exponential kernel, can be expressed as the following equation.

However, i and I' represent the inputs of the squared exponential kernel function, and | | | | represents the Euclidean distance.

Further, S232 may include setting the mean to 0 and the kernel function to k(x ₀ , x ₀ ') for the initial probability distribution for the fitting function. In this case, in k(x ₀ , x ₀ '), x ₀ and x ₀ ' represent a set of values of an axis (ie, an x-axis) for a pixel position in the target data.

Further, S232 may include setting the average to m _g (x ₀ ) and the kernel function to k _g (x ₀ , x ₀ ') for the updated probability distribution for the fitting function. At this time, m _g (x ₀ ) and k _g (x ₀ , x ₀ ') can be expressed as the following equation.

However, x _t represents a set of values of an axis (ie, an x-axis) of pixel position values in the second training data. In addition, x _t ' represents a set of values of an axis (ie, y-axis) for a focus value in the second training data.

However, the above-described S230 may be performed in another device. In this case, the second model according to S230 performed by the other device is received by the image information collection device 100 of the present invention through the communication unit 120 and can be used in S240 to be described later. In addition, the second model, which is a result of performing S230 described above in the image information collection device 100 of the present invention, may be transmitted to another device through the communication unit 120 . In this case, the image information collection device 100 of the present invention may operate as a server that performs Gaussian process regression learning and transmits a second model that is the result.

Next, in S241, the focus measurement operation unit 154 of the control unit 150 performs a focus measurement operation (hereinafter referred to as “second focus measurement operation”) on a plurality of pixels selected from the corrected target image. This focus measurement operation is the same as described above in S231, except that it is performed on a plurality of pixels selected from a target image other than a training image. In this case, the plurality of pixels selected may be randomly sampled from the target image, and may be pixels that are not adjacent to each other and are spaced apart from each other with at least one pixel interposed therebetween.

Next, in S242, the second application unit 156 of the controller 150 applies the focus values of the plurality of pixels calculated in S241 to the second model learned in S232. That is, the second application unit 156 inputs the focus values of the plurality of pixels derived as a result of the second focus measurement operation to the second model and fits a plurality of focus curves.

Next, in S243, the extraction unit 157 of the control unit 150 extracts a pixel position in the corrected target image having the maximum focus value from the plurality of fitted focus curves. For example, in FIG. 7 , a pixel position corresponding to number 10 among the focus curves of the fitted red graph has the maximum focus value (about 8.6). In this case, the extractor 157 may extract the pixel position of the number 10 as a pixel having the maximum focus value.

Next, in S244, the collection unit 158 of the controller 150 collects corresponding image information by performing depth estimation according to the DFF technique or 3D shape restoration according to the SFF technique based on the extracted pixel position. That is, using the maximum focus value obtained through S241 to S243 for the plurality of corrected target images, the collection unit 158 performs depth estimation according to the DFF technique or SFF on the corrected target image. It is possible to perform 3D shape restoration according to.

At this time, DFF and SFF find the position of the lens that is in focus of the image and obtain the distance of the in-focus part according to the lens formula. That is, DFF and SFF find the position where the focus value is maximized on a simple plane perpendicular to the optical axis of the lens in order to calculate the degree of focus, and calculate the depth (distance) or three-dimensional shape of the object. can be measured

13 illustrates an example of a principle of forming a depth estimation or a 3D reconstructed image by depth from focus (DFF) or shape from focus (SFF).

That is, in FIG. 13 , a focused image (P′) of the light source P located at a distance u from the lens L is obtained at a position v from the lens L. The luminous intensity of the focused image is proportional to the luminous intensity of the object, and the distance of the object has a relationship with the position of the focused image according to the following equation.

However, f is the focal length, u is the distance from the lens plane to the object, and v is the distance to the focused image. That is, given the luminous intensity and position of an image in focus from the lens formula, the luminous intensity and position of the object are determined and depth information or 3-dimensional information of the object can be obtained.

In summary, for DFF or SFF, a focus measurement operator SML is applied to each pixel of 2D images obtained by randomly sampling 2D images in S241. Then, in S242, focus data of pixels existing along the optical axis are fitted using a Gaussian process regression technique. Thereafter, a depth map of an object in an image may be obtained by obtaining a pixel position having a maximum value from the fitted focus curve in S243 and applying DFF or SFF to the pixel position.

14 shows a comparison result graph according to the prior art and the present invention.

Meanwhile, in FIG. 14, the conventional technique of performing SML on all pixels of an image is shown as a dotted line graph, and the focus curve obtained with only some pixels in the image based on Gaussian process regression according to the image information collection method of the present invention is a red graph Same as

Referring to FIG. 14, the present invention more accurately obtains a pixel position having the maximum focus value in an image even with a small amount of data, and collects image information according to optimal depth estimation or optimal 3D shape restoration using this. can do.

That is, the present invention has the advantage of being able to collect image information according to optimal depth estimation or optimal 3D shape restoration with only a small amount of data by using the Gaussian process regression technique. In addition, the present invention has the advantage of being able to collect image information according to optimal depth estimation or optimal 3D shape restoration by effectively removing jitter noise included in images when image information is collected in a rain control system or the like. there is In addition, the present invention can improve the accuracy of depth estimation or 3D shape restoration by more accurately correcting the frame position of the target image changed according to jitter noise by applying long and short term memory technology, which is one of deep learning technologies. There is an advantage.

In the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and equivalents thereof.

100: image acquisition device 110: input unit
120: communication unit 130: display
140: memory 150: control unit
151: data generator 152: first learning unit
153: first application unit 154: focus measurement operation unit
155: second learning unit 156: second application unit
157: extraction unit 158: collection unit

Claims (11)

A method for collecting image information performed by an electronic device,
For a plurality of first training images having different frame positions in the image sequence, n pieces of training position data (where n is a natural number of 2 or more) to which noise affecting the position is added for each first training image are generated. doing;
Performing learning according to a machine learning technique on a deep learning neural network each having an input layer inputting the training location data and an output layer outputting original location data for the original location of the corresponding first training image;
generating n object position data to which noise affecting the position of each target image is added for a plurality of target images having positions of different frames in the image sequence; and
inputting each object location data to the first model of the learned neural network, extracting the estimated location data for the estimated locations of a plurality of target images, and correcting the location of each target image using the extracted estimated location data; step;
performing depth estimation or 3D shape restoration using each corrected target image;
How to include.
According to claim 1,
The deep learning neural network is a long short-term memory (Long Short-Term Memory) neural network.
According to claim 1,
In the step of performing depth estimation or 3D shape restoration, a focus value of each target image is derived based on the corrected position of each target image, and the depth estimation or 3D shape restoration is performed using the derived focus value. A method comprising the steps of performing
According to claim 1,
The step of performing depth estimation or 3D shape restoration is based on the focus value of each pixel derived as a result of the first focus measurement operation on a plurality of pixels in the second training image and the corresponding focus value. Performing the depth estimation or 3D shape reconstruction using a second model in which learning according to machine learning of Gaussian Process Regression has been performed based on a fitting function of an estimated focus curve.
According to claim 4,
The step of performing the depth estimation or 3D shape restoration,
performing a second focus measurement operation on a plurality of pixels in the corrected target image;
fitting a plurality of focus curves by inputting focus values of a plurality of pixels derived as a result of the second focus measurement operation to the second model;
extracting a pixel position in the corrected target image having a maximum focus value from a plurality of fitted focus curves; and
performing the depth estimation or 3D shape restoration based on the extracted pixel position;
How to include.
According to claim 6,
The first and second focus measurement operations use a Sum of Modified Laplacian (SML).
According to claim 6,
The SML is a method using the following formula.

(However, I(x, y) is the gray level brightness at the pixel of (x, y), W is the video window size)
According to claim 4,
The learning method for the second model,
Selecting a kernel function of a probability distribution for a fitting function as k(i, i'), which is a square exponential kernel;

(However, i and I' are the inputs of the square exponential kernel function, | | | | is the Euclidean distance)
For the initial probability distribution for the fitting function, setting the mean to 0 and setting the kernel function to k(x ₀ , x ₀ '), where x ₀ and x ₀ ' are axes for pixel locations in the target data a set of values on the x-axis); and
For the updated probability distribution for the fitting function, setting the mean to m _g (x ₀ ) and the kernel function to k _g (x ₀ , x ₀ ');

(However, x _t is a set of x-axis values, which is an axis for pixel position values in training data, and x _t 'is a set of y-axis values, which is an axis for focus values in training data)
How to include.
According to claim 5,
The performing of the second focus measurement operation includes randomly sampling and extracting a plurality of pixels from the corrected target image, and performing a second focus measurement operation on the plurality of extracted pixels.
According to claim 1,
The method of claim 1, wherein the noise has a non-Gaussian distribution.
Memory; and
A control unit for controlling depth estimation or 3D shape restoration using information stored in a memory; includes,
The control unit,
For a plurality of first training images having different frame positions in the image sequence, generation of n (n is a natural number greater than or equal to 2) position data for learning to which noise affecting the position is added for each first training image. to control,
For a deep learning neural network having an input layer inputting each of the learning location data and an output layer outputting in situ data for the original location of the corresponding first training image, learning according to a machine learning technique is controlled,
For a plurality of target images having positions of different frames in the image sequence, controlling the generation of n target position data to which noise affecting the position of each target image is added;
Each target position data is input to the first model of the trained neural network, the estimated position data for the estimated positions of a plurality of target images is extracted, and position correction of each target image is controlled using the extracted estimated position data. and
An apparatus for controlling depth estimation or 3D shape restoration using each corrected target image.

Images