KR20220056599A - Learning-data generation method and apparatus for improving cctv image quality - Google Patents

Abstract

The present invention relates to a method for a computing device operated by at least one processor to generate learning data. The method comprises the following steps of: collecting low-definition and high-definition images photographed in the same photography condition; extracting frames with respect to each of the low-definition and high-definition images, and calculating photographing time information for each extracted frame; calculating similarities with high-definition frames for each low-definition frame, and matching a low-definition frame and a high-definition frame having the highest similarity to generate a plurality of pieces of candidate learning data; and extracting a difference value of photographing time information between a low-definition frame and a high-definition frame for each piece of candidate learning data, and generating candidate learning data having a difference value with the highest frequency among difference values, as final learning data. Therefore, the present invention is capable of generating learning data reflecting the characteristics of actual images.

Description

Learning data generation method and device for image quality improvement

The present invention relates to a method and apparatus for generating training data for an image quality improvement model.

With the development of various image content services, the demand for image quality improvement has increased. In particular, as deep learning has recently emerged, technologies for improving image quality without expensive equipment are being actively studied.

In order to improve the performance of a deep learning-based system, it is important to configure the learning data well and to perform learning reflecting the characteristics. However, in the case of image quality improvement technology, there is low image quality (input), but there is no high quality that is the correct answer, so it is difficult to teach its characteristics.

Since the conventional deep learning-based image quality improvement technology does not have an accurate answer for the low image quality to overcome the above problem, a method of converting a high-quality image to a low-quality through digital conversion to create a high-quality-low-quality learning set and restoring it again use. This method is effective for animation/media images with regular deformation, etc., and is applied and used in services.

However, images used in various real life such as CCTVs, black boxes, drones, and medical images have characteristics different from general contents due to distortion and deterioration due to changes in illuminance and communication environments. In images in such an environment, the effect of the conventional image quality improvement technology is insignificant, and in order to improve the image quality of an image used in an actual environment to a high quality, a technology for learning the characteristics of an actual image is required.

The task to be solved is to match the low-quality frame and the high-quality frame based on the similarity, and when the difference in shooting time between the matched low-quality frame and the high-quality frame corresponds to the delay time between the low-quality image and the high-quality image, the matched frames are used as learning data To provide a method and apparatus for generating

The task to be solved is to provide a technology for generating learning data for a deep learning model that improves the image quality while reflecting the characteristics of the actual image by matching the high-quality image and the low-quality image captured by a real camera.

According to an embodiment of the present invention, there is provided a method for a computing device operated by at least one processor to generate learning data, the method comprising: collecting low-quality images and high-quality images actually photographed under the same shooting conditions; Extracting each frame, calculating shooting time information for each extracted frame, calculating similarities with high-quality frames for each low-quality frame, and matching the low-quality frame and the high-quality frame with the highest similarity to a plurality of candidate learning data and extracting the difference value of the shooting time information between the low-resolution frame and the high-quality frame for each candidate learning data, and generating the candidate learning data having the most frequent difference value among the difference values as the final learning data. includes

In the collecting step, it is possible to parse the VOD information in which the low-resolution image and the high-quality image are stored, collect format information of each image, and check the start time and the number of frames per second of the entire image based on the format information.

The calculating of the shooting time information includes checking the start time of the image included in the format information, calculating the elapsed time from the start time to the extracted frame, and adding the elapsed time and the start time to the shooting time information for the frame. can be calculated as

In the step of generating the final training data, a difference value having the highest frequency may be estimated as a delay time between a low-quality image and a high-quality image.

The step of generating a plurality of candidate training data includes calculating an average similarity of high-quality frames with respect to one low-quality frame, comparing the average similarity with the highest similarity, and, when a difference value is less than or equal to a threshold, selects a low-quality frame as candidate training data can be excluded from

According to an embodiment of the present invention, there is provided an apparatus for generating learning data, comprising: a memory; and at least one processor executing instructions of a program loaded into the memory, wherein the program has a quality captured under the same shooting conditions. Extracting frames from different actual images, grouping frames having the same image quality and classifying them into a reference group and a comparison group, calculating a similarity for each frame of the reference group with the frames of the comparison group Matching frames of different image quality and estimating delay time between images of different image quality based on time difference values of shooting time between the matched frames, and generating a matched frame having a time difference value by the delay time as training data instructions that are described to perform the steps of

The shooting time may be a time obtained by adding the elapsed time and the start time by checking the start time of the image included in the format information of each image having different image quality, calculating the elapsed time from the start time to the extracted frame.

In the step of generating the learning data, the time difference value for the shooting time between frames of the comparison group matched for each frame of the reference group is calculated, and the time difference value having the highest frequency among the calculated time difference values is the delay time can be estimated as

In the grouping, frames of an image having a relatively small number of frames may be selected as a reference group, and frames of an image different from that of the image of the reference group may be selected as a comparison group.

In the step of generating the training data, the average similarity with the frames of the comparison group is calculated for each frame of the reference group, and the average similarity and the highest similarity are compared for each frame of the reference group, and the comparison result is a difference value less than a threshold If you have , you can exclude the frame.

According to an embodiment, in order to learn a deep learning model that improves the quality of an actual captured image, a high-quality image and a low-quality image are matched to generate learning data reflecting the characteristics of the actual image.

According to the embodiment, by estimating the total delay time of the image based on the time difference value between the low-resolution frame and the high-quality frame, and generating training data with high similarity in consideration of the delay time, a lot of training data optimized for the deep learning model can be obtained

1 is a structural diagram of an apparatus for generating learning data according to an embodiment of the present invention.
2 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a process of generating frame information for an extracted frame according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of calculating a degree of similarity between a low-quality frame and a high-quality frame according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a time difference calculated based on similarity measurement according to an embodiment of the present invention.
6 is a hardware configuration diagram of a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program to be executed in combination with the hardware is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the method of operation of the present invention described with reference to the drawings, and is combined with hardware such as a processor and a memory device to execute the present invention.

As used herein, “transmission or provision” may include not only direct transmission or provision, but also transmission or provision indirectly through another device or using a detour path.

In this specification, expressions described in the singular may be construed in the singular or plural unless an explicit expression such as “a” or “single” is used.

1 is a structural diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 1, the learning data generating device 200 includes an image collector 210 that collects actually photographed low-quality images and high-quality images, a frame extractor 220 that extracts frames from the collected images and stores them, and It includes a learning data generator 230 that generates learning data by matching the low-quality image and the high-quality image.

For description, the image collector 210 , the frame extractor 220 , and the learning data generator 230 are named and called, but these are computing devices operated by at least one processor. Here, the image collector 210 , the frame extractor 220 , and the learning data generator 230 may be implemented in one computing device or distributed in separate computing devices. When distributed in a separate computing device, the image collector 210 , the frame extractor 220 , and the learning data generator 230 may communicate with each other through a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

The learning data generating device 200 is a device for generating learning data for learning a deep learning model that improves the quality (image quality) of an image. create

At this time, in order to input to the deep learning model, images with different image quality under the same shooting conditions must be secured as training data.

However, even if a low-quality image and a high-definition image captured under the same shooting conditions are requested, the actually received stored images may not match the shooting time information. For example, the shooting time information of each captured image may be differently stored due to a difference in performance of a storage device in which the captured low-quality image and the high-quality image are stored, a difference in network performance, a difference in performance of the camera itself, etc., for example.

The learning data generating apparatus 200 receives a low-resolution image and a high-quality image as a plurality of transport stream (TS) pieces including the requested shooting time. At this time, the transmitted TS information is in a format format and is configured differently for each video stream, so that it is possible to check the shooting time information based on the TS information.

Accordingly, the learning data generating apparatus 200 receives the actually captured low-quality image and high-quality image as a plurality of TS pieces, so the process of matching the same low-quality image and high-quality image with each other based on the TS information of each TS fragment is essential. am.

However, as described above, due to an error in the stored photographing time information, even if the actual photographing time information is the same, when visually confirmed, the high-quality image and the low-quality image may be different from each other.

Therefore, in order to generate the images captured under the same shooting conditions, such as the same time, the same subject, and the same camera angle of view, as learning data, the learning data generating apparatus 200 calculates the shooting time, Based on the training data is generated.

First, the image collector 210 requests and receives a recorded image from the image platform 100 in which an actual photographed image is stored.

Here, the image platform 100 is a platform that stores and manages images captured by one or more cameras in an image storage (database), and each image in the image storage is captured together with data such as camera unique ID and shooting time. is saved

Here, the camera-specific ID means an ID set based on a camera type, a photographing module, image quality, and the like.

The camera includes a photographing module for photographing a high-quality image and a photographing module for photographing a low-quality image, so that images having different image quality can be photographed at the same time in real time. Therefore, images having different image quality can be secured under the same shooting condition.

In addition, the camera includes a cctv, a camera inside a black box, a camera mounted on a drone, a camera mounted on a user terminal, and the like, and is not limited to a specific camera.

The image collector 210 receives a VOD storage URL from the video platform 100 when video on demand (VOD) information including a camera ID (capturing module ID), an image capture time, and the like is requested.

The image collector 210 parses the corresponding VOD storage URL to collect TS information including the TS address, start time, and total TS playback time information of the actual image.

In addition, the image collector 210 may transmit the collected TS information to the frame extractor 220 in real time or generate a TS list for the entire collected image and then sequentially transmit it to the frame extractor 220 .

The frame extractor 220 receives the received TS information, collects an actual image through the TS address, and extracts a frame for the image. In addition, the frame extractor 220 may generate frame information by calculating a shooting time for each extracted frame.

The start time collected by the image collector 210 is the start time information of the image for the TS, and the frame shooting time is calculated by obtaining the elapsed time from the TS start time to the extracted frame and adding the elapsed time and the TS start time.

Specifically, the elapsed time is the time required per frame (1/ FPS) multiplied by the current frame index, and expressed as an equation as shown in Equation 1 below.

[Equation 1]

Frame Time = (1/ FPS * Frame idx) + TS start time

The image collector 210 calculates the shooting time for each frame through Equation 1 and stores it in each frame information (Frame Info).

In this case, the image collector 210 may store the frames for the low-quality image and the frames for the high-quality image in respective storages based on the collected camera ID.

The training data generator 230 may determine the similarity between the low-resolution image and the high-definition image with respect to the extracted frames, and match the low-quality frame and the high-quality frame at the same viewpoint having a high degree of similarity to generate the training data.

The training data generator 230 may be divided into a reference group (A) and a comparison group (B) to generate training data by matching between the collected low-quality frames and high-quality frames.

Accordingly, the training data generator 230 calculates a similarity between frames included in the comparison group B for each frame included in the reference group A, and selects a frame having the highest calculated similarity.

And the learning data generator 230 calculates a time difference value by comparing the shooting time of the frame included in the reference group (A) with the shooting time of the frame having the highest selected similarity.

In other words, since the time difference with the frame of the selected comparison group B is calculated for each frame included in the reference group A, time difference values are calculated based on the number of frames in the reference group A.

The training data generator 230 may select a time difference value having the highest frequency from among the calculated time difference values, extract low-resolution frames and high-quality frames having the corresponding time difference values, and generate them as training data.

In other words, the learning data generator 230 generates data having the same shooting time difference as learning data. As such, the training data generator 230 may generate training data using the frame of the comparison group matched one-to-one with the frame of the reference group and store it in the database.

Although it is illustrated in FIG. 1 that the storage DBs are included in the learning data generating device 200, the present invention is not limited thereto, and the storage DB may be implemented in a separate device or server.

Hereinafter, a method for generating the training data by the training data generating apparatus using FIGS. 2 and 3 will be described in detail.

2 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

As shown in FIG. 2 , the learning data generating apparatus 200 collects low-quality images and high-quality images captured under the same shooting conditions ( S110 ).

The learning data generating apparatus 200 may collect a low-quality image and a high-quality image captured under the same shooting conditions, such as a shooting time and a camera ID, through an image platform.

Here, the image platform is a platform that manages a storage for storing captured images, stores and manages captured images for each camera ID, and provides images requested by the user.

The learning data generating apparatus 200 may receive a VOD storage URL corresponding to the VOD information requested from the video platform. Then, the training data generating apparatus 200 parses the VOD storage URL to obtain TS information including the TS address, start time, and total TS reproduction time information of the actual video.

The learning data generating apparatus 200 extracts a frame of an image by sequentially acquiring an actual image based on the acquired TS information.

In this case, the learning data generating apparatus 200 may extract a frame from each of the low-quality image and the high-quality image at the same time or sequentially extract the frames of the low-quality image and then extract the frame of the high-definition image.

Next, the learning data generating apparatus 200 extracts frames and shooting time information of each frame from the low-quality image (S120).

The learning data generating apparatus 200 configures frame information by calculating a shooting time for each extracted frame. Here, the frame information includes, but is not limited to, a camera ID, a frame index (idx), an actual frame image, and a calculated shooting time.

Here, the frame index (idx) obtains the total number of frames of the TS and indicates a number set based on the total number of frames for each extracted frame.

In addition, the training data generating apparatus 200 may calculate the shooting time for each frame by adding the elapsed time to the TS start time for each frame.

The learning data generating apparatus 200 may store frame information secured based on the camera ID in the database.

In the same manner as in the method performed in step S120, the learning data generating apparatus 200 extracts frames and shooting time information of each frame from the high-definition image (S130).

The learning data generating apparatus 200 configures frame information for a high-quality image in the same way, and stores it in a database based on the camera ID.

Although steps S120 and S130 have been sequentially described for convenience of explanation, the corresponding steps may be performed simultaneously or a high-quality image may be performed first, and then a low-quality image may be performed.

The learning data generating apparatus 200 calculates each similarity and average similarity with high-quality frames for each low-resolution frame (S140).

In this case, the learning data generating apparatus 200 may select a reference group for the low-quality image and the high-quality image.

The learning data generating apparatus 200 may search for frame information of each image, select the same start time and end time between two images as a collection time, and select a reference group and a comparison group necessary for image comparison. In addition, frames of an image having a relatively small number of collected frames within the collection time are selected as a reference group.

In general, a low-quality image is generally selected because the frame rate per second (FPS) is lower than that of a high-quality image taken at the same time for a low-quality image.

However, depending on circumstances, if the number of frames in the high-definition image is smaller, the high-definition image may be selected as the reference group, but hereinafter, the reference group will be described assuming a low-quality image.

The training data generating apparatus 200 calculates a similarity between a plurality of high-definition frames for each of the plurality of low-quality frames, and calculates an average similarity based on similarities between a plurality of high-definition frames with respect to one low-quality frame.

Next, the learning data generating apparatus 200 matches the high-definition frame having the highest similarity for each frame, checks whether the matched frame has a false positive section, and excludes it (S150).

The learning data generating apparatus 200 may generate candidate learning data by matching a low-quality frame and a high-quality frame having a high degree of similarity. The training data generating apparatus 200 generates a plurality of candidate training data through a high-quality frame matched for each low-resolution frame.

In this case, the learning data generating apparatus 200 may identify the false positive section by comparing the average similarity and the highest similarity of a plurality of high-quality frames with respect to one low-quality frame.

Here, the false positive section is a condition for selecting only valid training data, and represents a case in which the average similarity and the highest similarity have a slight difference.

For example, in the case of a fixed image without change, the same image is repeated for a predetermined time. In this case, even if the low-quality frame and the high-quality frame captured at unequal points of view are matched, the similarity is calculated to be high, and thus the accuracy of the training data is lowered, and errors may occur.

Therefore, the learning data generating apparatus 200 compares the average similarity of each frame of the reference group with the highest similarity, and if a difference value is less than or equal to a threshold value, it may be estimated as a false positive section and the corresponding frame may be excluded.

In this case, the definition of the difference value less than the threshold can be set later by the user, and for example, it may mean a case where the integers of the similarity values are the same or the difference is less than a decimal point.

Next, the learning data generating apparatus 200 calculates a difference value with respect to the photographing time information between the matched low-quality frame and the high-quality frame ( S160 ).

The training data generating apparatus 200 may calculate a difference value of the shooting time information of the high-quality frame from the shooting time information of the low-quality frame for each candidate learning data (matched frames).

The training data generating apparatus 200 selects a time difference value having the highest frequency among the calculated difference values and generates a low-resolution frame and a high-quality frame having the corresponding time difference value as training data (S170).

The learning data generating apparatus 200 records the time difference value calculated for each matched data, and counts when the same time difference value is recorded. Accordingly, by selecting the most counted difference value, matching data having the corresponding difference value is generated as the final training data.

Accordingly, the final training data have the same time difference value, and the most data among the matched low-quality image and high-quality image can be secured as training data.

In this way, the learning data generating apparatus 200 may generate the final learning data by matching the low-quality frame and the high-quality frame photographed at the same time point with each other.

In other words, the training data generating apparatus 200 may generate a plurality of candidate training data based on the degree of similarity, and may generate final training data based on a time difference value among the plurality of candidate training data.

3 is an exemplary diagram illustrating a process of generating frame information for an extracted frame according to an embodiment of the present invention.

As shown in FIG. 3 , the learning data generating apparatus 200 receives the VOD storage URL corresponding to the VOD information requested from the video platform 100 .

The training data generating apparatus 200 parses the VOD storage URL to obtain TS information including a TS address of an actual video, a start time, total TS playback time information, and total FPS. And the learning data generating apparatus 200 acquires an actual image based on the TS address.

The learning data generating apparatus 200 generates frame information while extracting frames from each low-quality image or high-quality image.

Since the acquired TS information is information about a low-quality image or a high-quality image, the learning data generating apparatus 200 includes a camera ID (camID), a frame idx, an actual frame, and shooting time information for each frame separately for each frame. create information

Here, it is possible to classify whether the corresponding frame is for a high-quality image or a low-quality image through the camera ID, and it is possible to classify which frame the corresponding frame is for through the frame idx.

Since the photographing time information for the frame can be calculated using Equation 1 described above, a repeated description will be omitted.

When the learning data generating apparatus 200 generates frame information for each frame of each image, it is classified and stored in the A frame storage DB or the B frame storage DB according to the high or low quality of the image.

As described above, the apparatus 200 for generating training data generates frame information for each frame of each image, so that a low-quality frame and a high-quality frame can be matched with each other based on the corresponding frame information.

4 is an exemplary diagram for explaining a process of calculating a similarity between a low-quality frame and a high-quality frame according to an embodiment of the present invention, and FIG. 5 is an example showing a time difference calculated based on similarity measurement according to an embodiment of the present invention It is also

As shown in Fig. 4, there are M frames in the reference group, and for each frame, A1, A2, ... , the calculated imaging times for AM are described.

And there are N frames in the comparison group, and for each frame, B1, B2, ... , the calculated imaging times for BN are described. (M and N are different natural numbers)

The training data generating apparatus 200 calculates a similarity by comparing the frames of the comparison group for each frame of the reference group having a relatively small number of frames.

Here, the similarity between frames may be calculated based on a formula for calculating a peak signal-to-noise ratio (PSNR). Since it is measured on a log scale between frames, the unit of [db] is mainly used, and the smaller the loss, the higher the value.

This similarity measurement method is an example, and does not necessarily use the maximum signal-to-noise ratio. Mean square error (MSE), root mean square error (RMSE), structural similarity index (Structural Similarity Index, SSIM), etc., may use a similarity measurement method between frames most suitable for the situation.

In detail, with respect to A1, a total of N similarity scores are calculated for each frame from B1 to BN, and a similarity value having the largest average similarity and similarity score for the N similarities is calculated.

In addition, the learning data generating apparatus 200 calculates a difference value between the photographing time of the comparison group frame having the greatest similarity and the photographing time of A1.

If the process is repeated for all frames included in the reference group, a result value as shown in FIG. 5 is obtained.

The matched frames of FIG. 5 are candidate training data matched on the basis of similarity, and the final training data is generated based on a time difference among the candidate training data.

In this case, the learning data generating apparatus 200 determines a false positive section and excludes if the integer values are the same using only the integer values of the average similarity and the highest similarity.

Alternatively, when the learning data generating apparatus 200 has a difference value between the average similarity and the highest similarity equal to or less than a threshold value, the learning data generating apparatus 200 may determine it as a false positive section and exclude it. For example, if the threshold is set to less than a decimal point, a case in which the average similarity and the highest similarity have a difference value less than a decimal point may be determined as a false positive section.

In addition, a criterion for judging a false positive section may be set, such as rounding the similarity value to a decimal point.

The learning data generating apparatus 200 may compare the time difference for each frame of each reference group based on the comparison result between the remaining frames except for and estimate the most frequent time difference value as the total delay time.

Accordingly, matching data having the most frequent time difference value can be estimated as matching data to which the entire delay time is applied.

In FIG. 5 , the frequency of 0.00423 is 3, the frequency of 0.00342 is 2, and the time difference of 0.00423 is estimated as the delay time.

In this way, the training data generating apparatus 200 estimates the time difference as the delay time, and among the frames having the largest similarity value matched between the reference group and the comparison group, data having the corresponding delay time can be generated as training data. there is.

6 is a hardware configuration diagram of a computing device according to an embodiment of the present invention;

As shown in FIG. 6 , the hardware of the computing device 300 may include at least one processor 310 , a memory 320 , a storage 330 , and a communication interface 340 , and may be connected through a bus. there is. In addition, hardware such as an input device and an output device may be included. The computing device 300 may be loaded with various software including an operating system capable of driving a program.

The processor 310 is a device for controlling the operation of the computing device 300 and may be various types of processors 310 that process instructions included in a program, for example, a central processing unit (CPU), an MPU (Central Processing Unit) It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 320 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 310 . The memory 320 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 330 stores various data and programs required for executing the operation of the present invention. The communication interface 340 may be a wired/wireless communication module.

According to an embodiment, in order to learn a deep learning model that improves the quality of an actual captured image, a high-quality image and a low-quality image are matched to generate learning data reflecting the characteristics of the actual image.

In addition, by correcting the actual shooting time between the low-quality image and the high-quality image and matching frames captured at the same time, similarity accuracy between the matched frames can be secured even in an image in which a fast motion is captured.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. is within the scope of the right.

Claims (10)

A method of generating learning data by a computing device operated by at least one processor,
Collecting low-quality images and high-quality images actually taken under the same shooting conditions;
extracting respective frames from the low-resolution image and the high-quality image, and calculating shooting time information for each extracted frame;
Calculating similarities with high-definition frames for each low-quality frame, and generating a plurality of candidate learning data by matching the low-quality frame and the high-definition frame having the highest similarity; and
extracting a difference value of shooting time information between the low-resolution frame and the high-quality frame for each of the candidate training data, and generating candidate training data having the highest frequency difference among the difference values as final training data;
A method of generating training data including In claim 1,
The collecting step is
A method of generating training data that parses the VOD information in which the low-resolution image and the high-definition image are stored, collects format information of each image, and checks the start time, the image address, and the number of frames per second of the entire image based on the format information . In claim 2,
Calculating the shooting time information includes:
Learning to check the start time of the image included in the format information, calculate the elapsed time from the start time to the extracted frame, and calculate the time obtained by adding the elapsed time and the start time as the shooting time information for the frame How to generate data. In claim 1,
The step of generating the final learning data is,
A method of generating learning data for estimating a difference value with the highest frequency as a delay time between the low-quality image and the high-quality image. In claim 1,
The generating of the plurality of candidate learning data includes:
Calculate the average similarity of the high-quality frames to one low-quality frame, and compare the average similarity with the highest similarity. How to. As a device for generating learning data,
memory, and
at least one processor executing instructions of a program loaded into the memory;
the program is
Extracting each frame from actual images of different image quality captured under the same shooting condition, grouping the frames having the same image quality and classifying them into a reference group and a comparison group;
Calculating a degree of similarity with frames of the comparison group for each frame of the reference group and matching frames of different image quality with the highest degree of similarity; and
Estimating a delay time between images having different image quality based on time difference values of shooting times between the matched frames, and generating a matched frame having the time difference value by the delay time as training data.
Learning data generating device comprising instructions described to execute. In claim 6,
The shooting time is
An apparatus for generating learning data, which is a time obtained by adding the elapsed time and the start time by checking the start time of an image included in the format information of each image having different image quality, calculating the elapsed time from the start time to the extracted frame. In claim 7,
The step of generating the learning data is,
For each frame of the reference group, a time difference value for a shooting time between frames of a matched comparison group is calculated, and a time difference value having the highest frequency among the calculated time difference values is generated as the delay time. Device. In claim 6,
The grouping step is
A learning data generating apparatus for selecting frames of an image having a relatively small number of frames as a reference group, and selecting frames of an image different from that of the image of the reference group as a comparison group. In claim 6,
The step of generating the learning data is,
The average similarity with the frames of the comparison group is calculated for each frame of the reference group, and the average similarity and the highest similarity are compared for each frame of the reference group. Learning data generation device to exclude.

Images