JP2022129792A - Area conversion apparatus, area conversion method, and area conversion system - Google Patents

Abstract

To provide area conversion means for appropriately changing an aspect ratio of an image while maintaining quality and semantic information of the image to facilitate highly accurate object detection.SOLUTION: An area conversion apparatus comprises: an image frame identification unit that identifies a target image frame to be subjected to area conversion; an area-of-interest detection unit that detects an area of interest in the target image frame and calculates reliability of a processing operation for extracting an area-of-interest image including the area of interest from the target image frame; an image frame processing unit that extracts the area-of-interest image using the processing operation if the reliability of the processing operation satisfies a predetermined reliability criterion; a background synthesizing means determining unit that determines background synthesizing means for converting the target image frame into a predetermined aspect ratio if the reliability of the processing operation does not satisfy the predetermined reliability criterion or if the area-of-interest image does not satisfy a predetermined aspect ratio criterion; and a background synthesizing unit that converts the target image frame to the predetermined aspect ratio using the background synthesizing means.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to a domain conversion device, a domain conversion method, and a domain conversion system.

In recent years, with the progress of IT, a large number of sensors have been installed in society, and an extremely large amount of data has been accumulated. Under such circumstances, various measures for utilizing the accumulated image data are being considered. In particular, as video content such as photographs, moving pictures, and images increases, a machine learning model that can freely detect and accurately identify objects in the video is desired.

Deep convolutional neural networks are known as one of the machine learning models that can recognize arbitrary objects and activities with high accuracy. A deep convolutional neural network expresses nerve cells (neurons) and their connections in the human brain, that is, a neural network, with a mathematical model called an artificial neuron, and object detection by such a neural network is automatic. It has been applied to various fields such as driving, natural language processing, medical research, and robotics.

However, the structure of a conventional deep convolutional neural network does not have so-called shift invariance, and depending on the aspect ratio of the input image, the accuracy of object detection may decrease.

One of means for changing the aspect ratio of an image is, for example, EP 1968008 (Patent Document 1).

Patent Literature 1 describes a method for reconstructing a content-aware image, which includes enlarging the size of an image using image scaling and using it as a source image, and generating an energy image from the source image according to an energy function. determining one or more seams from the energy image according to a minimization function such that each seam extending from one edge of the source image to the opposite edge has a minimum energy; A method for content-aware image reconstruction comprising subtracting from a source image to obtain a target image preserving the content and rectangular shape of the source image, and reducing the source image to the size of the original image." is described.

European Patent No. 1968008

In Patent Document 1, the aspect ratio is converted by reconstructing an image using a seam, which is a set of connected pixels, and an energy function. The seams are found by minimizing the energy using dynamic programming.

However, the image reconstruction means using the seam and energy function described in Patent Document 1 requires a lot of computing resources for dynamic programming, and causes disturbance in the image, which may lead to quality deterioration. There is Furthermore, the determination of the semantic content in Patent Document 1 is performed based on the magnitude of the gradient of the image and the energy function. It is difficult to define an energy function that captures well, and important semantic information may be lost.

Therefore, the present disclosure appropriately changes the aspect ratio of the image while maintaining the quality and semantic information of the image by applying a background synthesizing means that considers the semantic content of the image, thereby achieving high-precision object detection. It is an object of the present invention to provide a region conversion means that enables

In order to solve the above problems, one of the representative domain conversion devices of the present disclosure is a domain conversion device, which identifies a target image frame to be domain-converted from an image sequence. a region-of-interest detection unit that detects a region of interest in the target image frame and calculates reliability of a processing operation for extracting a region-of-interest image including the region of interest from the target image frame; an image frame manipulator for extracting the region of interest image from the target image frame using the manipulating action if the confidence of the action meets a predetermined confidence criterion; and If a degree criterion is not met, or if the region of interest image extracted from the target image frame does not meet a predetermined aspect ratio criterion, adding or deleting background pixels to or from the target image frame will reduce the target image frame. Background pixels are added to or deleted from the target image frame using a background synthesis means determination unit that determines a background synthesis means for converting to a predetermined aspect ratio from a plurality of background synthesis means candidates, and the background synthesis means. a background synthesizing unit for generating a final image obtained by converting the target image frame to the predetermined aspect ratio.

According to the present disclosure, by applying a background synthesizing means that considers the semantic content of an image, while maintaining the quality and semantic information of the image, the aspect ratio of the image is appropriately changed, and high-precision object detection is achieved. It is possible to provide a region conversion means that enables
Problems, configurations, and effects other than the above will be clarified by the description in the following modes for carrying out the invention.

FIG. 1 is a diagram showing an example of deterioration in object detection accuracy by a neural network when the aspect ratio is inappropriate. FIG. 2 is a block diagram of a computer system for implementing embodiments of the present invention. FIG. 3 is a diagram illustrating an example of the configuration of a domain conversion system according to an embodiment of the present disclosure; FIG. 4 is a diagram for explaining background synthesis, which is an example of area conversion according to an embodiment of the present disclosure. FIG. 5 is a flowchart showing the flow of area conversion processing according to the embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of processing of the region-of-interest detection unit according to the embodiment of the present disclosure. FIG. 7 is a diagram showing an example of the reliability of the machining operation calculated by the region-of-interest detection unit according to the embodiment of the present disclosure. FIG. 8 is a diagram illustrating another example of reliability of machining operation calculated by the region-of-interest detection unit according to the embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of processing of a background synthesizing means determination unit according to an embodiment of the present disclosure; FIG. 10 is a diagram showing an example of first background synthesizing means according to an embodiment of the present disclosure. FIG. 11 is a diagram showing an example of second background synthesizing means according to an embodiment of the present disclosure. FIG. 12 is a diagram showing an example of a third background synthesizing means according to an embodiment of the present disclosure. FIG. 13 is a diagram showing an example of fourth background synthesizing means according to an embodiment of the present disclosure. FIG. 14 is a diagram for explaining a Gaussian process regression model according to an embodiment of the present disclosure; FIG. 15 is a diagram showing an example of a fifth background synthesizing means according to an embodiment of the present disclosure. FIG. 16 is a diagram showing an example of a sixth background synthesizing means according to an embodiment of the present disclosure. FIG. 17 is a diagram showing an example of a seventh background synthesizing means according to an embodiment of the present disclosure. FIG. 18 is a diagram showing an example of eighth background synthesizing means according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the present invention is not limited by this embodiment. Moreover, in the description of the drawings, the same parts are denoted by the same reference numerals.

First, with reference to FIG. 1, an example in which the accuracy of object detection by a neural network decreases when the aspect ratio is inappropriate will be described.

FIG. 1 is a diagram showing an example of deterioration in object detection accuracy by a neural network when the aspect ratio is inappropriate.

As described above, the structure of the conventional deep convolutional neural network does not have so-called shift invariance, and when the aspect ratio of the input image is inappropriate, the accuracy of object detection decreases. There is
In general, a deep convolutional neural network performs object detection after transforming an input image into a predetermined aspect ratio. As means for converting the aspect ratio of an image, for example, resizing, cropping, and the like are known.

Image resizing involves expanding or contracting a region of interest in an image without considering the semantic content of the image. However, if the image is processed using the resizing means, there is no guarantee that the aspect ratio of the target area to be detected is maintained, and objects existing in the target area may be distorted or deformed. After that, when the neural network receives an image whose aspect ratio has been converted by resizing, the detection accuracy is limited by the distortion or deformation of the object.
As an example, as shown in FIG. 1, if the target area 102 in the image 101 is converted to a predetermined aspect ratio by resizing 103, the aspect ratio of the object in the converted image 104 will not be maintained and the object will be deformed. . This deformation may reduce the object detection accuracy of the neural network 105 .

Also, in cropping, a region of a predetermined size is cut out of an image without considering the semantic content of the image. However, when an image is processed using cropping, an object other than the object to be detected may be included in the cropped area, or a part of the object may not be included. After that, when the neural network inputs the image whose aspect ratio has been converted by cropping, the detection accuracy is limited by the presence of unnecessary objects and deformation of the object to be detected.
As an example, as shown in FIG. 1, when a target area 112 in an image 111 is converted to a predetermined aspect ratio by cropping 113, the target area in an image 114 after conversion includes a plurality of objects. In this way, since a plurality of objects exist in the target area, the object detection accuracy of the neural network 115 may deteriorate.

Therefore, as described above, according to the present disclosure, by applying a background synthesizing means that considers the semantic content of an image, the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image. It is possible to provide area conversion means capable of high-precision object detection.

Referring now to Figure 2, a computer system 200 for implementing embodiments of the present disclosure will be described. The mechanisms and apparatus of various embodiments disclosed herein may be applied to any suitable computing system. The major components of computer system 200 include one or more processors 201 , memory 202 , terminal interfaces 203 , storage interfaces 204 , I/O (input/output) device interfaces 205 and network interfaces 206 . These components may be interconnected via memory bus 210 , I/O bus 211 , bus interface unit 220 and I/O bus interface unit 221 .

Computer system 200 may include one or more general-purpose programmable central processing units (CPUs) 201A and 201B, collectively referred to as processors 201 . In some embodiments, computer system 200 may include multiple processors, and in other embodiments, computer system 200 may be a single CPU system. Each processor 201 executes instructions stored in memory 202 and may include an on-board cache.

In some embodiments, memory 202 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 202 may store all or part of the programs, modules, and data structures that implement the functions described herein. For example, memory 202 may store domain transformation application 230 . In some embodiments, domain transformation application 230 may include instructions or descriptions that perform the functions described below on processor 201 .

In some embodiments, the domain conversion application 230 may be used in semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to processor-based systems. may be implemented in hardware via In some embodiments, domain conversion application 230 may include data other than instructions or descriptions. In some embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with bus interface unit 220, processor 201, or other hardware of computer system 200. .

Computer system 200 may include a bus interface unit 220 that provides communication between processor 201 , memory 202 , display system 240 and I/O bus interface unit 221 . I/O bus interface unit 221 may be coupled to I/O bus 211 for transferring data to and from various I/O units. I/O bus interface unit 221 communicates, via I/O bus 211, a plurality of I/O interface units 203, 204, 205, also known as I/O processors (IOPs) or I/O adapters (IOAs); and 206.

Display system 240 may include a display controller, a display memory, or both. The display controller can provide video, audio, or both data to display device 241 . Computer system 200 may also include devices such as one or more sensors configured to collect data and provide such data to processor 201 .

For example, the computer system 200 may include a biometric sensor that collects heart rate data, stress level data, etc., an environmental sensor that collects humidity data, temperature data, pressure data, etc., and a motion sensor that collects acceleration data, motion data, etc. may include Other types of sensors can also be used. The display system 240 may be connected to a display device 241 such as a single display screen, television, tablet, or handheld device.

The I/O interface unit provides the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 203 may include user output devices such as video displays, speaker televisions, etc., and user input devices such as keyboards, mice, keypads, touch pads, trackballs, buttons, light pens, or other pointing devices. Such user I/O devices 250 can be attached. A user inputs input data and instructions to the user I/O device 250 and the computer system 200 by operating the user input device using the user interface, and receives output data from the computer system 200. good too. The user interface may be displayed on a display device, played by a speaker, or printed via a printer, for example, via user I/O device 250 .

Storage interface 204 connects to one or more disk drives or direct access storage device 260 (typically a magnetic disk drive storage device, but an array of disk drives or other storage device configured to appear as a single disk drive). ) can be attached. In some embodiments, storage device 260 may be implemented as any secondary storage device. The contents of memory 202 may be stored in storage device 260 and read from storage device 260 as needed. I/O device interface 205 may provide an interface to other I/O devices such as printers, fax machines, and the like. Network interface 206 may provide a communication pathway to allow computer system 200 and other devices to communicate with each other. This communication path may be, for example, network 270 .

In some embodiments, computer system 200 is a device that receives requests from other computer systems (clients) that do not have a direct user interface, such as multi-user mainframe computer systems, single-user systems, or server computers. There may be. In other embodiments, computer system 200 may be a desktop computer, handheld computer, laptop, tablet computer, pocket computer, phone, smart phone, or any other suitable electronic device.

Next, the configuration of the domain conversion system according to the embodiment of the present disclosure will be described with reference to FIG.

FIG. 3 is a diagram illustrating an example configuration of a domain conversion system 300 according to an embodiment of the present disclosure. The domain conversion system 300 shown in FIG. 3 is a system for providing a domain conversion means that maintains image quality and semantic information while appropriately changing the aspect ratio of an image to facilitate highly accurate object detection. is. As shown in FIG. 3 , the domain conversion system 300 according to the embodiment of the present disclosure mainly includes an image acquisition device 301 , a storage unit 302 , a domain conversion device 303 and a client terminal 304 . The function of each functional unit of domain conversion system 300 shown in FIG. 3 may be implemented by computer system 200 described with reference to FIG.
Also, the image acquisition device 301, storage unit 302, area conversion device 303, and client terminal 304 shown in FIG. may be connected.

The image acquisition device 301 is a functional unit for acquiring an image sequence to be analyzed by a neural network. The image acquisition device 301 here may be, for example, an RGB camera, an infrared camera, a LiDar sensor, or a device configured to acquire any image or video. As an example, the image acquisition device 301 may be a surveillance camera installed to monitor a station platform. The image acquisition device 3601 may store the acquired image sequence in the storage unit 302 and transmit it to the domain transformation device 303 .
Note that the image sequence here is a set of images including at least one image, and may be, for example, a video.

The storage unit 302 is a storage unit for storing the image sequence acquired by the image acquisition device 301 and the analysis result by the neural network 330 which will be described later. The storage unit 302 here may be, for example, a local storage such as a hard disk drive or a solid state drive, or may be a distributed storage service such as cloud.

The area conversion device 303 is a device for appropriately changing the aspect ratio of an image while maintaining the quality and semantic information of the image by applying background synthesizing means that considers the semantic content of the image. Semantic information here is information for determining an object or activity in an image. Also, the background synthesizing means here is means for converting the frame of the target image into a predetermined aspect ratio by adding or deleting background pixels.
As shown in FIG. 3, the domain transforming device 303 includes a preprocessing unit 310 , a domain transforming unit 320 , a neural network 330 and an output unit 340 .

The preprocessing unit 310 is a functional unit for performing preprocessing on the image sequence acquired by the image acquisition device 301 . For example, as preprocessing, the preprocessing unit 3710 converts the image sequence acquired by the image acquisition device 301 into a predetermined data format, deletes non-target images from the image sequence, and converts encrypted information. It may be decrypted.

The area conversion unit 320 is a functional unit for determining and executing the above-described background synthesizing means. As shown in FIG. 3 , the region conversion section 320 includes an image frame identification section 321 , a region of interest detection section 322 , an image frame processing section 323 , a background synthesis means determination section 324 and a background synthesis section 325 .

The image frame identification unit 321 is a functional unit that identifies a target image frame to be subjected to area conversion from among the image sequences acquired by the image acquisition device 301 .
The region-of-interest detection unit 322 detects the region of interest in the target image frame specified by the image frame specifying unit 321, and determines the reliability of the processing operation for extracting the region-of-interest image including the region of interest from the target image frame. It is a function part that calculates.
When the reliability of the processing operation for extracting the region of interest image from the target image frame satisfies a predetermined reliability standard, the image frame processing unit 323 uses the processing operation to extract the region of interest image from the target image frame. is the functional part of
If the reliability of the processing operation for extracting the region-of-interest image from the target image frame does not satisfy a predetermined reliability standard, or if the region-of-interest image extracted from the target image frame A functional unit for determining a background synthesizing means for converting the target image frame to a predetermined aspect ratio by adding or deleting background pixels to the target image frame from a plurality of background synthesizing means candidates when the aspect ratio standard is not satisfied. is.
The background synthesizing unit 325 uses the background synthesizing means determined by the background synthesizing means determining unit 324 to add or delete background pixels to or from the target image frame, thereby converting the target image frame into a predetermined aspect ratio. Department.

The neural network 330 is a neural network for inputting and analyzing an image (final image) that has been converted to an appropriate aspect ratio by the domain conversion section 320 . For example, neural network 330 may be a deep convolutional neural network configured to perform object detection on images that have been converted to the appropriate aspect ratio by domain converter 320 . An analysis result indicating the result by the neural network 330 is stored in the storage unit 302 and transferred to the output unit 340 .
In one embodiment, neural network 330 includes an input layer, one or more hidden layers, and an output layer as convolution layers. In the neural network 330, the N-th intermediate layer receives the value output from the (N−1)-th layer as an input value, and performs a convolution operation on the input value using a plurality of weight filters having weight coefficients. is configured to generate a value to be output to the (N+1)-th layer. By this convolution operation, the neural network 3830 can extract the feature amount of the image and perform processing such as object detection.

The output unit 340 is a functional unit for outputting analysis results generated by the neural network 330 . The output unit 340 may, for example, transmit the analysis result generated by the neural network 330 to a predetermined notification destination via a communication network such as the Internet. For example, in some embodiments, output unit 340 may transmit analysis results generated by neural network 330 to client terminal 304 .
The client terminal 304 is a device used by the client who requested the analysis of the neural network, and may be any device such as a desktop computer, a notebook computer, or a mobile terminal such as a smart phone or tablet.

According to the area conversion system 300 configured as described above, the aspect ratio of the image is appropriately adjusted while maintaining the quality and semantic information of the image by applying the background synthesizing means that considers the semantic content of the image. to provide a region transformation means that facilitates highly accurate object detection.

Next, background synthesis, which is an example of area conversion according to an embodiment of the present disclosure, will be described with reference to FIG.

FIG. 4 is a diagram for explaining background synthesis, which is an example of area conversion according to an embodiment of the present disclosure. As described above, the present disclosure can appropriately change the aspect ratio of an image while maintaining the quality and semantic information of the image by applying the background synthesis means that considers the semantic content of the image. .

The background synthesizing means according to the embodiment of the present disclosure is a target image frame to be subjected to area conversion,
Means for converting the frame of the target image to a predetermined aspect ratio by adding or deleting background pixels. Also, the background synthesis according to the embodiment of the present disclosure may be performed after considering the semantic content of the image. For example, as will be described later, in one embodiment, the background synthesizing means for converting frames of the target image to a predetermined aspect ratio is based on regions of interest and saliency centers that contain a lot of semantic information useful for neural network analysis. may be done.

As an example of the background synthesizing means according to the embodiment of the present disclosure, for example, as shown in FIG. 4, a target image frame 401 to be subjected to area conversion is uniformly scaled to a desired size, and a first A composite region 402 can be added to the top of the target image frame 401 and a second composite region 403 consisting of background pixels can be added to the bottom of the target image frame 401 . In some embodiments, the location of adding the first 402 and second 403 synthesis regions may be based on the region of interest or saliency center of the target image frame 401 .
As a result, the aspect ratio of the image can be appropriately changed while maintaining semantic information useful for neural network analysis.

In the above, an example of background synthesis has been described in order to explain the concept of background synthesis according to the embodiment of the present disclosure. is selected from among several candidates for background synthesizing means. As a result, for each target image frame to be subjected to area conversion, the area conversion process suitable for the target image frame can be performed.

Next, with reference to FIG. 5, the flow of area conversion processing according to the embodiment of the present disclosure will be described.

FIG. 5 is a flow chart showing the flow of region conversion processing 500 according to an embodiment of the present disclosure. The area conversion process 500 shown in FIG. 5 is a process for performing area conversion that appropriately changes the aspect ratio of an image while maintaining the quality and semantic information of the image, thereby facilitating highly accurate object detection. , are performed by each functional unit of the domain converter (for example, the domain converter 320 shown in FIG. 3).

First, in step S501, an image frame identification unit (for example, the image frame identification unit 321 shown in FIG. 3) selects an image sequence acquired by an image acquisition device (for example, the image acquisition device 301 shown in FIG. Identify the target image frame to be the target of. The target image frame here may be, for example, an image frame including an object to be analyzed by the neural network from among a plurality of frames forming an image sequence. For example, the image frame identifying unit may identify, as the target image frame, an image frame containing an object whose degree of similarity to an object specified in advance by the user satisfies a predetermined similarity criterion.
Further, as will be described later, in one embodiment, the image frame identifying unit may identify, as the target image frame, an image frame that includes an object of a class that satisfies a predetermined influence criterion.

Next, in step S502, a preprocessing unit (for example, the preprocessing unit 310 shown in FIG. 3) performs texture filtering (texture smoothing) on the target image frame identified in step S501. Texture filtering here is a method used to determine texture colors for texture mapping to pixels using neighboring colors. In other words, in the texture filtering process, the preprocessor divides the texture pixels into smaller pixel units and mixes them. Texture filtering methods here include, for example, Gaussian Blur, Median Blur, Total Relative Variation, Regularization, etc. By performing texture filtering on the target image frame, the accuracy of detection of the region of interest and background synthesis, etc., which will be described later, is improved. can be made

Next, in step S503, a region-of-interest detection unit (for example, the region-of-interest detection unit 322 shown in FIG. 3) detects a region of interest in the target image frame that has undergone the processing of step S502, and detects a region of interest including the region of interest. Compute the reliability of the processing operation for extracting the image from the target image frame.
Here, the region of interest is a region including the object to be detected in the target image frame. For example, if the object to be detected is a "black car", the region of interest is the region in the target image frame that is likely to contain the black car.
Further, the processing operation here is an operation for extracting the region-of-interest image from the target image frame, and may include cropping, carving, and the like, for example.
Further, the reliability here is a scale representing the appropriateness of the processing operation described above, and is used to determine the area conversion means for changing the aspect ratio of the target image frame, as described later.
Furthermore, in step S503, the region-of-interest detection unit calculates a saliency map (saliency map) of the target image frame, and determines a saliency center indicating the saliency center based on this saliency map. good. The saliency map here is a data structure representing the strength of saliency corresponding to the target image frame, and indicates the degree of "human interest" in the target image frame. This saliency map may be determined, for example, based on features such as edges and colors of objects in the target image frame.
Note that the details of the region-of-interest detection unit will be described with reference to FIG. 6, so description thereof will be omitted here.

Next, in step S504, the image frame processing unit (for example, the image frame processing unit 323 shown in FIG. 3) determines that the reliability of the processing operation for extracting the region-of-interest image from the target image frame calculated in step S503 is A determination is made as to whether or not a predetermined reliability criterion is met. The reliability standard here may be set, for example, by an administrator of the area conversion system 300, or may be set based on the results of past processing operations or the like.
If the reliability of the machining operation satisfies the predetermined reliability criterion, the process proceeds to step S505, and if the reliability of the machining operation does not satisfy the predetermined reliability criterion, the process proceeds to step S507.

In step S505, the image frame processing unit executes a processing operation for extracting the region-of-interest image from the target image frame. For example, the image frame processor may perform image cropping or carving as the processing operation for extracting the region of interest image from the target image frame.
Here, cropping is means for obtaining a region-of-interest image showing only the region of interest by deleting unnecessary pixels (not included in the region-of-interest image) from the periphery of the target image frame. Carving here is means for obtaining a region-of-interest image showing only the region of interest by cutting out and deleting an arbitrary region (not limited to the peripheral portion) of the target image frame. Carving is effective, for example, when there are multiple regions of interest and a region of interest image showing only the plurality of regions of interest is obtained by removing unnecessary background pixels existing between the regions of interest. .

In step S506, the background synthesizing means determining unit (for example, the background synthesizing means determining unit 324 shown in FIG. 3) determines whether or not the region of interest image extracted from the target image frame satisfies a predetermined aspect ratio standard. This aspect ratio criterion may be, for example, a criterion that defines the desired aspect ratio of the target image frame input to the neural network.
If the region of interest image extracted from the target image frame meets the predetermined aspect ratio criterion, the process proceeds to step S509; if the region of interest image extracted from the target image frame does not meet the predetermined aspect ratio criterion, the present Processing proceeds to step S507.

In step S507, the background synthesizing means determination unit 324 determines a background synthesizing means for converting the target image frame to a predetermined aspect ratio by adding or deleting background pixels to or from the target image frame from a plurality of background synthesizing means candidates. do. In one embodiment, the background synthesizing means determination unit 324 selects the background synthesizing means for each of the plurality of candidates for background synthesizing means based on the characteristics of the target image frame (size of the region of interest, configuration of the object to be detected, etc.). , after assigning a suitability score indicating suitability of the background synthesizing means, a predetermined suitability score (for example, a candidate for the background synthesizing method with the highest suitability score) may be determined.
Since the details of the plurality of candidates for background synthesizing means will be described later, the description thereof will be omitted here.

Next, in step S508, the background synthesizing unit 325 uses the background synthesizing means determined by the background synthesizing means determining unit 324 in step S507 to add or delete background pixels to or from the target image frame. to a given aspect ratio. As noted above, in some embodiments, the background composition may be based on regions of interest or saliency centers. As a result, the aspect ratio of the image can be appropriately changed while maintaining semantic information useful for neural network analysis.
In the above, the case where the target image frame is converted to a predetermined aspect ratio has been described as an example, but the present disclosure is not limited to this. Performing this on the region image may convert the region of interest image to a predetermined aspect ratio.

Next, in step S509, the preprocessing unit 310 performs preprocessing for inputting the target image frame to the neural network. For example, in one embodiment, the preprocessing unit may convert the target image frame into a predetermined data format, compress the target image frame, or rotate the target image frame as preprocessing.

In step S510, a neural network (for example, neural network 330 shown in FIG. 3) analyzes the target image frame and outputs an analysis result indicating the result of this analysis. As an example, the neural network may perform object detection on the image that has been converted to the appropriate aspect ratio by the area detector, and generate an analysis result indicating the result of this object detection.

According to the region conversion processing 500 described above, the aspect ratio of the image can be appropriately changed while maintaining the quality and semantic information of the image by applying the background synthesizing means that considers the semantic content of the image. can be done. In addition, the accuracy of object detection can be improved by subjecting an image whose aspect ratio has been changed to an appropriate aspect ratio to analysis by a neural network.

Next, processing of the region-of-interest detection unit according to the embodiment of the present disclosure will be described with reference to FIGS. 6 to 8. FIG.

FIG. 6 is a diagram illustrating an example of processing of the region-of-interest detection unit 322 according to the embodiment of the present disclosure.
As described above, the region of interest detection unit 322 according to the embodiment of the present disclosure detects the region of interest 602 and the saliency center 603 in the input target image frame 601, and converts the region of interest image including the region of interest 602 into the target image. It is a functional unit for calculating the reliability 604 of the processing operation for extracting from the frame.
The region of interest 602 here is a region including the object to be detected in the target image frame 601 . For example, if the object to be detected is a “black car”, the region of interest 602 is the region in the target image frame 601 that is likely to contain a black car.
The saliency center 603 here is the saliency center in the target image frame 601 . In other words, saliency center 603 indicates the coordinates of the center of the visually salient region in target image frame 601 . As an example, saliency center 603 may be coordinates indicating the center of region of interest 602, for example.

In one embodiment, region of interest detector 322 may be a neural network model trained to detect region of interest 602 in an image. The region-of-interest detector 322 may be trained, for example, to maximize an IoU (Intersection over Union) score between a predicted region-of-interest and a region-of-interest indicated in the ground truth.

The reliability 604 is a measure representing the appropriateness of processing operations such as cropping and carving, and is used to determine area conversion means for changing the aspect ratio of the target image frame. As shown in FIG. 6, the reliability 604 here includes a bounding box that defines the coordinates of an area extracted by a processing operation such as cropping or carving, a cropping affinity that indicates the appropriateness of cropping, and an appropriateness of carving. A curve affinity indicating saliency center 603 and a saliency center confidence indicating the certainty of the saliency center 603 may also be included.

In some embodiments, the region of interest detector 322 may perform statistical analysis on the spatial distribution of the regions of interest in the target image frame. For example, in one embodiment, the region-of-interest detection unit 322 estimates the region-of-interest of the target image frame currently being analyzed based on the spatial distribution and appearance frequency of the regions of interest detected in previously analyzed images. good too.
As an example, in the image of the road analyzed in the past, when the frequency of traffic lights existing on the left side is high (for example, a predetermined appearance frequency criterion is satisfied), the region of interest detection unit 322, when the image of the road is input, detects the traffic lights It may be estimated that a region of interest containing is on the left side of the image. The result of statistical analysis on the spatial distribution of this region of interest may be used for detection of the region of interest and may be used for determining the background synthesizing means.

FIG. 7 is a diagram showing an example of the reliability of the machining operation calculated by the region-of-interest detection unit according to the embodiment of the present disclosure.

As described above, the region-of-interest detection unit 322 according to the embodiment of the present disclosure calculates the reliability indicating the appropriateness of each processing operation such as cropping and carving. In addition, this reliability may include, for example, a cropping affinity indicating the appropriateness of cropping and a carving affinity indicating the appropriateness of carving. If the cropping affinity calculated by the region-of-interest detection unit 322 satisfies a predetermined cropping affinity criterion, the above-described image frame processing unit processes the target image frame using the cropping means so that the carving affinity meets the predetermined cropping affinity. If the carving affinity criterion is satisfied, the image frame processing section processes the target image frame using the carving means.
On the other hand, if the reliability of the machining operation does not meet the predetermined reliability criterion (that is, if the cropping affinity does not satisfy the predetermined cropping affinity criterion and the carving affinity does not satisfy the predetermined carving affinity criterion ), or if the region-of-interest image extracted from the target image frame does not meet the predetermined aspect ratio standard, the above-described background synthesis means determining unit adds or deletes background pixels to the target image frame to render the target image frame as A background synthesizing means for converting to a predetermined aspect ratio is determined from a plurality of background synthesizing means candidates.

Consider the target image frame 701 shown in FIG. 7 as an example. As described above, after receiving the target image frame 701 , the region of interest detection unit 322 calculates the region of interest 702 and the saliency center 703 of the target image frame 701 . After that, the region of interest detection unit 322 calculates cropping affinity and carving affinity based on the calculated region of interest 702 and saliency center 703 .
As mentioned above, the cropping affinity is lower the more semantic information (eg, region of interest pixels, saliency center pixels) is lost if cropping is performed on the target image frame. In other words, the cropping affinity is lower the more information is expected to be lost by cropping.
Similarly, carving affinity is lower the more semantic information (eg, region of interest pixels, saliency center pixels) is lost if carving is performed on the target image frame. In other words, the carving affinity is lower the more information is expected to be lost by carving.
For example, if there are many objects to be verified in a region of interest 702 as shown in FIG. 7, if cropping is performed, pixels of the objects to be verified are likely to be lost. Therefore, the suitability of cropping is low, and the cropping affinity becomes a value that does not satisfy the cropping affinity standard.
Similarly, if the objects under verification are close together in the region of interest 702 and the distances between the objects are narrow, pixels of the object under verification are likely to be lost if carving is performed. Therefore, the appropriateness of carving is low, and the carving affinity is a value that does not satisfy the carving affinity standard.

FIG. 8 is a diagram showing another example of the reliability of the machining operation calculated by the region-of-interest detection unit 322 according to the embodiment of the present disclosure.

Consider the target image frame 801 shown in FIG. 8 as an example. As described above, after receiving the target image frame 801 , the region of interest detection unit 322 calculates the region of interest 802 and the saliency center 803 of the target image frame 801 . After that, the region of interest detection unit 322 calculates cropping affinity and carving affinity based on the calculated region of interest 802 and saliency center 803 .
If there is only one region of interest 802 and only one object in the region of interest 802, such as the target image frame 801, a region of interest image showing only the region of interest 802 can be extracted by cropping. be. Therefore, the suitability of cropping is high, and the cropping affinity becomes a value that satisfies the cropping affinity criteria.
Similarly, if there is only one region of interest 802 and only one object in the region of interest 802, such as the target image frame 801, carving extracts a region of interest image showing only the region of interest 802. is also possible. Therefore, the appropriateness of carving is high, and the carving affinity is a value that satisfies the carving affinity standard. However, in principle, cropping requires less computing resources than carving, so if both cropping and carving meet the affinity criteria, it is preferable to use cropping from the viewpoint of saving computing resources.

Also consider the target image frame 811 shown in FIG. 8 as another example. As described above, after receiving the target image frame 811 , the region of interest detection unit 322 calculates the region of interest 812 and the saliency center 813 of the target image frame 811 . After that, the region of interest detection unit 322 calculates cropping affinity and carving affinity based on the calculated region of interest 812 and saliency center 813 .
If there are multiple regions of interest 812 , such as the target image frame 811 , if cropping is performed, there is a high probability of including unwanted background pixels between the multiple regions of interest 812 . Therefore, the suitability of cropping is low, and the cropping affinity becomes a value that does not satisfy the cropping affinity standard.
On the other hand, when a plurality of regions of interest 812 exist, each of the plurality of regions of interest 812 is cut out, unnecessary background pixels between the plurality of regions of interest 812 are removed, and a region of interest image showing only the regions of interest 812 is extracted by carving. It is possible to Therefore, the appropriateness of carving is high, and the carving affinity is a value that satisfies the carving affinity standard.

As described above, according to the processing of the region-of-interest detection unit 322 described with reference to FIGS. It is possible to calculate the reliability of the processing operation for extracting the region of interest image containing from the target image frame. Further, according to this, it is possible to determine the area conversion means that considers the semantic content of the image.

Next, with reference to FIG. 9, processing of the background synthesizing means determination unit according to the embodiment of the present disclosure will be described.

FIG. 9 is a diagram illustrating an example of processing of the background synthesizing means determination unit 324 according to the embodiment of the present disclosure. As described above, the background synthesizing means determining unit 324 determines whether the reliability of the processing operation for extracting the region-of-interest image from the target image frame does not satisfy a predetermined reliability standard, or If the area image does not satisfy a predetermined aspect ratio standard, a background synthesizing means for converting the target image frame to a predetermined aspect ratio by adding or deleting background pixels to the target image frame is selected from a plurality of background synthesizing means candidates. do.

The background synthesizing means determination unit 324 may be a machine learning model calculated to calculate an adequacy score indicating suitability of each background synthesizing means candidate 903 based on the target image frame 901 and the region of interest 902. good. For example, in some embodiments, the background composition means determiner 324 may be a neural network model, a support vector machine model, or a decision tree model.
As an example, after inputting the target image frame 901 and the region of interest 902 detected by the region of interest detection unit 322, the background synthesizing means determination unit 324 inputs the characteristics of the target image frame 901 (the size of the region of interest 902, the detection For each of the plurality of background synthesizing means candidates 903, an adequacy score indicating the appropriateness of the background synthesizing means may be assigned based on the configuration of the target object, etc.). After that, the background synthesizing means determination unit 324 may determine a predetermined appropriateness score (for example, the background synthesizing means candidate 903 with the highest appropriateness score) as the background synthesizing means to be applied to the target image frame 901 .

As described above, the background synthesizing means here is means for converting the frame of the target image into a predetermined aspect ratio by adding or deleting background pixels. An existing method may be included, and first to eighth background synthesizing means, which will be described later, may be included.
In the present disclosure, several background synthesizing means will be described as examples, but the present disclosure is not limited to this, and any background synthesizing means may be used.

According to the background synthesizing means determining unit 324 described above, even if the target image frame cannot be changed to the desired aspect ratio by processing operations such as cropping and carving, the semantic content of the image can be taken into consideration. Appropriate background composition means can be selected. As a result, it is possible to appropriately change the aspect ratio of the image while maintaining the quality and semantic information of the image, thereby facilitating highly accurate object detection.

Next, the background synthesizing means according to the embodiment of the present disclosure will be described with reference to FIGS. 10 to 18. FIG.
Several background synthesizing means for converting the aspect ratio of an image will be described below, but these background synthesizing means may be used singly or in combination.

FIG. 10 is a diagram showing an example of the first background synthesizing means 1000 according to the embodiment of the present disclosure. The first background synthesizing means 1000 shown in FIG. 10 is a means for converting the target image frame to a target aspect ratio by adding background pixels to the periphery of the target image frame. This is performed by the background synthesizing unit 325 shown in FIG.

First, the background synthesizing unit generates channel images 1002 by dividing the target image frame 1001 into channels constituting the target image frame 1001 . For example, if the target image frame 1001 is an RGB image, the background synthesizing unit generates a channel image 1002 by decomposing the target image frame 1001 into three channels of R, G, and B.
The background synthesizer then calculates 1010 the median value of the pixels of the channel image 1002 .

Next, after calculating the median value of the channel image 1002, the background synthesizing unit generates a synthesized background image 1003 having the same number of channels as the target image frame 1001 and satisfying a predetermined size standard for the size of the target image frame 1001. to generate The pixel value of each pixel of this synthetic background image 1003 may be, for example, the median value of the channel image 1002 .
Also, the size standard here may be any magnification such as a size larger than the target image frame 1001 by 20%, a size larger than 30%, or a size larger than 50%.

Next, the background synthesizing unit combines the synthesized background image 1003 and the target image frame 1001 by the predetermined frame blending means 1020 based on the region of interest and the saliency center 1004 detected by the region of interest detection unit 322 described above. By doing so, a first combined image 1005 is generated. For example, in one embodiment, the background synthesizer calculates the center coordinates of the synthetic background image 1003 using the saliency center, then aligns the synthetic background image 1003 with the target image frame 1001 based on the center coordinates, and may be combined.
The frame blending means 1020 here is means for blurring the edge between the synthetic background image 1003 and the target image frame 1001, and includes any existing frame blending means such as Poisson Blending, Wavelet-based blending, alpha blending, etc. It's okay.

Next, the background synthesizing unit generates a final image 1006 by converting the first combined image 1005 into a target aspect ratio using a predetermined size converting means (for example, scaling).
As described above, according to the first background synthesizing unit 1000 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 11 is a diagram showing an example of second background synthesizing means 1100 according to an embodiment of the present disclosure. The second background synthesizing means 1100 shown in FIG. 11 uses an encoder-decoder model to add a synthesizing area that approximates the background area of the target image frame to the target image frame so that the target image frame has a desired aspect ratio. A means for transforming, which is performed by a background synthesizer (eg, the background synthesizer 325 shown in FIG. 3).

First, the background synthesis unit generates trained encoder-decoder models 1111 and 1121 by training encoder-decoder models 1110 and 1120 that generate a first synthesis region 1103 that approximates the target region 1102 in the learning image 1101. do.
More specifically, during the training phase, encoder model 1110 inputs region of interest 1102 in training image 1101 . This region of interest 1102 is a set of continuous pixels along one or more directions, preferably a background region excluding the region of interest of the learning image 1101 . Also, in some embodiments, this region of interest 1102 may include multiple regions that are ordered.

After inputting the region of interest 1102 , the encoder model 1110 generates a latent representation that approximates the region of interest 1102 and outputs it to the decoder model 1120 . Next, the decoder model 1120 generates a first synthesized region 1103 indicating regions before and after the region of interest 1102 based on the input latent representation that approximates the region of interest 1102 . Trained encoder-decoder models 1111, 1121 can be generated by training the encoder-decoder models 1110, 1120 to generate a first synthetic region 1103 of higher accuracy.

Next, during the inference stage, the trained encoder model 1111 inputs the background region 1105 in the target image frame 1104 . This background region 1105 is a collection of contiguous pixels along one or more directions, similar to the region of interest 1102 described above, and is preferably the background region of the target image frame 1104 excluding the region of interest. . Also, in some embodiments, this background region 1105 may include multiple regions that are ordered.

After inputting background region 1105 , trained encoder model 1111 generates a latent representation that approximates background region 1105 and outputs it to trained decoder model 1121 . The trained decoder model 1121 then generates a second synthetic region 1106 showing regions before and after the background region 1105 based on the input latent representation approximating the background region 1105 .

Next, the background synthesis unit inserts a second synthesis region 1106 into the target image frame 1104 based on the target image frame 1104 region of interest/salience center 1107 detected by the region of interest detection unit 322 described above. , converts the target image frame 1104 to a predetermined aspect ratio to produce the final image 1108 . Here, the background synthesizing unit may determine the position where the second synthesizing region 1106 is to be inserted into the target image frame 1104 based on the region of interest/salience center 1107 of the target image frame 1104 .

As described above, according to the second background synthesizing unit 1100 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 12 is a diagram showing an example of a third background synthesizing means 1200 according to an embodiment of the present disclosure.
The third background synthesizing means 1200 shown in FIG. 12 uses a Gaussian process regression model to add a synthesizing area that approximates the background area of the target image frame to the periphery of the target image frame. , and is executed by a background synthesizing unit (for example, the background synthesizing unit 325 shown in FIG. 3).

First, the background synthesizing unit trains a Gaussian process regression model that generates a synthetic fringe area (for example, a first synthetic fringe area) that approximates the fringe area present in the fringe of the training image. generates a Gaussian process regression model 1210 of .

Next, the background synthesizing unit inputs the peripheral edge region 1202 existing at the edge of the target image frame 1201 to the trained Gaussian process regression model 1210 . The Gaussian process regression model 1210 then generates 1220 a synthetic fringe region (eg, a second synthetic fringe region) that approximates this fringe region 1202 .
For example, trained Gaussian process regression model 1210 may predict a plurality of synthetic marginal regions and then select among the predicted synthetic marginal regions those whose prediction error for marginal region 1202 satisfies a predetermined criterion. . The prediction error here may be, for example, the mean square error of pixel values.

A trained Gaussian process regression model 1210 loops around the target image frame 1201 to generate a synthetic fringe region that approximates each region of the periphery, and the generated synthetic fringe region is of a size larger than the target image frame 1201. By appropriately arranging them in accordance with the shape of the target image frame 1201, a synthesized peripheral image 1203 having a size larger than the target image frame 1201 can be obtained.

After that, the background synthesizing unit combines the synthesized peripheral edge image 1203 and the target image frame 1201 by the predetermined frame blending means 1230 based on the region of interest and the saliency center 1204 detected by the region of interest detecting unit 322 described above. to generate a second combined image 1205 . For example, in one embodiment, the background synthesizer 325 uses the saliency center to calculate the center coordinates of the target image frame 1201 and then aligns the synthesized peripheral image 1203 with the target image frame 1201 based on the center coordinates. , may be combined.
As described above, the frame blending means 1230 here is a means for blurring the edge between the synthesized peripheral image 1203 and the target image frame 1201, and any existing blending such as Poisson Blending, Wavelet-based blending, alpha blending, etc. A frame blending means may be included.

Next, the background synthesizing unit generates a final image 1206 by converting the second combined image 1205 into a desired aspect ratio using a predetermined size converting means (for example, scaling).
As described above, according to the third background synthesizing unit 1200 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 13 is a diagram showing an example of a fourth background synthesizing means 1300 according to an embodiment of the present disclosure. The fourth background synthesizing means 1300 shown in FIG. 13 uses a Gaussian process regression model to generate patch seams in the target image frame, and adds or deletes the generated patch seams to or from the target image frame. to a desired aspect ratio, and is executed by a background synthesizing unit (for example, the background synthesizing unit 325 shown in FIG. 3).

First, the background synthesizing unit divides the target image frame 1301 into regions that do not overlap each other along a predetermined direction (for example, the vertical direction in the case of the target image frame 1301 shown in FIG. 13). For each region using regression model 1310, a synthetic region is generated that approximates the region.
As an example, as shown in FIG. 13, a trained Gaussian process regression model 1310 uses a patch seam for a plurality of continuous synthesized regions arranged across a plurality of regions along the direction in which the target image frame 1301 is divided. 1302 may be generated. Here, a patch seam is an n-connected set of pixels in an image that extends from top to bottom or from left to right. In some embodiments, the width of the patch seam 1302 may be determined according to the desired aspect ratio.

Next, the background synthesizing unit can convert the target image frame 1301 to the target aspect ratio by deleting or adding the patch seam 1302 in the target image frame 1301 . For example, if the target aspect ratio is smaller than the target image frame 1301, the background synthesizing unit deletes the generated patch seam 1302 from the target image frame 1301, thereby reducing the target image frame 1301 to the target aspect ratio. 1303 can be generated.
On the other hand, if the target aspect ratio is larger than the target image frame 1301 , the background synthesizing unit generates additional patches based on the generated patch seam 1302 and inserts them into the target image frame 1301 . to the desired aspect ratio to produce a final image 1304 .

As described above, according to the fourth background synthesizing unit 1300 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 14 is a diagram for explaining a Gaussian process regression model 1400 according to an embodiment of the present disclosure. As described above, a so-called Gaussian process regression model 1400 may be used in the background synthesis means according to embodiments of the present disclosure. The Gaussian process regression model 1400 here is a nonparametric kernel-based probabilistic model that estimates a function from an input variable to a real-valued output variable. In other words, Gaussian process regression model 1400 is a probabilistic model that can predict unobserved data points based on the similarity of multiple data points.

In the present disclosure, the Gaussian process regression model 1400 can be used to take a background region in a target image frame as input and generate a synthetic background region with a high degree of similarity to the background region. By adding the generated synthesized background area to the target image frame, the target image frame can be converted to an arbitrary aspect ratio.

For example, patch seams in the target image frame can be generated using the Gaussian process regression model 1400, as described above for the fourth background synthesis means described with reference to FIG. As mentioned above, a patch seam is a set of n-connected pixels in an image that extends from top to bottom or from left to right. More specifically, a patch seam is a set of regions P ^r in an image of height H and width W, patches with center point C(r) at coordinates (i,j).
where i is contained in (1, W) and j is contained in (1, H). Also, the patch has a height h and a width w.

ここで、r=1のとき、パッチシームが縦方向に延びている場合、画像の１行目の画素が選択され、パッチシームが横方向に延びている場合、画像の一例目の画素が選択される。パッチの高さ及び幅は、画像の高さ及び幅に基づいて選択されてもよく、目的のアスペクト比に基づいて選択されてもよい。
また、パッチシームが縦方向に延びている場合、Ｎ＝Ｈ／ｈであり、パッチシームが横方向に延びている場合、Ｎ＝Ｗ／ｗ。 In other words, the patch seam S is the set of patches indexed by r contained in (1, N) and defined by Equations 1-3 below.

Here, when r=1, if the patch seam extends vertically, the pixel in the first row of the image is selected, and if the patch seam extends horizontally, the pixel in the first image is selected. be done. The height and width of the patch may be selected based on the height and width of the image and may be selected based on the desired aspect ratio.
Also, if the patch seam runs longitudinally, N=H/h, and if the patch seam runs laterally, N=W/w.

Gaussian process regression computes the probability distribution over the space of all admissible functions that fit the data. This procedure is described below.

ここで、yは観測データ（ｏｂｓｅｒｖｅｄ ｄａｔａ）であり、f_*は未観測データ（Ｕｎｏｂｓｅｒｖｅｄ ｄａｔａ）であり、Ｘは学習用データであり、Ｘ_*は推論用のテストデータであり、Ｋはカーナル関数であり、μは観測データの平均であり、μ_*はテストデータの平均である。 First, assuming that the prior distribution of the Gaussian process such as the mean m(x) and the covariance function such as k(x,x') are defined in advance, the Gaussian process is represented by the following equations 4 and 5. Become.

where y is observed data, f _* is unobserved data, X is training data, X _* is test data for inference, and K is the kernel function. where μ is the mean of the observed data and μ _* is the mean of the test data.

In the training phase the parameters of the kernel function K are learned and in the inference phase a so-called stochastic sampling is performed on the learned distribution.
As an example, a kernel function with σ and M as parameters is shown in Equation 6 below.

In the fourth background synthesizing means 1300 described above, X is the pixel value in the target image frame, indexed as {1, 2...N+k}.

Here, X represents the intensity of pixel values indexed as {1, 2...N-k} and X _* represents the intensity of pixel values indexed as {N-k+1, N-k+2,...N}. , y represents the intensity of pixel values _indexed as {k+1, k+1, . represents a kernel function

In one embodiment, in the fourth background synthesis means 1300 described above, the parameters of the kernel function K may be determined by a neural network using gradient descent. Also, in some embodiments, the parameters of the kernel function K may be determined by a likelihood-maximizing gradient computation technique.

The covariance values K(X,X), K(X _* ,X), K(X _* ,X _* ), K(X,X _* ) may be calculated by a neural network.

During the learning phase, the mean squared error between the observed and predicted values may be used to tune the parameters of the Gaussian process regression model. Also, during the inference stage, the learned parameters may be used to sample from the Gaussian process regression model.

It may also be used in the inference stage to estimate the salience of pixels indexed as {N+1, k+1, . Further, the estimated saliency may be used to identify patch seams to add or remove from the target image frame.

As an example, as shown in FIG. 14 , a Gaussian process regression model 1400 according to an embodiment of the present disclosure trains a learning image using learning data X, and then inputs test data X _* . A synthetic background region approximating data X _* can be output as unobserved data f _* . By adding this unobserved data f _* to the target image frame, the target image frame can be converted to the target aspect ratio.

FIG. 15 is a diagram showing an example of a fifth background synthesizing means 1500 according to an embodiment of the present disclosure.
The fifth background synthesizing means 1500 shown in FIG. 15 is a means for converting the target image frame to a target aspect ratio by adding background pixels having a constant pixel value to the target image frame. It is performed by a synthesizer (eg, the background synthesizer 325 shown in FIG. 3).

First, the background synthesizing unit generates a background synthetic image 1502 by adding background pixels having constant pixel values to the target image frame 1501 . For example, as shown in FIG. 15, the background synthesizing unit may generate a background synthesized image 1502 by adding background pixels having a constant pixel value to the periphery of the target image frame 1501 . After that, the background synthesizing unit generates a final image 1503 by converting the background synthetic image 1502 into a target aspect ratio using a predetermined size conversion means (for example, scaling).

In one embodiment, the final image 1503 generated here can be used to generate training images for training a given neural network. For example, in the final image 1503, a perturbed region 1504 is generated by perturbing the pixel values of the added background pixels. After that, the trained neural network 330 uses the final image 1503 containing this perturbed region 1504 and the region of interest and/or saliency center of the target image frame 1501 detected by the region of interest detection unit 322 described above to determine the adversarial An adversarial training image 1505 can be generated.
In some embodiments, trained neural network 330 may be generated by a projected gradient descent technique. Also, the input image may be processed based on a predetermined constraint requiring that the pixel values be within the range of 0-255.
Since this adversarial learning image 1505 contains perturbation regions, it is more difficult to detect objects than general images, and is useful for training a neural network for object detection.

Thus, according to the fifth background synthesizing means 1500 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 16 is a diagram showing an example of a sixth background synthesizing means 1600 according to an embodiment of the present disclosure.
The sixth background synthesizing means 1600 shown in FIG. 16 is a means for converting the target image frame to the target aspect ratio by duplicating the background pixels existing on one side of the target image frame to the opposite side. Yes, and is performed by a background synthesizer (eg, the background synthesizer 325 shown in FIG. 3).

First, the region-of-interest detection unit 322 described above receives and processes the target image frame 1601 to detect the region of interest and/or the saliency center 1602 in the target image frame 1601 . After that, the background synthesizing unit synthesizes the background pixels 1603 present on one side of the target image frame 1601 as a synthetic background region 1604 based on the region of interest and/or the saliency center 1602 detected by the region of interest detection unit 322 . Duplicating the opposite side of the image frame 1601 produces a final image 1605 that converts the target image frame to the desired aspect ratio. The size of the background pixels to replicate on the opposite side of the target image frame 1601 may be determined based on the region of interest and/or the saliency center 1602, for example.
Here, the case of copying the background pixels present on the left side of the target image frame 1601 to the right side has been described as an example, but the present disclosure is not limited to this. can also be duplicated on the left side, and background pixels existing above the target image frame 1601 can be duplicated below, and there is no particular limitation here.

ここでは、i_offsetは、合成背景領域が配置される横方向の座標を指定するi_specifiedと、背景画素の横方向の座標を指定するi_actualとの差分であり、ｊ_offsetは、合成背景領域が配置される縦方向の座標を指定するｊ_specifiedと、背景画素の縦方向の座標を指定するｊ_actualとの差分である。また、I(i,j)は、座標(i,j)での画素値の強度を表し、Hは対象画像フレームの高さを表し、Wは対象画像フレームの幅を表す。 More specifically, the background composition unit may duplicate background pixels according to Equation 7 below.

Here, i _offset is the difference between i _specified , which specifies the horizontal coordinates of the synthetic background area, and i _actual , which specifies the horizontal coordinates of the background pixels, and j _offset is the synthetic background area. It is the difference between j _specified , which specifies the vertical coordinates of the arranged pixels, and j _actual , which specifies the vertical coordinates of the background pixels. Also, I(i,j) represents the intensity of the pixel value at the coordinates (i,j), H represents the height of the target image frame, and W represents the width of the target image frame.

As described above, according to the sixth background synthesizing means 1600 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 17 is a diagram showing an example of a seventh background synthesizing means 1700 according to an embodiment of the present disclosure.
The seventh background synthesizing means 1700 shown in FIG. 17 converts the target image frame to the target aspect ratio by reflecting background pixels present in a specific region of the target image frame in the target region. and is executed by a background synthesizing unit (for example, the background synthesizing unit 325 shown in FIG. 3).

First, the above-described region-of-interest detection unit 322 receives the target image frame 1701 and processes it to detect the region of interest and/or the saliency center 1702 in the target image frame 1701 . After that, the background synthesizing unit synthesizes background pixels 1703 present in a specific region of the target image frame 1701 as a synthesized background region 1704 based on the region of interest and/or the saliency center 1702 detected by the region of interest detection unit 322. , is reflected in the target area of the target image frame 1701 to generate a final image 1705 in which the target image frame 1701 is converted to the target aspect ratio. The size of the background pixels to be reflected may be determined based on the region of interest and/or the saliency center 1702, for example. Also, the background pixels 1703 and the synthetic background region 1704 may be selected by the user, or may be selected by the background synthesizing unit based on past background synthesizing performance.
Here, the case where the background pixels existing on the left side of the target image frame 1701 are reflected immediately adjacent thereto has been described as an example, but the present disclosure is not limited to this, and the background pixels can be reflected at any position. may

ここでは、i_offsetは、合成背景領域が配置される横方向の座標を指定するi_specifiedと、背景画素の横方向の座標を指定するi_actualとの差分であり、ｊ_offsetは、合成背景領域が配置される縦方向の座標を指定するｊ_specifiedと、背景画素の縦方向の座標を指定するｊ_actualとの差分である。また、I(i,j)は、座標(i,j)での画素値の強度を表し、Hは対象画像フレームの高さを表し、Wは対象画像フレームの幅を表す。 More specifically, the background synthesizing unit may reflect the background pixels according to Equation 8 below.

Here, i _offset is the difference between i _specified , which specifies the horizontal coordinates of the synthetic background area, and i _actual , which specifies the horizontal coordinates of the background pixels, and j _offset is the synthetic background area. It is the difference between j _specified , which specifies the vertical coordinates of the arranged pixels, and j _actual , which specifies the vertical coordinates of the background pixels. Also, I(i,j) represents the intensity of the pixel value at the coordinates (i,j), H represents the height of the target image frame, and W represents the width of the target image frame.

In some embodiments, the background synthesizer may calculate the similarity between the background pixel 1703 and the synthesized background region 1704 . For example, the background synthesizing unit may calculate the mean square error between the background pixel 1703 and the synthetic background region 1704 as the degree of similarity between the background pixel 1703 and the synthetic background region 1704 . may be calculated, or a distance calculation method such as Euclidean distance may be used. Thereafter, in some embodiments, the background synthesizer may be trained to maximize the similarity between background pixels 1703 and synthetic background regions 1704 .

As described above, according to the seventh background synthesizing means 1700 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

FIG. 18 is a diagram showing an example of the eighth background synthesizing means 1800 according to the embodiment of the present disclosure.
The eighth background synthesizing means 1800 shown in FIG. 18 determines the affine transformation suitability of each region in the target image frame using the affine transformation suitability determination model, and then enlarges the image based on the affine transformation suitability of each region. By doing so, it is a means for converting the target image frame to a target aspect ratio, and is executed by a background synthesizing unit (for example, the background synthesizing unit 325 shown in FIG. 3).

First, the background synthesizing unit generates a grid image 1802 by dividing the target image frame 1801 into non-overlapping blocks having the same size. After that, the background synthesizing unit inputs this grid image 1802 to the affine transformation suitability determination model 1810 .
This affine transformation compatibility determination model 1810 is a model for determining the compatibility of each block included in the grid image 1802 with respect to the affine transformation. Affine transformations refer to transformations including translation (the process of moving all points by a fixed distance in a fixed direction) and linear transformations (scaling, shearing, rotation).
Further, the suitability for affine transformation here indicates the probability that distortion such as distortion or deformation does not occur in a specific region when affine transformation is performed on the region. In other words, a region that is highly compatible with affine transformation is difficult to deform even if affine transformation is performed. On the other hand, regions that are highly compatible with affine transformation tend to be deformed when affine transformation is applied.

ここでは、行列の各要素（a11, a12, a21, a22）は、異なるブロックのアフィン変換パラメータを示す。 By analyzing the grid image 1802 using the affine transformation suitability determination model 1810, it is possible to determine the suitability of each block included in the grid image 1802 for the affine transformation. The affine transform suitability determination model 1810 then calculates an affine transform parameter 1820 indicating the magnification factor for each block based on the determined suitability of each block for the affine transform. This affine transformation parameter 1820 may be expressed as a matrix shown in Equation 9, for example.

Here, each element of the matrix (a11, a12, a21, a22) indicates the affine transformation parameters of a different block.

Next, the background synthesizing unit enlarges (or reduces) each block in the grid image 1802 based on the enlargement ratio specified in the affine transformation parameter 1820 determined by the affine transformation suitability determination model 1810, thereby making the object A final image 1803 can be generated by converting the image frame 1801 to the desired aspect ratio.
Since this final image 1803 is generated based on the enlargement ratio specified in the affine transformation parameters 1820 determined by the affine transformation compatibility determination model 1810, blocks with high suitability for affine transformation (that is, blocks that are difficult to deform) blocks) are magnified larger.

As described above, according to the eighth background synthesizing means 1800 described above, an image in which the aspect ratio of the image is appropriately changed while maintaining the quality and semantic information of the image without causing distortion or deformation of the object. can be obtained.

In one embodiment, after the final image converted to the desired aspect ratio generated by the background compositing means described above is analyzed by the neural network, the background compositor uses a predetermined influence function to classify the class in the final image. , the degree of influence on the analysis result is calculated. Then, the image frame identification unit identifies an image containing a class that satisfies a predetermined influence criterion as a target image frame, and does not identify an image that does not contain a class that satisfies the predetermined influence criterion as a target image frame (that is, neural excluded from training or inference of the network).

ここで、L(z_i,θ)は損失関数であり、δはzにおける摂動である。
これにより、影響関数を用いることで、ニューラルネットワークによる解析結果に対して影響度の高いクラスを含む画像を効率良く選択することででき、ニューラルネットワークによる解析結果に対して影響度の低いクラスを含む画像を排除することができる。また、このように、ニューラルネットワークの訓練効率及び検出精度を向上させることができる。 The influence function here is a function for calculating the degree of influence that individual training data has on inference (by a learned model) in machine learning. As an example, the influence function of a data point z (eg, class) whose model parameter θ is included in Θ may be defined as shown in Equation 10 below and FIG.

where L(z _i , θ) is the loss function and δ is the perturbation in z.
As a result, by using the influence function, it is possible to efficiently select images that include classes that have a high degree of influence on the analysis results of the neural network, and include classes that have a low degree of influence on the analysis results of the neural network. Images can be eliminated. Also, in this way, the training efficiency and detection accuracy of the neural network can be improved.

According to the area conversion means according to the embodiment of the present disclosure described above, the aspect of the image is maintained while maintaining the quality and semantic information of the image by applying the background synthesizing means that considers the semantic content of the image. The ratio can be changed accordingly to facilitate highly accurate object detection. Also, the domain conversion means according to the embodiments of the present disclosure may be applied to problems in various fields. For example, the domain transformation means according to embodiments of the present disclosure can perform object detection in an image, object classification, object key-point detection, object tracking, image segmentation of images, processing or preprocessing of images input to neural networks, manipulation of robots, adjustment of images acquired from devices of different resolutions, generation of image thumbnails, performance benchmark parameter adjustment. adjustment), etc., may be applied to any field or subject.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present invention.

300 region conversion system 301 image acquisition device 302 storage unit 303 region conversion device 310 preprocessing unit 320 region conversion unit 321 image frame identification unit 322 region of interest detection unit 323 image frame processing unit 324 background synthesis means determination unit 325 background synthesis unit 330 neural Network 340 Output unit 304 Client terminal

Claims (10)

A domain conversion device,
an image frame identification unit that identifies a target image frame to be subjected to region conversion from the image sequence;
a region of interest detection unit that detects a region of interest in the target image frame and calculates reliability of a processing operation for extracting a region of interest image including the region of interest from the target image frame;
an image frame processing unit for extracting the region of interest image from the target image frame using the processing operation if the reliability of the processing operation satisfies a predetermined reliability criterion;
background pixels in the target image frame if the confidence of the manipulation operation does not meet a predetermined confidence criterion or if the region of interest image extracted from the target image frame does not meet a predetermined aspect ratio criterion; a background synthesizing means determination unit that determines, from a plurality of background synthesizing means candidates, a background synthesizing means for converting the target image frame to a predetermined aspect ratio by addition or deletion;
a background synthesizing unit that generates a final image in which the target image frame is converted to the predetermined aspect ratio by adding or deleting background pixels to or from the target image frame using the background synthesizing means;
A domain conversion device comprising: The background synthesizing unit
As the first background synthesizing means included in the candidates for the plurality of background synthesizing means,
identifying a saliency center in the target image frame;
generating a channel image by decomposing the target image frame into each channel constituting the target image frame;
calculating the median value of the pixels of said channel image;
generating a synthetic background image having the same number of channels as the target image frame, pixels having the median value of the channel image, and satisfying a predetermined size criterion for the size of the target image frame;
The final aspect ratio is converted to the predetermined aspect ratio by combining the synthetic background image and the target image frame by predetermined frame blending means based on the region of interest and the saliency center of the target image frame. generate an image,
2. The domain conversion device according to claim 1, characterized by: The background synthesizing unit
As the second background synthesizing means included in the candidates for the plurality of background synthesizing means,
generating a trained encoder-decoder model by training an encoder-decoder model that generates a first synthetic region that approximates the region of interest in the training image;
identifying a saliency center in the target image frame;
using the trained encoder-decoder model to generate a second synthetic region that approximates a background region in the target image frame;
The final image obtained by converting the target image frame to the predetermined aspect ratio by inserting the second composite region into the target image frame based on the region of interest and the saliency center of the target image frame. to generate
2. The domain conversion device according to claim 1, characterized by: The background synthesizing unit
As a third background synthesizing means included in the candidates for the plurality of background synthesizing means,
Generating a trained Gaussian process regression model by training a Gaussian process regression model that generates a first synthetic marginal region approximating a marginal region present at the margin of the training image,
identifying a saliency center in the target image frame;
using the trained Gaussian process regression model to generate a second synthetic fringe region that approximates a fringe region present at the fringes of the target image frame;
Based on the region of interest and the saliency center of the target image frame, the second synthetic peripheral region and the target image frame are combined by a predetermined frame blending means to convert to the predetermined aspect ratio. generating said final image with
2. The domain conversion device according to claim 1, characterized by: The background synthesizing unit
As the fourth background synthesizing means included in the candidates for the plurality of background synthesizing means,
dividing the target image frame into a first image portion and a second image portion in a region other than the region of interest;
generating overlapping patch seams across the first and second image portions using a Gaussian process regression model;
generating a processed first image portion and a processed second image portion by removing regions overlapped by the patch seam from the first image portion and the second image portion;
combining the processed first image portion and the processed second image portion to generate the final image in which the target image frame is converted to the predetermined aspect ratio;
2. The domain conversion device according to claim 1, characterized by: The background synthesizing unit
As the fifth background synthesizing means included in the candidates for the plurality of background synthesizing means,
generating the final image by converting the target image frame to the predetermined aspect ratio by adding background pixels having a constant pixel value to the target image frame;
2. The domain conversion device according to claim 1, characterized by: The domain conversion device is
further comprising a neural network that performs a predetermined analysis process on the final image and generates an analysis result;
2. The domain conversion device according to claim 1, characterized by: The background synthesizing unit
Using a predetermined influence function, calculate the influence of the class in the final image on the analysis result,
The image frame identification unit
identifying an image containing a class that satisfies a predetermined impact criterion as the target image frame;
8. The domain conversion device according to claim 7, characterized by: A domain conversion method comprising:
identifying target image frames to be domain-transformed from an image sequence;
detecting a region of interest in the target image frame and calculating a reliability of a processing operation for extracting a region of interest image including the region of interest from the target image frame;
extracting the region of interest image from the target image frame using the manipulation operation if the confidence of the manipulation operation meets a predetermined confidence criterion;
background pixels in the target image frame if the confidence of the manipulation operation does not meet a predetermined confidence criterion or if the region of interest image extracted from the target image frame does not meet a predetermined aspect ratio criterion; a step of determining, from a plurality of candidates of background synthesizing means, a background synthesizing means for converting the target image frame to a predetermined aspect ratio by adding or deleting;
converting the target image frame to the predetermined aspect ratio by adding or deleting background pixels to or from the target image frame using the background synthesizing means;
A domain transformation method comprising: A domain conversion system,
The domain conversion system includes:
an image acquisition device for acquiring an image sequence;
an area transformation device for performing area transformation on an image;
a client terminal;
the image acquisition device, the domain conversion device, and the client terminal are connected via a communication network;
The domain conversion device is
an image frame identification unit that receives the image sequence from the image acquisition device and identifies a target image frame to be subjected to region conversion from the image sequence;
a region of interest detection unit that detects a region of interest in the target image frame and calculates reliability of a processing operation for extracting a region of interest image including the region of interest from the target image frame;
an image frame processing unit for extracting the region of interest image from the target image frame using the processing operation if the reliability of the processing operation satisfies a predetermined reliability criterion;
background pixels in the target image frame if the confidence of the manipulation operation does not meet a predetermined confidence criterion or if the region of interest image extracted from the target image frame does not meet a predetermined aspect ratio criterion; a background synthesizing means determination unit that determines, from a plurality of background synthesizing means candidates, a background synthesizing means for converting the target image frame to a predetermined aspect ratio by addition or deletion;
a background synthesizing unit that generates a final image in which the target image frame is converted to the predetermined aspect ratio by adding or deleting background pixels to or from the target image frame using the background synthesizing means;
a neural network that performs a predetermined analysis process on the final image and generates an analysis result;
an output unit that transmits the analysis result from the neural network to the client terminal;
A domain conversion system comprising:

Images