KR102637112B1 - Environment vector-based image generation method and apparatus - Google Patents

Abstract

A method and apparatus for generating an environmental vector-based image are disclosed. An environmental vector-based image generation method according to an embodiment includes selecting a background image from a background image database, selecting an object image from an object image database, and indicating brightness, contrast, and saturation of the background image from the background image. It may include obtaining an environment vector, converting the object image based on the environment vector, segmenting an object in the converted object image, and blending the segmented object and the background image.

Description

Environmental vector-based image generation method and device {ENVIRONMENT VECTOR-BASED IMAGE GENERATION METHOD AND APPARATUS}

The disclosure below relates to an environmental vector-based image generation method and device.

As one-person media becomes more active, interest in the production technology of content served by one-person media is also increasing. One-person media services require tasks that are difficult for an individual to perform alone, such as video editing, transmission, and sound control. In particular, video editing requires a lot of effort and time.

In order to add a new object to some area of filmed or produced content, the video must be edited by selecting and setting the location to place the object, the number of objects to be placed, and the type of object.

In order to naturally add an object to an existing image, illuminance (color), camera angle, and object style must be considered so that the object matches the existing image.

According to various embodiments, a technology for generating a new image by converting an object image to match the background image based on environmental vectors representing the brightness, contrast, and saturation of the background image can be provided.

According to various embodiments, it is possible to provide an image conversion technology that allows a user to intuitively control the generated image.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An environmental vector-based image generation method according to an embodiment includes selecting a background image from a background image database, selecting an object image from an object image database, and indicating brightness, contrast, and saturation of the background image from the background image. It may include obtaining an environment vector, converting the object image based on the environment vector, segmenting an object in the converted object image, and blending the segmented object and the background image.

The operation of converting the object image may include receiving an environment vector whose value of the environment vector has changed and converting the object image based on the changed environment vector.

An environmental vector-based image generation method according to an embodiment includes the operation of selecting a background image from a background image database, the operation of selecting an object image from an object image database, and the brightness and contrast of a portion of the object image from the object image. , including an operation of obtaining an environment vector indicating saturation, an operation of converting the background image based on the environment vector, an operation of segmenting an object in the object image, and an operation of blending the segmented object and the converted background image. can do.

The operation of converting the background image may include receiving an environment vector whose value of the environment vector has changed and converting the background image based on the changed environment vector.

An environmental vector-based image generation method according to an embodiment includes selecting a background image from a background image database, selecting an object image from an object image database, and indicating brightness, contrast, and saturation of the background image from the background image. Obtaining an environment vector, receiving an environment vector whose value has been changed, converting the background image based on the changed environment vector, and converting the object image based on the changed environment vector; It may include an operation of segmenting the converted object in the converted object image and an operation of blending the segmented object and the converted background image.

1 is a schematic block diagram of an image generating device according to an embodiment.
Figure 2 is a flowchart of an image generation method according to an embodiment.
Figure 3 is a diagram for explaining the operation of converting an image based on an environment vector.
Figure 4 is a diagram for explaining the operation of converting an object image.
Figure 5 is a diagram for explaining the operation of segmenting an object in an object image.
Figure 6 is a diagram for explaining the operation of converting a background image.
Figures 7 and 8 are diagrams for explaining the operation of generating a new image based on the environmental vector of the background image.
Figure 9 is a diagram for explaining the operation of controlling image transformation by adjusting the value of the environment vector.
Figure 10 is a diagram for explaining the operation of generating a new image based on the environment vector of the object image.
Figure 11 is a diagram to explain the operation of creating a new image by adjusting the value of the environment vector.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

FIG. 1 is a schematic block diagram of an image generating device according to an embodiment, and FIG. 2 is a flowchart of an image generating method according to an embodiment.

The image generating device 100 may add a new object to a partial area of existing captured or produced content.

The image generating device 100 can generate a new image by converting the object image to match the background image based on an environmental vector representing the brightness, contrast, and saturation of the entire or partial area of the background image, and a partial area of the object image. A new image can be created by converting the background image to match the object image based on the environmental vector representing the brightness, contrast, and saturation of the area (e.g., the area where the object exists).

The image generating device 100 converts the image based on the environmental vector received from the user (an environmental vector whose value has been changed), thereby allowing the user to intuitively control the conversion of the image.

After the existing image-to-image translation artificial neural network converted the image, it was difficult to intuitively adjust it even if the user wanted to make it darker or brighter, but the image generating device 100 converted the image. Even after this, users can intuitively adjust the generated image by changing the environment vector value.

Users can adjust the conversion of the image as desired by individually adjusting the brightness, contrast, and saturation values included in the environment vector.

The image generating device 100 may include a memory 110 and a processor 130.

The memory 110 may store a background image database including a background image and an object image database including object images.

The memory 110 may store instructions (eg, programs) that can be executed by the processor 130 . For example, the instructions may include instructions for executing the operation of the processor 130 and/or the operation of each component of the processor 130.

The processor 130 may process data stored in the memory 110. The processor 130 may execute computer-readable code (eg, software) stored in the memory 110 and instructions triggered by the processor 130 .

The processor 130 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The processor 130 may generate a new image by converting the object image based on the environment vector of the entire area or a partial area (eg, an area to which an object is to be added) of the background image.

In operation 210, the processor 130 may select a background image from the background image database. The background image may be an image to which objects, etc. can be added.

In operation 220, the processor 130 may select an object image from the object image database. The object image may be an image that includes an object to be added to the background image.

In operation 230, the processor 130 may obtain an environment vector representing the brightness, contrast, and saturation of the background image from the background image. The operation of acquiring the environmental vector will be explained in detail with reference to FIG. 3.

In operation 240, the processor 130 may convert the object image based on the environment vector. The processor 130 may receive an environment vector whose value of the environment vector has changed from the user and convert the object image based on the changed environment vector.

In operation 250, the processor 130 may segment an object in the converted object image. Additionally, the processor 130 may obtain an object area segmentation map in advance from an object image before conversion and extract the converted object by applying the segmentation map to the converted object image.

In operation 260, the processor 130 may create a new image by blending the segmented object and the background image.

The processor 130 may generate a new image by converting the background image based on the environmental vector of a partial area of the object area (eg, an area where the object exists).

The processor 130 may select a background image from the background image database and select an object image from the object image database.

The processor 130 may obtain an environment vector representing the brightness, contrast, and saturation of a portion of the object image from the object image, and convert the background image based on the environment vector.

The processor 130 may receive an environmental vector whose value of the environmental vector has changed from the user and convert the background image based on the changed environmental vector.

The processor 130 may segment an object in an object image and create a new image by blending the segmented object and the converted background image.

The processor 130 may generate a new image by converting the background image and the object image based on an environment vector in which the value of the environment vector of the background image (or object image) has changed.

The processor 130 may select a background image from the background image database and select an object image from the object image database.

The processor 130 may obtain an environment vector representing the brightness, contrast, and saturation of the background image from the background image, and receive an environment vector whose value has been changed.

The processor 130 may convert the background image based on the changed environment vector and convert the object image based on the changed environment vector.

The processor 130 may segment the converted object from the converted object image and create a new image by blending the segmented object and the converted background image.

The image generating device 100 can generate a new image by converting the object image to match the background image based on the environmental vector representing the brightness, contrast, and saturation of the background image, and a partial area of the object image (area where the object exists). ), a new image can be created by converting the background image to match the object image based on the environmental vector representing the brightness, contrast, and saturation.

The user can intuitively control the image conversion performed by the image generating device 100 by changing the value of the environment vector. Specifically, the user can control the conversion of the image as desired by individually adjusting the brightness, contrast, and saturation values included in the environment vector.

Figure 3 is a diagram for explaining the operation of converting an image based on an environment vector.

The image generating device (image generating device 100 in FIG. 1) can obtain environmental vectors representing the brightness, contrast, and saturation of the image.

The brightness included in the environment vector is the average of pixel-wise luminance and can be calculated through Equation 1.

[Equation 1]

In Equation 1, R, G, and B may be the red, green, and blue values of the pixel.

The contrast included in the environment vector is the standard deviation of luminance for each pixel, and can be calculated through Equation 2.

[Equation 2]

In equation 2, is the brightness value of pixel (i,j), is the average brightness value of the entire image, N may be the height, and M may be the width.

The saturation included in the environment vector is the average of the saturation for each pixel, and can be calculated through Equation 3.

[Equation 3]

In Equation 3, R, G, and B may be the red, green, and blue values of the pixel.

The values of the environment vector may be average values of all or part of the image. The environment vector may include normalized brightness values, contrast values, and saturation values of the image. The brightness, contrast, and saturation values of the environment vector are normalized and can have values between -1 and 1.

Referring to (A) of FIG. 3, you can check the original video (image).

Referring to Figure 3 (B), the image converted from the original image based on the environmental vector whose value was changed to (-0.5, -0.5, -0.5) is overall darker than the original image, and the contrast and saturation are changed. You can see that has decreased.

Referring to Figure 3 (C), it can be seen that the image converted from the original image based on the environmental vector whose value was changed to (0, 0, 0) is overall darker and the contrast is greater than the original image. there is.

Referring to (D) in Figure 3, the image converted from the original image based on the environmental vector whose value was changed to (-0.8, -0.8, -0.8) becomes much darker than the original image, and the contrast is very low. You can see that it has gotten smaller.

The image generating device 100 may convert the image by changing the brightness, contrast, and saturation values of the environment vector.

FIG. 4 is a diagram for explaining an operation for converting an object image, FIG. 5 is a diagram for explaining an operation for segmenting an object in an object image, and FIG. 6 is a diagram for explaining an operation for converting a background image.

Referring to (A) of FIG. 4, the original image including the object (ship) can be confirmed. Referring to (B) of FIG. 4, the processor (processor 130 of FIG. 1) may generate an image converted to be darker than the original image based on the changed environment vector value. Referring to (C) of FIG. 4, the processor 130 may generate a converted image with richer contrast than the original image based on the changed environment vector value.

(A) in Figure 5 may represent an original image including an object (ship), (B) in Figure 5 may be an image segmented from an object in the original image, and (C) in Figure 5 may represent an image containing an object (ship) in the original image. It may be a segmented object. The processor 130 may segment and use objects in an object image. Specifically, the processor 130 can segment the converted object from the converted object image, obtain an object area segmentation map in advance from the object image before conversion, and apply the segmentation map to the converted object image to convert it. objects can be extracted.

Referring to (A) of FIG. 6, the original image including the object (ship) can be confirmed. Referring to (B) of FIG. 6, the processor 130 may generate an image converted to be darker than the original image based on the changed environment vector value. Referring to (C) of FIG. 6, the processor 130 may generate a converted image with richer contrast than the original image based on the changed environment vector value.

Figures 7 and 8 are diagrams for explaining the operation of generating a new image based on the environmental vector of the background image.

7(A) and 8(A) may be the original background image, and FIGS. 7(B) and 8(B) may be object images converted based on the environment vector of the background image. 7(C) and FIG. 8(C) may be a new image created by blending the background image and the converted object image.

The original source of the object image used in FIGS. 7 and 8 is the same, but the original source of the background image used in FIGS. 7 and 8 may be different. The background image in FIG. 8 may be darker than the background image in FIG. 7 . Therefore, the object image converted based on the environment vector of the background image of FIG. 8 may be darker than the object image converted based on the environment vector of the background image of FIG. 7.

The new images generated in FIGS. 7 and 8 may be created by converting the object image to match the background image based on the environment vector of the background image. The new image created in FIG. 7 may be an image created by blending a bright background image and a brightly converted object image, and the new image created in FIG. 8 may be an image created by blending a dark background image and a darkly converted object image. It can be.

Figure 9 is a diagram for explaining the operation of controlling image transformation by adjusting the value of the environment vector.

Figure 9 (A) may be the original background image, Figure 9 (B) may be an object image converted based on an environment vector whose value of the environment vector has been changed, and Figure 9 (C) may be a background image It may be a new image created by blending the converted object image.

Referring to Figures 9 (A) and (B), the original background image may be dark, but the converted object image may be bright. Comparing the new image generated in FIG. 9 ((C) of FIG. 9) with the new image ((C) of FIG. 7 and (C) of FIG. 8) generated in FIGS. 7 and 8, the In a new video, the background video and object video may not match. However, the new image generated in FIG. 9 may be meaningful in that the user can intuitively control the transformation of the image by changing the value of the environment vector.

Figure 10 is a diagram for explaining the operation of generating a new image based on the environment vector of the object image.

Figure 10 (A) may be a background image converted based on the environment vector of a partial area of the object image, Figure 10 (B) may be the original object image, and Figure 10 (C) may be a converted background image. It may be a new image created by blending a background image and an object image.

Considering the original background image shown in (A) of FIG. 9, the converted background image of FIG. 10 (A) is a background converted to match the area where the object of the object image of FIG. 10 (B) exists. It could be a video.

The new image generated in FIG. 10 may be created by converting the background image to match the object image based on the environmental vector of a partial area of the object image. The new image generated in FIG. 10 may be an image created by blending a bright object and a brightly converted background image.

Figure 11 is a diagram to explain the operation of creating a new image by adjusting the value of the environment vector.

Figure 11 (A) may be a background image converted based on the changed environment vector, Figure 11 (B) may be an object image converted based on the changed environment vector, and Figure 11 (C) may be a converted background image. It may be a new image created by blending the converted background image and the converted object image.

The image generating device (image generating device 100 shown in FIG. 1) can generate a new image by converting the object image to match the background image based on environmental vectors representing the brightness, contrast, and saturation of the background image, A new image can be created by converting the background image to match the object image based on an environmental vector representing the brightness, contrast, and saturation of a portion of the object image (area where the object exists).

The user can intuitively control the image conversion performed by the image generating device 100 by changing the value of the environment vector. Specifically, the user can control the conversion of the image as desired by individually adjusting the brightness, contrast, and saturation values included in the environment vector.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (5)

An operation of selecting a background image from a background image database;
An operation of selecting an object image from an object image database;
Obtaining an environment vector representing brightness, contrast, and saturation of the background image from the background image;
converting the object image based on the environment vector;
An operation of segmenting an object in a converted object image; and
An operation of blending the segmented object and the background image
Including,
The operation of converting the object image is,
An image-to-image translation operation in which an artificial neural network transforms the object image based on the environment vector;
An operation of receiving an environmental vector whose value has changed from a user; and
An operation of generating the converted object image by adjusting the object image based on the changed environment vector.
Including,
The brightness, contrast, and saturation values of the environment vector are normalized to have values between -1 and 1,
Environmental vector-based image generation method.
delete An operation of selecting a background image from a background image database;
An operation of selecting an object image from an object image database;
Obtaining an environment vector representing brightness, contrast, and saturation of a portion of the object image from the object image;
converting the background image based on the environment vector;
Segmenting an object in the object image; and
An operation of blending the segmented object and the converted background image
Including,
The operation of converting the background image is,
An image-to-image translation operation in which an artificial neural network transforms the background image based on the environment vector;
An operation of receiving an environmental vector whose value has changed from a user; and
An operation of generating the converted background image by adjusting the background image based on the changed environment vector.
Including,
The brightness, contrast, and saturation values of the environment vector are normalized to have values between -1 and 1,
Environmental vector-based image generation method.
delete An operation of selecting a background image from a background image database;
An operation of selecting an object image from an object image database;
Obtaining an environment vector representing brightness, contrast, and saturation of the background image from the background image;
An image-to-image translation operation of an artificial neural network converting the object image and the background image based on the environment vector; An operation of receiving an environment vector with a changed value of the environment vector from a user;
generating a converted background image by adjusting the background image based on the changed environment vector;
generating a converted object image by adjusting the object image based on the changed environment vector;
Segmenting a converted object from the converted object image; and
Operation of blending segmented objects and the converted background image
Including,
The brightness, contrast, and saturation values of the environment vector are normalized to have values between -1 and 1,
Environmental vector-based image generation method.