KR20230063164A - Environment vector-based image generation method and apparatus - Google Patents

Abstract

An environment vector-based image generation method and apparatus are disclosed. The environmental vector-based image generation method according to an embodiment may include the operations of: selecting a background image from a background image database; selecting an object image from the object image database; and obtaining an environmental vector representing the brightness, contrast, and saturation of the background image from the background image; converting the object image based on the environment vector; segmenting an object in the converted object image; and blending the segmented object and the background image.

Description

Environment vector-based image generation method and apparatus {ENVIRONMENT VECTOR-BASED IMAGE GENERATION METHOD AND APPARATUS}

The disclosure below relates to a method and apparatus for generating an image based on an environment vector.

As single-person media becomes more active, interest in production technology for content serviced in single-person media is also increasing. One-person media services require tasks that are difficult for individuals to perform independently, such as video editing, transmission, and sound control, and in particular, video editing takes a lot of effort and time.

In order to add a new object to a part of the filmed or produced content, the video must be edited by selecting and setting the position to place the object, the number of objects to be placed, and the type of object.

In order to naturally add an object to an existing image, it is necessary to consider illuminance (color), camera angle, and object style so that the object matches the existing image.

According to various embodiments, a technique of generating a new image by converting an object image to match the background image based on an environment vector representing brightness, contrast, and saturation of the background image may be provided.

According to various embodiments, it is possible to provide an image conversion technology in which a user can intuitively control a generated image.

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

An environment vector-based image generation method according to an embodiment includes an operation of selecting a background image from a background image database, an operation of selecting an object image from an object image database, and brightness, contrast, and saturation of the background image from the background image. It may include obtaining an environment vector, transforming the object image based on the environment vector, segmenting an object from the transformed object image, and blending the segmented object and the background image.

Transforming the object image may include receiving an environment vector having a changed value of the environment vector and transforming the object image based on the changed environment vector.

An environment vector-based image generation method according to an embodiment includes an operation of selecting a background image from a background image database, an operation of selecting an object image from an object image database, and brightness and contrast of a partial region of the object image from the object image. , Obtaining an environment vector representing saturation, transforming the background image based on the environment vector, segmenting an object in the object image, and blending the segmented object and the transformed background image. can do.

Transforming the background image may include receiving an environment vector having a changed value of the environment vector and transforming the background image based on the changed environment vector.

An environment vector-based image generation method according to an embodiment includes an operation of selecting a background image from a background image database, an operation of selecting an object image from an object image database, and brightness, contrast, and saturation of the background image from the background image. An operation of obtaining an environment vector, an operation of receiving an environment vector whose value of the environment vector is changed, an operation of transforming the background image based on the changed environment vector, and an operation of transforming the object image based on the changed environment vector; An operation of segmenting the transformed object in the transformed object image and an operation of blending the segmented object and the transformed background image may be included.

1 is a schematic block diagram of an image generating device according to an exemplary embodiment.
2 is a flowchart of a method for generating an image according to an exemplary embodiment.
3 is a diagram for explaining an operation of transforming an image based on an environment vector.
4 is a diagram for explaining an operation of converting an object image.
5 is a diagram for explaining an operation of segmenting an object in an object image.
6 is a diagram for explaining an operation of converting a background image.
7 and 8 are diagrams for explaining an operation of generating a new image based on an environment vector of a background image.
9 is a diagram for explaining an operation of adjusting the conversion of an image by adjusting the value of an environment vector.
10 is a diagram for explaining an operation of generating a new image based on an environment vector of an object image.
11 is a diagram for explaining an operation of generating a new image by adjusting a value of an environment vector.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a schematic block diagram of an image generating device according to an exemplary embodiment, and FIG. 2 is a flowchart of an image generating method according to an exemplary embodiment.

The image generating device 100 may add a new object to a partial region of previously captured or produced content.

The image generating apparatus 100 may generate a new image by converting the object image to match the background image based on an environment vector representing the brightness, contrast, and saturation of all or a partial region of the background image, and may generate a new image, and may generate a new image. A new image may be generated by converting a background image to match an object image based on an environment vector representing brightness, contrast, and saturation of (eg, an area where an object exists).

The image generating apparatus 100 transforms an image based on an environment vector (an environment vector having a changed value of the environment vector) received from the user, so that the user can intuitively control the conversion of the image.

After the conventional image-to-image translation artificial neural network converts the image, it is difficult for the user to intuitively adjust it to make it darker or brighter, but the image generating device 100 converts the image. Even after this, the user can intuitively adjust the generated image by changing the value of the environment vector.

The user can adjust the conversion of the image as desired by individually adjusting brightness, contrast, and saturation values included in the environment vector.

The image generating device 100 may include a memory 110 and a processor 130 .

The memory 110 may store a background image database including a background image and an object image database including an object image.

The memory 110 may store instructions (eg, programs) executable by the processor 130 . For example, the instructions may include instructions for executing an operation of the processor 130 and/or an operation of each component of the processor 130 .

The processor 130 may process data stored in the memory 110 . The processor 130 may execute computer readable code (eg, software) stored in the memory 110 and instructions triggered by the processor 130 .

The processor 130 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program.

For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 130 may generate a new image by transforming an object image based on an environment vector of an entire region or a partial region (eg, a region to which an object is to be added) of the background image.

At operation 210, the processor 130 may select a background image from the background image database. The background image may be an image to which an object or the like may be added.

In operation 220, the processor 130 may select an object image from the object image database. The object image may be an image including an object to be added to the background image.

In operation 230, the processor 130 may obtain environment vectors representing brightness, contrast, and saturation of the background image from the background image. An operation of obtaining an environment vector will be described in detail with reference to FIG. 3 .

In operation 240, the processor 130 may transform the object image based on the environment vector. The processor 130 may receive an environment vector having a changed value of the environment vector from a user and transform an object image based on the changed environment vector.

In operation 250, the processor 130 may segment an object in the transformed object image. Also, the processor 130 may obtain an object region segmentation map in advance from an object image before transformation, and apply the segmentation map to the transformed object image to extract the transformed object.

In operation 260, the processor 130 may generate a new image by blending the segmented object and the background image.

The processor 130 based on the environment vector of a partial area (eg, the area where the object exists) of the object area A new image can be created by transforming the background image.

The processor 130 may select a background image from a background image database and an object image from an object image database.

The processor 130 may obtain an environment vector indicating brightness, contrast, and saturation of a partial region of the object image from the object image, and transform the background image based on the environment vector.

The processor 130 may receive an environment vector having a changed value of the environment vector from a user, and convert a background image based on the changed environment vector.

The processor 130 may generate a new image by segmenting an object in the object image and blending the segmented object and the transformed background image.

The processor 130 may generate a new image by transforming the background image (or object image) based on the environmental vector value of which the value of the environment vector of the background image (or object image) is changed.

The processor 130 may select a background image from a background image database and an object image from an object image database.

The processor 130 may obtain an environment vector representing brightness, contrast, and saturation of the background image from the background image, and may receive an environment vector having a changed value of the environment vector.

The processor 130 may transform the background image based on the changed environment vector and transform the object image based on the changed environment vector.

The processor 130 may generate a new image by segmenting the transformed object from the transformed object image and blending the segmented object and the transformed background image.

The image generating apparatus 100 may generate a new image by converting an object image to match the background image based on an environment vector representing brightness, contrast, and saturation of the background image, and may generate a new image, and may generate a partial area of the object image (the area where the object exists) ), a new image may be generated by converting the background image to suit the object image based on the environment vector representing brightness, contrast, and saturation.

The user can intuitively control the conversion of the image performed by the image generating apparatus 100 by changing the value of the environment vector. Specifically, the user can adjust the conversion of the image as desired by individually adjusting brightness, contrast, and saturation values included in the environment vector.

3 is a diagram for explaining an operation of transforming an image based on an environment vector.

An image generating device (the image generating device 100 of FIG. 1 ) may obtain environment vectors representing brightness, contrast, and saturation of an image.

The brightness included in the environment vector is an average of pixel-wise luminance, and may be calculated through Equation 1.

[Equation 1]

In Equation 1, R, G, and B may be a red value, a green value, and a blue value of a pixel.

Contrast included in the environment vector is a standard deviation of luminance for each pixel, and may be calculated through Equation 2.

[Equation 2]

는 픽셀 (i,j)의 명도 값이고,

는 영상 전체의 명도 평균 값이고, N은 높이, M은 넓이일 수 있다.In Equation 2,

is the brightness value of pixel (i,j),

is an average brightness value of the entire image, N may be a height, and M may be a width.

The degree of saturation included in the environment vector is an average of the degree of saturation for each pixel, and may be calculated through Equation 3.

[Equation 3]

In Equation 3, R, G, and B may be a red value, a green value, and a blue value of a pixel.

Values of the environment vector may be average values of all or part of the image. The environment vector may include a value obtained by normalizing brightness, contrast, and saturation values of the image. The brightness, contrast, and saturation values of the environment vector may be normalized to have a value between -1 and 1.

Referring to (A) of FIG. 3 , an original video (image) can be checked.

Referring to (B) of FIG. 3 , an image obtained by converting an original image based on an environment vector in which the value of the environment vector is changed to (-0.5, -0.5, -0.5) is generally darker than the original image, and the contrast and saturation It can be seen that is reduced.

Referring to (C) of FIG. 3, it can be seen that the image obtained by converting the original image based on the environmental vector whose value is changed to (0, 0, 0) is generally darker than the original image and the contrast is increased. there is.

Referring to (D) of FIG. 3 , an image obtained by converting an original image based on an environment vector in which the value of the environment vector is changed to (-0.8, -0.8, -0.8) is much darker than the original image and has a very high contrast. You can see that it is getting smaller.

The image generating apparatus 100 may convert an image by changing the brightness value, the contrast value, and the saturation value of the environment vector.

4 is a diagram for explaining an operation of transforming an object image, FIG. 5 is a diagram for explaining an operation of segmenting an object in an object image, and FIG. 6 is a diagram for explaining an operation of transforming a background image.

Referring to (A) of FIG. 4 , an original image including an object (ship) can be checked. Referring to (B) of FIG. 4 , a processor (processor 130 of FIG. 1 ) may generate an image converted to be darker than an original image based on a changed environment vector value. Referring to (C) of FIG. 4 , the processor 130 may generate a converted image having a richer contrast than the original image based on the changed environment vector value.

(A) of FIG. 5 may represent an original image including an object (ship), (B) of FIG. 5 may be an image obtained by segmenting an object from the original image, and (C) of FIG. It can be a segmented object. The processor 130 may segment and use an object in an object image. Specifically, the processor 130 may segment the transformed object in the transformed object image, obtain an object region segmentation map in advance from the object image before transformation, apply the segmentation map to the transformed object image, and perform transformation. object can be extracted.

Referring to (A) of FIG. 6 , an original image including an object (ship) can be checked. Referring to (B) of FIG. 6 , the processor 130 may generate a converted image that is darker than the original image based on the changed environment vector value. Referring to (C) of FIG. 6 , the processor 130 may generate a converted image having a richer contrast than the original image based on the changed environment vector value.

7 and 8 are diagrams for explaining an operation of generating a new image based on an environment vector of a background image.

7(A) and 8(A) may be original background images, and FIGS. 7(B) and 8(B) may be object images converted based on environment vectors of the background images. 7(C) and 8(C) may be new images generated by blending the background image and the converted object image.

The originals of the object images used in FIGS. 7 and 8 are the same, but the originals of the background images used in FIGS. 7 and 8 may be different. The background image of FIG. 8 may be darker than the background image of FIG. 7 . Accordingly, the object image converted based on the environment vector of the background image of FIG. 8 may be darker than the object image converted based on the environment vector of the background image of FIG. 7 .

The new images generated in FIGS. 7 and 8 may be generated by converting an object image to match the background image based on an environment vector of the background image. The new image generated in FIG. 7 may be an image generated by blending a bright background image and a brightly converted object image, and the new image generated in FIG. 8 may be an image generated by blending a dark background image and a darkly converted object image. can be

9 is a diagram for explaining an operation of adjusting the conversion of an image by adjusting the value of an environment vector.

(A) of FIG. 9 may be an original background image, (B) of FIG. 9 may be an object image converted based on an environment vector having a changed value of the environment vector, and (C) of FIG. 9 may be a background image. It may be a new image generated by blending the converted object image with .

Referring to (A) and (B) of FIG. 9 , the original background image may be dark, but the converted object image may be bright. Comparing the new image (FIG. 9(C)) generated in FIG. 9 with the new image (FIG. 7(C), FIG. 8(C)) generated in FIGS. 7 and 8, In the new image, the background image and the object image may not match. However, the new image generated in FIG. 9 may have significance in that the user can intuitively control the conversion of the image by changing the value of the environment vector.

10 is a diagram for explaining an operation of generating a new image based on an environment vector of an object image.

10(A) may be a background image converted based on an environment vector of a partial area of an object image, FIG. 10(B) may be an original object image, and FIG. 10(C) may be a converted background image. It may be a new image generated by blending the background image and the object image.

Considering the original background image shown in (A) of FIG. 9, the converted background image of (A) of FIG. 10 is converted to match the region where the object of the object image of FIG. it could be a video.

The new image generated in FIG. 10 may be generated by converting a background image to suit the object image based on an environment vector of a partial region of the object image. The new image generated in FIG. 10 may be an image generated by blending a bright object and a brightly converted background image.

11 is a diagram for explaining an operation of generating a new image by adjusting a value of an environment vector.

11(A) may be a background image transformed based on the changed environment vector, FIG. 11(B) may be an object image transformed based on the changed environment vector, and FIG. 11(C) may be transformed. It may be a new image created by blending the converted background image and the converted object image.

The image generating device (the image generating device 100 shown in FIG. 1 ) may generate a new image by converting the object image to match the background image based on an environment vector representing brightness, contrast, and saturation of the background image, A new image may be generated by converting the background image to match the object image based on an environment vector representing brightness, contrast, and saturation of a partial region of the object image (the region where the object exists).

The user can intuitively control the conversion of the image performed by the image generating apparatus 100 by changing the value of the environment vector. Specifically, the user can adjust the conversion of the image as desired by individually adjusting brightness, contrast, and saturation values included in the environment vector.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or the components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

selecting a background image from a background image database;
selecting an object image from an object image database;
obtaining environment vectors representing brightness, contrast, and saturation of the background image from the background image;
transforming the object image based on the environment vector;
segmenting an object in the transformed object image; and
Blending the segmented object and the background image
Including, environment vector-based image generation method.
According to claim 1,
The operation of converting the object image,
receiving an environment vector in which a value of the environment vector is changed; and
Transforming the object image based on the changed environment vector
Including, environment vector-based image generation method.
selecting a background image from a background image database;
selecting an object image from an object image database;
obtaining an environment vector representing brightness, contrast, and saturation of a partial region of the object image from the object image;
converting the background image based on the environment vector;
segmenting an object in the object image; and
Blending the segmented object and the transformed background image
Including, environment vector-based image generation method.
According to claim 3,
The operation of converting the background image,
receiving an environment vector in which a value of the environment vector is changed; and
Converting the background image based on the changed environment vector
Including, environment vector-based image generation method.
selecting a background image from a background image database;
selecting an object image from an object image database;
obtaining environment vectors representing brightness, contrast, and saturation of the background image from the background image;
receiving an environment vector whose value is changed;
converting the background image based on the changed environment vector;
transforming the object image based on the changed environment vector;
segmenting the transformed object in the transformed object image; and
Blending the segmented object and the converted background image
Including, environment vector-based image generation method.

Images