KR102236904B1 - Method and apparatus for compositing images - Google Patents

Abstract

Disclosed are a method and an apparatus thereof for synthesizing images by executing an artificial intelligence (AI) algorithm and/or a machine learning algorithm in a 5G environment connected for Internet of Things. According to one embodiment of the present invention, the image synthesis method includes the steps of: obtaining a first image including a face image; detecting a characteristic point of the first image by applying a pre-trained deep neural network model; obtaining a second image to be synthesized with the first image; extracting a border of the second image; matching a coordinate value corresponding to the border of the second image based on a coordinate value of characteristic values of the first image; and merging and outputting the first and second images.

Description

Image composition method and apparatus {METHOD AND APPARATUS FOR COMPOSITING IMAGES}

The present invention relates to a method and apparatus for synthesizing images by executing an onboard artificial intelligence (AI) algorithm and/or a machine learning algorithm.

Recently, the spread of smartphones has become popular, and the performance of small cameras attached to mobile devices including smartphones is steadily developing. Not only can you easily take a picture of yourself using such a small camera, but you can also modify the picture to suit your taste based on the taken picture. As a result, software that uses photographed photos is constantly being developed. In particular, due to the development of the SNS (Social Network System) industry, the production volume and spread (propagation) speed of digital contents are increasing exponentially.

Meanwhile, in order to photograph a face with a small camera attached to a mobile device, correct it based on the photographed picture, or synthesize another image, a face detection technology must be the basis. Face detection is a branch of computer vision, a technology that tells the location of a face in an image. Face detection can be implemented through various algorithms.

Unexamined Patent Publication No. 10-1206132 (corresponds to the disclosure case described in the invention report)

Registered Patent Publication No. 10-1862128 (corresponds to the disclosure case described in the invention report)

However, in the case of conventional image synthesis software (applications, etc.), when a face recognition technology is applied to an actual environment, recognition performance is degraded depending on the surrounding environment, leading to a problem of inaccurate face image synthesis. In addition, the compositing technology installed in the conventional image compositing software does not consider the position or size of the face in the image, and combines the image at an arbitrary size at a preset position or a position set by the developer to achieve harmony with the face in the image. There is a problem that cannot be done.

The above-described background technology is technical information possessed by the inventor for derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily known to be known to the general public prior to filing the present invention.

An object of the present disclosure is to enable an onboard artificial intelligence (AI) algorithm and/or a machine learning algorithm to be executed to synthesize an image.

An object of the present disclosure is to extract feature points from an image including a face based on artificial intelligence deep learning technology, so that a face in the image and an image to be synthesized can be naturally synthesized.

An object of the present disclosure is to enable optimal image synthesis by combining a technology of an intelligent level such as face recognition technology and face feature point detection, and a technology for accurately reducing an image.

An object of the present disclosure is to detect accurate facial feature points using a predefined deep learning network, thereby improving recognition performance irrespective of the surrounding environment, and enabling accurate face image synthesis.

An object of the present disclosure is to synthesize an image at an optimal position in consideration of the position or size of a face in the image so that it can be harmonized with the face in the image.

The object of the embodiments of the present disclosure is not limited to the above-mentioned problems, and other objects and advantages of the present invention that are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. will be. In addition, it will be appreciated that the objects and advantages of the present invention can be realized by the means shown in the claims and combinations thereof.

An image synthesis method according to an embodiment of the present disclosure may include executing an onboard artificial intelligence (AI) algorithm and/or a machine learning algorithm to synthesize an image. .

Specifically, the image synthesis method according to an embodiment of the present disclosure includes obtaining a first image including a face image, detecting a feature point of the first image by applying a pretrained deep neural network model, and Acquiring a second image to be combined with the first image, extracting the boundary of the second image, and matching coordinate values corresponding to the boundary of the second image based on the coordinate values of the feature points of the first image. And merging and outputting the first image and the second image.

Through the image synthesis method according to an embodiment of the present disclosure, by extracting feature points based on artificial intelligence deep learning technology for an image including a face, recognition performance can be improved regardless of the surrounding environment, and face recognition technology And an image to be synthesized with a face in the image may be naturally synthesized by combining an intelligent level technology such as detecting feature points of a face and a technology for accurately reducing the image.

In addition, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present disclosure, by extracting feature points from an image including a face based on artificial intelligence deep learning technology, recognition performance may be improved regardless of a surrounding environment.

In addition, by synthesizing an image by combining an intelligent level technology such as face recognition technology and face feature point detection, and a technology that accurately reduces the image, the face in the image and the image to be combined are naturally synthesized, thereby improving the performance of the image synthesis device. Can be improved.

In addition, by executing onboard artificial intelligence (AI) algorithms and/or machine learning algorithms to synthesize images, accurate facial feature points can be detected to enable accurate facial image synthesis. have.

In addition, by synthesizing the image at an optimal position in consideration of the position or size of the face in the image, it is possible to achieve harmony with the face in the image.

In addition, by implementing the facial feature point detection technology as a deep learning technology, a face in the image and an image to be synthesized can be naturally synthesized and applied to image services based on various faces.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram illustrating a deep neural network model according to an embodiment of the present disclosure.
2 is a block diagram schematically illustrating an image synthesizing apparatus according to an embodiment of the present disclosure.
3 is a block diagram schematically illustrating a processing unit according to an embodiment of the present disclosure.
4 is an exemplary diagram for describing boundary extraction according to an embodiment of the present disclosure.
5 and 6 are exemplary diagrams for describing feature point detection according to an embodiment of the present disclosure.
7 is an exemplary view showing an image synthesis result according to an embodiment of the present disclosure.
8 is a flowchart illustrating a method of synthesizing an image according to an embodiment of the present disclosure.

Advantages and features of the present invention, and a method of achieving them will be apparent with reference to embodiments described in detail together with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to completely inform the scope of the invention to those of ordinary skill in the art to which the present invention pertains. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as include or have are intended to designate the existence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and one or more other features, numbers, and steps. It is to be understood that it does not preclude the possibility of the presence or addition of, operations, components, parts, or combinations thereof. Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding components are given the same reference numbers, and redundant descriptions thereof are omitted. I will do it.

First, the present embodiment relates to detecting feature points of a face image based on a pretrained deep neural network model and synthesizing another image with the face image. In particular, in this embodiment, the deep neural network model may be a convolution neural network (CNN). In the present embodiment, briefly describing the image synthesis method, training may be performed to recognize a face from a face image and extract feature points. And after extracting the boundary of the image to be synthesized, the size can be adjusted corresponding to the face image. In addition, the result of synthesis may be output by merging the scaled image to be synthesized with the coordinates predicted in the face image.

Here, the feature points of the face image are specific parts of the face that can be reference points for synthesizing the image to be combined with the face image. For example, the ends of the eyebrows, the ends of the eyes, the pupil, the central end of the nose , Both end points of the lips, the center of the upper lip, the center of the lower lip, etc., and these points can be arbitrarily determined.

1 is a schematic diagram illustrating a deep neural network model according to an embodiment of the present disclosure.

The network structure of the CNN may include a convolutional layer and a pooling layer. The convolutional layer is intended to interpret how a system reacts when an input is applied to a particular system. The convolutional layer is mainly used for filter operation in the field of image processing, and can be used when implementing a filter for extracting specific features from an image. For example, if a 3*3 or more window or mask is repeatedly performed on the entire image, an appropriate result can be obtained according to the weight values of the mask. The input/output data in the convolutional layer may be referred to as a feature map.

The pooling layer can be used to prevent overfitting by reducing the space in the vertical and horizontal directions. Pooling may include max pooling, average pooling, and the like. The maximum pooling is an operation that takes the maximum value in the target area, and the average pooling is calculating the average of the target area.

As shown in Fig. 1, the CNN network is a feature extraction part that repeatedly stacks a convolution layer and a maxpooling layer, and a form in which all neurons of the previous layer are combined. It can be divided into a category that composes the fully connected layer, which is a layer of, and applies SoftMax to the last output layer.

For example, CNN applies a first-order convolution for an input image of size 96*96*1, using a kernel (variable to be trained) with a size of 5*5 first, and then applies a size of 96*96*n1. It is possible to generate feature maps with. At this time, the padding option may be set to valid. Next, the feature map generated by the first convolution may undergo max-pooling and an intermediate image having a size of 48*48*n1 may be generated. And again, a second order convolution is performed using a kernel having a size of 5*5, and feature maps having a size of 48*48*n2 can be generated. In addition, the feature map generated by the second convolution may undergo max-pooling again, and images having a size of 24*24*n2 may be generated. In the CNN of the present embodiment, 256 of them are selected and connected to the FC (fully connected) layer, and 128 of them are selected and connected to the FC, and finally 30 classes can be classified.

That is, in the convolution in CNN, coefficients for filtering are not fixed, but coefficient values can be determined through learning. In other words, the coefficient of the final convolution kernel may vary depending on the task to be processed through the CNN algorithm. Even for the same task, it may vary according to the learning data used for learning, and may vary according to the value of the set hyper-parameter. In this case, the value of the coefficient may be determined by back-propagation based on a gradient.

In addition, CNN is a neural network operation that utilizes the characteristics of convolution, and can have a feature of local connectivity. That is, the CNN can utilize local information similar to the receptive field. Here, the receptive region does not affect the entire external stimulus, but only a specific region, and may refer to a region of the input image that affects one pixel of the final output. In addition, the CNN can extract correlations between spatially adjacent signals by applying a nonlinear filter. Applying multiple such filters makes it possible to extract various local features. That is, if the size of the image is reduced while undergoing a subsampling process and filter operations for local features are repeatedly applied, global features can be gradually obtained. In addition, CNN can drastically reduce the number of variables by repeatedly applying filters having the same coefficients to the entire image, and can obtain invariance independent of changes in topology. On the other hand, the network structure of the CNN is not limited to the above description.

2 is a block diagram schematically illustrating an image synthesizing apparatus according to an embodiment of the present disclosure.

Referring to FIG. 2, the image synthesizing apparatus may include a communication interface 100, a camera 200, a sensing unit 300, a processor 400, a memory 500, a display unit 600, and a processing unit 700. have.

The communication interface 100 may be a communication means for receiving a first image including a face image and a second image to be combined with the first image. The communication interface 100 may receive a first image and a second image from the camera 200, and may receive an image through a server (not shown) or a separate input means. In addition, the communication interface 100 may transmit the received images to the processor 400.

The communication interface 100 may interwork with a network (not shown) to provide a transmission/reception signal between an image synthesizing apparatus and/or a server in the form of packet data. In addition, the communication interface 100 may transmit a predetermined information request signal from the image synthesizing apparatus to the server or transmit a predetermined information request signal from the server to the image synthesizing apparatus. In addition, the communication interface 100 may receive a response signal processed by the server and transmit it to the image synthesizing device, or may receive a response signal processed by the image synthesizing device and transmit it to the server.

In addition, the communication interface 100 may be a device including hardware and software necessary for transmitting and receiving a signal such as a control signal or a data signal through a wired or wireless connection with another network device.

In addition, the communication interface 100 may support various intelligent communication of things (internet of things (IoT), internet of everything (IoE), internet of small things (IoST), etc.)), and machine to machine (M2M) communication, V2X (vehicle to everything communication) communication, D2D (device to device) communication, etc. can be supported.

Meanwhile, the server may be a database server that provides big data required to apply various artificial intelligence algorithms and data for operating an image synthesis device. Depending on the processing capability of the image synthesizing device, functions performed in the server may vary.

In addition, when the server is an AI server, the server may include a server that performs AI processing and a server that performs operations on big data. In addition, the server may include a web server or an application server that enables a user to use the image synthesis system by using an image synthesis system application installed in a user terminal (not shown) or an image synthesis system web browser.

Here, the user terminal may be provided with a service for operating or controlling the image synthesis system through an authentication process after accessing the image synthesis system application or the image synthesis system site. In this embodiment, the user terminal that has completed the authentication process may operate and control the image synthesis system. In the present embodiment, the user terminal is a desktop computer, a smartphone, a notebook computer, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, and a global positioning system (GPS) device operated by the user. , E-book terminals, digital broadcasting terminals, navigation, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but are not limited thereto. In addition, the user terminal may be a wearable terminal such as a watch, glasses, hair band, and ring having a communication function and a data processing function. The user terminal is not limited to the above description, and a terminal capable of web browsing may be borrowed without limitation.

Meanwhile, the server is connected to the AI devices through a network, and may help at least part of the AI processing of the connected AI devices. The AI device may include, for example, not only a user terminal, but also a robot, an autonomous vehicle, an XR device, and a home appliance, and an image synthesis system environment may be configured through the AI devices. In this case, the server may train an artificial neural network according to a machine learning algorithm in place of the AI device, and may directly store the learning model or transmit it to the AI device.

Here, artificial intelligence (AI) is a field of computer science and information technology that studies how computers can do the thinking, learning, and self-development that human intelligence can do. It could mean being able to imitate intelligent behavior.

In addition, artificial intelligence does not exist by itself, but is directly or indirectly related to other fields of computer science. In particular, in modern times, attempts to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in the field are being made very actively.

Machine learning is a branch of artificial intelligence that can include a field of research that gives computers the ability to learn without explicit programming. Specifically, machine learning can be said to be a technology that studies and builds a system that learns based on empirical data, performs prediction, and improves its own performance, and algorithms for it. Rather than executing strictly defined static program instructions, machine learning algorithms can take a way to build specific models to derive predictions or decisions based on input data.

Meanwhile, the network may play a role of connecting the image synthesis device and the server. Such networks include wired networks such as LANs (local area networks), WANs (wide area networks), MANs (metropolitan area networks), ISDNs (integrated service digital networks), and wireless LANs, CDMA, Bluetooth, and satellite communications. The network may be covered, but the scope of the present invention is not limited thereto. In addition, the network may transmit and receive information using short-range communication and/or long-distance communication. Here, short-range communication may include Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, and wireless fidelity (Wi-Fi) technologies, and Communication includes code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), single carrier frequency division multiple access (SC-FDMA) technology. I can.

The network may also include the connection of network elements such as hubs, bridges, routers, switches and gateways. The network may include one or more connected networks, such as a multi-network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to the network may be provided through one or more wired or wireless access networks. Furthermore, the network may support an Internet of Things (IoT) network and/or 5G communication that exchanges and processes information between distributed components such as objects.

The camera 200 photographs the surrounding environment, and in particular, in the present embodiment, an image including a face may be photographed. The camera 200 is a means for capturing an image, and an image captured by the camera 200 may be transmitted to the processor 400.

The sensing unit 300 may include various sensors that sense the surrounding situation of the image synthesizing device. In this embodiment, information other than the camera 200 may be obtained through the sensing unit 300. For example, in the case of synthesizing a face image with an image corresponding to the user's age, not only age determination through face recognition but also age through the sensing unit 300 may be accurately determined to accurately determine an image corresponding to the age. .

Meanwhile, the sensing unit 300 may include an image sensor (not shown). The image sensor may include a camera (not shown) capable of photographing around the image synthesizing device, and a plurality of image sensors may be installed for photographing efficiency. In this case, the camera included in the image sensor may be the same module as the camera 200.

For example, the camera includes at least one optical lens and a plurality of photodiodes (eg, pixels) formed by light passing through the optical lens. ), and a digital signal processor (DSP) that configures an image based on signals output from photodiodes. The digital signal processor can generate not only still images but also moving images composed of frames composed of still images. Meanwhile, an image acquired by photographing a camera as an image sensor may be stored in the memory 500.

In this embodiment, the sensing unit 300 is not limited to an image sensor, and a sensor capable of detecting the surrounding situation of the image synthesis device, for example, a lidar sensor, a weight detection sensor, an illumination sensor. ), touch sensor, acceleration sensor, magnetic sensor, gravity sensor, gyroscope sensor, motion sensor, RGB sensor, infrared Sensor (IR sensor: infrared sensor), fingerprint sensor (finger scan sensor), ultrasonic sensor (ultrasonic sensor), optical sensor (optical sensor), microphone (microphone), battery gauge (battery gauge), environmental sensor (e.g. , A barometer, a hygrometer, a thermometer, a radiation sensor, a heat sensor, a gas sensor, etc.), a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, in the present embodiment, the image synthesizing apparatus may combine and utilize information sensed by at least two or more of these sensors.

The processor 400 may transmit the first image and the second image received through the camera 200 and/or the communication interface 100 to the processing unit 700. In addition, the processor 400 may match coordinate values corresponding to the boundary of the second image based on coordinate values of the feature points of the first image detected by applying the pretrained deep neural network model. In addition, the processor 400 may merge the first image and the second image and output it through the display unit 600.

Here, the pre-trained deep neural network model for detecting feature points is a supervised learning method to specify the location of the corresponding feature point when facial images are input using training data including a plurality of face images labeled with the feature points. It can be a trained neural network model. The processor 400 is a kind of central processing unit and may control the operation of the entire image synthesis apparatus by driving control software installed in the memory 500. The processor 400 may include all types of devices capable of processing data. Here, the'processor' may refer to a data processing device embedded in hardware, which has a circuit physically structured to perform a function represented by a code or instruction included in a program. As an example of a data processing device built into the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific integrated (ASIC) circuit), a field programmable gate array (FPGA), and the like, but the scope of the present invention is not limited thereto.

In this embodiment, the processor 400 may perform machine learning such as deep learning for the first image feature detection so that the image synthesis device outputs the optimal image synthesis result, and the memory 500 may store data, result data, and the like used for machine learning.

Deep learning technology, a kind of machine learning, can learn by going down to the deep level in multiple stages based on data. Deep learning can represent a set of machine learning algorithms that extract core data from a plurality of data as the level increases.

The deep learning structure may include an artificial neural network (ANN), and for example, the deep learning structure consists of a deep neural network (DNN) such as a convolutional neural network (CNN), a recurrent neural network (RNN), and a deep belief network (DBN). Can be. The deep learning structure according to the present embodiment may use various known structures. For example, the deep learning structure according to the present invention may include CNN, RNN, DBN, and the like. RNN is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes with the passage of time, and an artificial neural network structure can be constructed by stacking layers every moment. The DBN may include a deep learning structure constituted by stacking RBM (restricted boltzman machine), which is a deep learning technique, in multiple layers. By repeating RBM learning, when a certain number of layers is reached, a DBN having the corresponding number of layers can be configured. CNN can include a model that simulates human brain function, which is made based on the assumption that when a person recognizes an object, the basic features of the object are extracted, and then the brain undergoes complex calculations and recognizes the object based on the result. have.

On the other hand, learning of the artificial neural network can be accomplished by adjusting the weight of the connection line between nodes (if necessary, adjusting the bias value) so that a desired output is produced for a given input. In addition, the artificial neural network may continuously update the weight value by learning. In addition, a method such as back propagation may be used for learning of the artificial neural network.

The memory 500 may store various types of information necessary for the operation of the image synthesis device and control software capable of operating the image synthesis device, and may include a volatile or nonvolatile recording medium.

The memory 500 is connected to one or more processors, and when executed by the processor, may store codes that cause the processor to control the image synthesis apparatus.

Here, the memory 500 may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto. The memory 500 may include internal memory and/or external memory, and volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. A flash drive such as a compact flash (CF) card, an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

The display unit 600 is an output means for outputting an image synthesis result, and may include, for example, a display. The display may display the image synthesis process or result of the image synthesis apparatus under the control of the processor 400. Depending on the embodiment, the display may be configured as a touch screen by forming a layer structure with a touch pad. In this case, the display may also be used as a manipulation unit capable of inputting information by a user's touch. To this end, the display may be configured with a touch-sensitive display controller or various input/output controllers. Such a display may be, for example, a predetermined display member such as an organic light emitting display (OLED) capable of recognizing a touch, a liquid crystal display (LCD), or a light emitting display (LED).

In addition, in the present embodiment, an input means such as a microphone and a display may be provided, and a face image may be generated by analyzing the speech characteristics of a talker input to the microphone. In addition, in the present embodiment, a face image spoken by a speaker may be generated through a microphone or a face image may be generated based on text input through a display. Accordingly, a feature point of the generated face image may be detected, and an image to be synthesized may be matched and synthesized.

The processing unit 700 may perform learning in connection with the processor 400 or may receive a learning result from the processor 400. In this embodiment, the processing unit 700 may be provided outside the processor 400 as shown in FIG. 2, or may be provided inside the processor 400 to operate like the processor 400. Also, the processing unit 700 may be implemented in a server. That is, the processing unit 700 may process to perform optimal image synthesis based on codes stored in the memory 500. Details of the processing unit 700 will be described below with reference to FIG. 3.

3 is a block diagram schematically illustrating a processing unit according to an embodiment of the present disclosure. Parts that overlap with the description of FIG. 3 will be omitted.

3, the processing unit 700 includes a boundary extraction unit 710, a face recognition unit 720, a preprocessing unit 730, a feature point detection unit 740, a matching unit 750, a scale unit 760, and an output. It may include a unit 770.

The processing unit 700 may perform image synthesis by receiving a first image including a face image and a second image to be combined with the first image. In this case, the first image may be obtained through the camera 200, but may be received or generated from a user's input interface or a separate server. In addition, the second image may be received from a user or a separate server, but may be acquired through a camera or generated by a user or a separate server.

First, the boundary extraction unit 710 may extract the boundary of the second image. That is, the boundary extraction unit 710 may extract a contour from a two-dimensional array based on the level value of the second image and convert the second image into a grayscale image. In addition, the boundary extraction unit 710 may extract the boundary of the second image by calculating the intermediate value of the pixels in the second image and designating the intermediate value of the array.

More specifically, the boundary extraction unit 710 may calculate the strength and direction of the contour by obtaining a derivative value from a two-dimensional array based on the level value of the second image. An edge means a line representing the outer angle of an object in dictionary, and in the dimension of image processing, it can mean a line element that characterizes an image. That is, recognizing a boundary line and obtaining a pixel corresponding to an outline is called edge detection. The outline is where the value of a pixel changes abruptly in an image. That is, the outline may refer to a boundary between two areas having different intensity and intensity, and may refer to a portion in which the brightness of a pixel changes larger than a threshold value. Here, the threshold value may mean an arbitrary reference value for determining the presence or absence of a boundary.

The algorithm for extracting the contour can determine the size of the gradient vector that has been differentiated from the image. That is, the slope can be calculated based on the calculation of the partial differential operator. In other words, since the contour is a part where the shade value changes rapidly, a derivative operation that takes a change in the function can be used for contour extraction. Differentiation has a first derivative (gradient) and a second derivative (laplacian), and checks the existence of an outline in the image by the size of the graph slope at the first derivative value, and the outline pixel by the sign of the graph slope at the second derivative value. You can check the location of the bright and dark parts of the. The algorithm for extracting the contour may include sobel edge detection and canny edge detection.

In addition, the boundary extracting unit 710 may convert the second image into a grayscale image to find a'bright value' and a'dark value' in order to find an accurate boundary of the second image.

The boundary extractor 710 may extract the boundary of the second image by calculating the intermediate value of the pixels in the second image and designating the intermediate value of the array. That is, the image converted to grayscale can be binarized to extract an intermediate value and an outline. For example, you can use a mask as a way to apply a differentiation method to an image. In other words, it is possible to detect the contour line by determining the center pixel after calculation by covering 3*3 pixels of the image with a mask. The size of a mask is generally 3*3, but masks with a size of 5*5 and 7*7 are also applicable. When the mask is enlarged, the edge becomes thicker and can be clearly seen.

That is, the boundary extraction unit 710 may perform filtering taking an intermediate value among pixels in the mask area as a representative value. Using the intermediate value, the adaptive weight is set and filtered according to the difference from the surrounding value, so impulsive noise such as spot noise can be reduced, smoothing of the image (edge or boundary protection) is possible, and the sharpness difference is large. Without it, noise reduction can be performed.

In this embodiment, the boundary extraction unit 710 may apply a smoothing filter such as a median filter to remove noise before performing the final edge detection and apply a spatial filter to detect the edge. That is, the boundary extraction unit 710 may perform a recursive call on the candidate homogeneous region determined based on the intermediate value of the array, and calculate an average value of changes in pixel values of the second image as a result of performing the recursive call. Further, the boundary extraction unit 710 detects the final homogeneous region of the second image based on the average value, determines the arrangement of the final homogeneous regions to be combined with the first image from the second image, and extracts the boundary of the second image. can do.

That is, the boundary extraction unit 710 may regard pixels having similar characteristics as the same region through comparison with neighboring pixels based on one or more reference pixels. The boundary extraction unit 710 may set pixels considered to be the same area as adjacent areas. In addition, the boundary extraction unit 710 may apply and compare a statistical method capable of determining whether the property is similar.

4 is an exemplary diagram for describing boundary extraction according to an embodiment of the present disclosure. For example, the boundary extraction unit 710 allocates a label to each pixel, and based on this, assumes that if the difference in brightness value is less than 10, it is assumed to be viewed as the same region, and boundary extraction based on the homogeneous region You can do it. As shown in FIG. 4, the boundary extraction unit 710 may determine the beard-shaped region and the external region as homogeneous regions, respectively. Accordingly, the boundary extraction unit 710 may extract a beard-shaped boundary. However, the boundary extraction method is not limited thereto, and boundary extraction may be performed by various algorithms.

5 and 6 are exemplary diagrams for describing feature point detection according to an embodiment of the present disclosure.

Referring to FIG. 5, the face recognition unit 720 may adjust the number of face regions included in the first image. That is, the face recognition unit 720 may extract a face candidate region included in the first image and calculate the accuracy of the face candidate region. In addition, the face recognition unit 720 may determine the final face region based on the accuracy of the face candidate region, and adjust the number of face regions included in the first image. Here, the process of adjusting the number of face regions may be a process of determining a face region to be targeted for feature point detection among the plurality of candidate face regions.

That is, the face recognition unit 720 may recognize a face area from an image corresponding to an image signal acquired through the camera 200. In this case, the face recognition unit 720 may first extract a candidate region determined to be a face, and quantify the accuracy of the extracted candidate region based on a face recognition algorithm.

And, for example, the face recognition unit 720 is less sensitive to changes in lighting, detects skin color with an HR ratio that detects various skin colors, and determines an area of an arbitrary size as a candidate face area after dividing the area by labeling. I can. In addition, the face recognition unit 720 may detect a final face region by detecting eyes and a mouth using geometric location information such as facial features and color information in the candidate face region. However, the face recognition method is not limited to the above description, and various face recognition algorithms may be applied.

Here, the face recognition algorithm may be a neural network model trained to recognize a human face, and the neural network model is based on training data including a plurality of human face images and a data set labeled as a human face. It may be a pretrained neural network model.

In addition, the training data may further include an image of an animal's face labeled as an animal's face in addition to the human face.

The neural network model trained for face recognition can recognize where the face image part is in the image input to the model, and can output the accuracy of determining that the face image of a person is a probability together.

Meanwhile, the face recognition unit 720 may acquire a 3D face profile by using 2D feature points of at least one image among a plurality of consecutively photographed similar images.

Referring to FIG. 6, the preprocessor 730 may perform preprocessing of a face area included in the first image. That is, the preprocessor 730 may perform preprocessing of the face area by adjusting at least one of the size, position, color, brightness, and direction of the finally determined face area.

The preprocessor 730 may perform a preprocessing operation to facilitate detection of the feature point of the first image. In this case, the preprocessor 730 may automatically adjust the size of the first image to correspond to the size of the input data for detecting the feature point. For example, the preprocessor 730 may extract a portion corresponding to the region of interest from the first image to form a square shape. In addition, the preprocessor 730 may change the first image to a size suitable for feature extraction and operation processing. Also, the preprocessor 730 may reduce or increase the size of the image to an appropriate range by using an interpolation method, and in some cases, the output may be the same as the input. In addition, the preprocessor 730 may subtract and divide a certain value from the R, G, and B pixels from the first image so that the brightness values of all pixels have a preset range. In addition to the above-described preprocessing method, the preprocessor 730 may perform preprocessing of changing values for preset categories in the first image to a preset range.

The feature point detection unit 740 may detect the feature point of the first image by applying a pretrained deep neural network model. In this embodiment, the pretrained deep neural network model is a training model trained to extract feature points by inputting face data of the first image, and may be a learning model based on a convolution neural network (CNN). Accordingly, in the present embodiment, facial feature points can be quickly detected even in an unconstrained environment using a single model.

A facial landmark is a point displayed on a part that becomes a feature of the face, and may be displayed on the eyes, nose, mouth, and ears. That is, the feature point detection unit 740 may detect and track positions of eyebrows, eyes, nose, mouth, chin, and ears from the first image.

In this embodiment, 15 optimal feature points may be extracted using the deep neural network model, for example, with the center of the pupil as a starting point. In addition, in the present embodiment, through the detection of feature points, a contour line, a pupil, an eye shape, a nose shape, a mouth shape, a forehead shape, a cheekbone shape, a jaw shape, and the like may be detected. Furthermore, in the present embodiment, feature information of the face in the camera image may be extracted by calculating face ratio data including the width: height ratio, eye size, mouth size, forehead: eyebrow: tip of nose: tip of chin position, etc. have.

Meanwhile, in the present embodiment, a feature point may be extracted corresponding to a characteristic of the second image or a position to be synthesized. For example, when the second image is a beard, whiskers, hair style, or the like, the feature point extraction unit 740 may extract feature points (recognition areas) on the outer face of the face because it is important to recognize gender or face shape. In addition, the feature point extracting unit 740 may extract feature points within the face when age recognition, such as wrinkles, is based. Through this, the present embodiment can be applied to a program for estimating the transformed form of a criminal.

On the other hand, since the result of the deep neural network model trained in advance to find the feature point on the face as described above may be less accurate, a process of more elaborately determining the feature point may be added as described below.

Hereinafter, for convenience of description, feature points output by the deep neural network model may be referred to as primary feature points, and feature points determined through an elaboration process based on the first feature points may be referred to as secondary feature points.

The processor of the image synthesizing apparatus may apply a pretrained deep neural network model to the input first image to detect primary feature points in the first image including the face image.

Since the positions of the primary feature points may not exactly match the original target feature points, the processor uses a normal distribution of pixel values defined for each of the adjacent pixels of the primary feature points to obtain a more precise location of the second order feature. Feature points can be determined. Meanwhile, the pixel value may include a value of each pixel, such as color, brightness, and intensity of the pixel.

Here, pixels adjacent to the primary feature point that are the target of the normal distribution graph may be preferentially determined as pixels included in a window of a predetermined size around the primary feature point.

The processor generates a distribution graph for pixel values for adjacent pixels, evaluates the similarity between the generated distribution graph and the normal distribution, and specifies the window size if the generated distribution graph is similar to the normal distribution by a certain degree or more. , Find the vertex in the distribution graph targeting the pixels in the window.

As a result of similarity evaluation, if the generated distribution graph is less than a certain degree similar to the normal distribution, the window size is enlarged, and the similarity between the distribution graph and the normal distribution for pixels within the enlarged window size is reevaluated. Here, the enlarged size may be arbitrarily determined in consideration of the resolution of the image to be analyzed and the processing power of the processor.

This re-evaluation process increases the size of the window and repeats until the similarity reaches a predetermined threshold or higher, and when the similarity exceeds a predetermined threshold, the size of the window is determined and vertices can be identified in the distribution graph of the target adjacent pixels. have.

The processor may determine a position corresponding to the vertex as more elaborate secondary feature points, and feature points serving as a reference in a subsequent synthesis operation may be secondary feature points.

The matching unit 750 may match a coordinate value corresponding to the boundary of the second image based on the coordinate values of the feature points of the first image. That is, the matching unit 750 may extract a ratio of the coordinate value of the feature point of the first image and the coordinate value of the boundary of the second image, and calculate the length of the vector based on the coordinate value of the feature point of the first image. In other words, the matching unit 750 reduces or enlarges the second image by using the maximum and minimum values in the arrangement of the boundary of the second image based on the ratio and direction, based on the coordinate values of the feature points of the first image. As a result, coordinate values corresponding to the boundary of the second image may be matched.

Accordingly, in the present embodiment, even if the composite area of the second image to be combined with the first image is the same, different matching may be performed based on the feature points of the first image, and a different synthesis result may be provided.

The scale unit 760 may reduce or enlarge the second image based on a coordinate value corresponding to a position where the second image of the first image is to be synthesized. That is, the scale unit 760 may reduce or enlarge the second image by using the maximum value and the minimum value in the arrangement of the boundary of the second image based on the ratio and direction. In other words, the scale unit 760 may reduce or enlarge the second image within the range of the maximum value and the minimum value based on the ratio of the boundary between the feature points of the first image and the second image and the direction of the vector. The image can be changed to a size that can be combined with the first image. That is, the scale unit 760 may have the same position and posture with respect to the face region of the first image and the region to be combined with the second image. In this case, size information may be determined based on the size of the face. For example, size information may be determined so that the size of the recognized face and the size of the synthesis region of the second image are similar, so that the synthesis may be more natural. In addition, location information may be determined based on the location of the face. For example, position information may be determined in consideration of the position of the face so that the composite region of the second image is output so as not to cover the recognized face.

7 is an exemplary view showing an image synthesis result according to an embodiment of the present disclosure. As shown in FIG. 7, the output unit 770 may merge and output the first image and the second image. That is, the output unit 770 may merge and output the area to be combined with the second image with the face area of the first image. That is, in the present embodiment, based on a coordinate value corresponding to a position where the second image of the first image is to be combined, the reduced or enlarged second image may be combined with the first image and output.

Meanwhile, in the present embodiment, the face region of the first image is recognized, face information including at least one of age, gender, skin type, feature feature, and emotional state of the face region is detected, and the face information A second image to be combined with the first image may be set based on.

For example, as a result of analyzing the first image, if it is analyzed as being'female' and'child', content suitable for female children may be synthesized and outputted on the image. Here, the appropriate content may be a content selected according to a preset criterion among content previously stored in the database. For example, when a content most preferred by a girl child is set as a reference, the content most selected for a girl child may be set as a suitable content. The gender, age, race, skin type, and emotional state of the face image of the first image may be achieved through a neural network model trained to analyze each feature.

In addition, in this embodiment, by analyzing the first image, the content according to any one of gender recognition, age recognition, and facial expression recognition is read from a pre-stored database, and the read content is synthesized and outputted to the image. I can. For example, when it is analyzed that the gender of the face of the first image is a man and that the age is in his twenties, content for estimating the transformed appearance of the criminal based on the analysis result may be synthesized and output. That is, even if the user does not select the second image (content) to be synthesized, the user-customized content is synthesized based on information according to the user's face analysis, thereby providing a natural and interactive face synthesis service.

8 is a flowchart illustrating a method of synthesizing an image according to an embodiment of the present disclosure. Parts that overlap with the description of FIGS. 1 to 7 will be omitted.

Referring to FIG. 8, in step S100, the processor 400 acquires a first image including a face image. In this case, the first image may be acquired through the camera 200. However, the present invention is not limited thereto, and the first image may be input from a user through an input interface or may be received from a separate server. In addition, an image may be generated by describing an image input through an input interface.

In addition, in the present embodiment, after acquiring the first image, the number of face regions included in the first image may be adjusted. That is, the processor 400 may extract facial candidate regions included in the first image and calculate the accuracy of the facial candidate regions. Further, the processor 400 may determine the final face region based on the accuracy of the face candidate region, and adjust the number of face regions included in the first image.

Also, the processor 400 may perform preprocessing of the face area included in the first image. That is, the processor 400 may perform pre-processing of the face region by adjusting at least one of the size, position, color, brightness, and direction of the finally determined face region. That is, the processor 400 may perform a pre-processing operation to facilitate detection of the feature point of the first image.

In step S200, the processor 400 detects a feature point of the first image by applying the deep neural network model. Here, the pretrained deep neural network model is a training model trained to extract feature points by inputting face data of the first image, and may be a learning model based on a convolution neural network (CNN). Accordingly, in the present embodiment, facial feature points can be quickly detected even in an unconstrained environment using a single model.

A facial landmark is a point displayed on a part that becomes a feature of the face, and may be displayed on the eyes, nose, mouth, and ears. That is, the feature point detection unit 740 may detect and track positions of eyebrows, eyes, nose, mouth, chin, and ears from the first image. In this embodiment, 15 optimal feature points may be extracted using the deep neural network model, for example, with the center of the pupil as a starting point.

In step S300, the processor 400 acquires a second image to be combined with the first image. In this case, the second image may be received from a user or a separate server, but may be acquired through a camera or may be generated by a user or a separate server.

In step S400, the processor 400 extracts the boundary of the second image. That is, the processor 400 may extract an outline from the 2D array based on the level value of the second image and convert the second image into a grayscale image. In addition, the processor 400 may calculate an intermediate value of the pixels in the second image and designate an intermediate value of the array to extract a boundary of the second image. That is, the processor 400 may extract the boundary of the second image by calculating the median value of the pixels in the second image and designating the median value of the array. That is, the image converted to grayscale can be binarized to extract an intermediate value and an outline. For example, you can use a mask as a way to apply a differentiation method to an image. That is, the processor 400 may perform filtering taking as a representative value an intermediate value among pixels in the mask area. Since the filter process is performed by setting the adaptive weight according to the difference from the surrounding value using the intermediate value, impulsive noise such as spot noise can be reduced, smoothing of the image (edge or boundary protection) is possible, and the sharpness difference is large. Without it, noise reduction can be performed.

In the present embodiment, a smoothing filter such as a median filter may be applied to remove noise before the final edge detection is performed, and a spatial filter may be applied for edge detection. That is, the processor 400 may perform a recursive call on the candidate homogeneous region determined based on the intermediate value of the array, and calculate an average value of changes in pixel values of the second image as a result of performing the recursive call. Further, the processor 400 may detect the final homogeneous region of the second image based on the average value, determine the arrangement of the final homogeneous regions to be combined with the first image from the second image, and extract the boundary of the second image. have.

In step S500, the processor 400 matches the first image and the second image. That is, the processor 400 may match coordinate values corresponding to the boundary of the second image based on the coordinate values of the feature points of the first image. In other words, the processor 400 may extract a ratio of the coordinate value of the feature point of the first image and the coordinate value of the boundary of the second image, and calculate the length of the vector based on the coordinate value of the feature point of the first image. . That is, the processor 400 reduces or enlarges the second image using the maximum value and the minimum value in the arrangement of the boundary of the second image based on the ratio and direction, and provides a resolution based on the coordinate values of the feature points of the first image. 2 Coordinate values corresponding to the boundary of the image can be matched. Accordingly, in the present embodiment, even if the composite area of the second image to be combined with the first image is the same, different matching may be performed based on the feature points of the first image, and a different synthesis result may be provided.

In step S600, the processor 400 merges and outputs the first image and the second image. First, the processor 400 may reduce or enlarge the second image based on a coordinate value corresponding to a position where the second image of the first image is to be synthesized. That is, the processor 400 may reduce or enlarge the second image within the range of the maximum value and the minimum value, based on the ratio of the boundary between the feature point of the first image and the second image and the direction of the vector. It can be changed to a size that can be combined with the first image. In addition, the processor 400 may synthesize and output the reduced or enlarged second image with the first image based on a coordinate value corresponding to a position where the second image of the first image is to be synthesized.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium is a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magnetic-optical medium such as a floptical disk, and a ROM. , A hardware device specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to a person skilled in the computer software field. Examples of computer programs may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (especially in the claims), the use of the term "above" and the reference term similar thereto may correspond to both the singular and the plural. In addition, when a range is described in the present invention, the invention to which an individual value falling within the range is applied (unless otherwise stated), and each individual value constituting the range is described in the detailed description of the invention. Same as.

If there is no explicit order or contradictory description of the steps constituting the method according to the present invention, the steps may be performed in a suitable order. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or illustrative terms (for example, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited by the above examples or illustrative terms unless limited by the claims. It does not become. In addition, those skilled in the art can recognize that various modifications, combinations, and changes may be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

100: communication interface
200: camera
300: sensing unit
400: processor
500: memory
600: display
700: processing unit
710: boundary extraction unit
720: face recognition unit
730: pretreatment unit
740: feature point detection unit
750: matching unit
760: scale part
770: output

Claims (20)

As an image composition method,
Obtaining a first image including a face image;
Applying a pretrained deep neural network model to detect a plurality of primary feature points for each feature with the center of each feature of the face of the first image as a starting point;
Determining secondary feature points based on a normal distribution of pixel values defined for adjacent pixels of each of the primary feature points;
Obtaining a second image for combining with the first image;
Extracting a boundary of the second image;
Matching coordinate values corresponding to a boundary of the second image based on coordinate values of secondary feature points of the first image; And
Including the step of merging and outputting the first image and the second image,
The image synthesis method,
Recognizing a face area of the first image;
Detecting face information including two or more of age, gender, skin type, feature features, and emotional state of the face area; And
Further comprising setting a second image to be combined with the first image based on the at least two face information,
The step of determining the secondary feature points,
Generating a distribution graph of pixel values for adjacent pixels included in a window of a predetermined size based on the primary feature points;
Evaluating the similarity between the distribution graph and the normal distribution;
Increasing the size of the window and evaluating the degree of similarity; when the degree of similarity is greater than or equal to a predetermined threshold, determining the size of the window and confirming a vertex in a distribution graph of adjacent pixels; And
Including the step of determining a position corresponding to the vertex as the secondary feature points,
Image composition method.
The method of claim 1,
The step of obtaining the first image,
Adjusting the number of face regions included in the first image; And
Comprising the step of performing a preprocessing (preprocess) of the face region included in the first image,
Image composition method.
The method of claim 2,
Adjusting the number of face regions,
Extracting a face candidate region included in the first image;
Calculating the accuracy of the face candidate region; And
Including the step of determining a final face region based on the accuracy of the face candidate region,
Image composition method.
The method of claim 3,
The step of performing pre-processing of the face area,
Including the step of adjusting at least one or more of the determined size, position, color, brightness, and direction of the final face area,
Image composition method.
delete The method of claim 1,
Extracting the boundary of the second image,
Extracting an outline from the 2D array based on the level value of the second image;
Converting the second image to a grayscale image; And
Comprising the step of calculating the median value of the pixels in the second image to designate the median value of the array,
Image composition method.
The method of claim 1,
The pretrained deep neural network model,
Using training data including a plurality of face images labeled with feature points, when face images are input, a neural network model trained in advance to specify the positions of corresponding feature points in the input face images,

Image composition method.
The method of claim 6,
The matching step,
Extracting a ratio between the size of the image to be merged of the second image and the size of the face image of the first image based on coordinate values of secondary feature points of the first image and coordinate values of the boundary of the second image ;
Calculating a vector connecting the secondary feature points of the first image and the coordinate values of the border of the second image based on the coordinate values of the secondary feature points of the first image and the coordinate values of the boundary of the second image step; And
Reducing or expanding the second image based on the ratio and the length and direction of the vector,
Image composition method.
The method of claim 8,
The merging and outputting,
Comprising the step of synthesizing the reduced or enlarged second image with the first image based on a coordinate value corresponding to a position where the second image is to be combined in the first image,
Image composition method.
delete As an image synthesis device,
A communication interface for receiving a first image including a face image and a second image to be combined with the first image;
One or more processors; And
Including a memory connected to the one or more processors,
The memory, when executed by the processor, causes the processor to:
Applying a pretrained deep neural network model to detect a plurality of primary feature points for each feature with the center of each feature of the face of the first image as a starting point,
Secondary feature points are determined based on a normal distribution of pixel values defined for adjacent pixels of each of the primary feature points,
Extracting the boundary of the second image,
Matching a coordinate value corresponding to a boundary of the second image based on the coordinate values of the secondary feature points of the first image,
Store codes that cause the first image and the second image to be merged and output, and
The memory, when executed by the processor, causes the processor to
Recognizes the face region of the first image, detects face information including two or more of age, gender, skin type, feature feature, and emotional state of the face region, and based on the two or more face information, the first Additionally store codes that cause to set up a second image to be composited to the image,
The operation of determining the secondary feature points,
Generating a distribution graph of pixel values for adjacent pixels included in a window of a predetermined size based on the primary feature points;
Evaluating the similarity between the distribution graph and the normal distribution;
Increasing the size of the window and evaluating the similarity, determining the size of the window and checking a vertex in a distribution graph for adjacent pixels when the similarity is greater than or equal to a predetermined threshold; And
Including an operation of determining a position corresponding to the vertex as the secondary feature points,
Image compositing device.
The method of claim 11,
The memory, when executed by the processor, causes the processor to:
Storing codes that cause the number of face regions included in the first image to be adjusted and to perform a preprocess of the face regions included in the first image,
Image compositing device.
The method of claim 12,
The memory, when executed by the processor, causes the processor to:
A face candidate region included in the first image is extracted, an accuracy of the face candidate region is calculated, a final face region is determined based on the accuracy of the face candidate region, and the face region included in the first image is Storing the codes that cause the number to be adjusted,
Image compositing device.
The method of claim 13,
The memory, when executed by the processor, causes the processor to:
Adjusting at least one or more of the determined size, position, color, brightness, and direction of the final face region, and storing codes that cause preprocessing of the face region to be performed,
Image compositing device.
delete The method of claim 11,
The memory, when executed by the processor, causes the processor to:
Based on the level value of the second image, an outline is extracted from a two-dimensional array, the second image is converted to a grayscale image, and the median value of the pixels in the second image is calculated to calculate the median value of the array. To store codes that cause to extract the boundary of the second image,
Image compositing device.
The method of claim 11,
The pretrained deep neural network model,
A neural network model trained in advance to specify the positions of corresponding feature points when face images are input using training data including a plurality of face images labeled with feature points,
Image compositing device.
The method of claim 16,
The memory, when executed by the processor, causes the processor to:
Extracting a ratio of the size of the image to be merged of the second image and the size of the face image of the first image based on coordinate values of secondary feature points of the first image and coordinate values of the boundary of the second image, A vector connecting the secondary feature points of the first image and the coordinate values of the border of the second image is calculated based on the coordinate values of the secondary feature points of the first image and the coordinate values of the boundary of the second image, And storing codes that cause the second image to be reduced or enlarged based on the ratio and the length and direction of the vector.
The method of claim 18,
The memory, when executed by the processor, causes the processor to:
Storing codes that cause the reduced or enlarged second image to be combined with the first image based on a coordinate value corresponding to a position at which the second image of the first image is to be combined,
Image compositing device.
delete

Images