KR102238036B1 - Method and system for image processing - Google Patents

Abstract

Regarding an image processing method and system, in more detail, an object such as a user's face included in an input image is mapped to a virtual object such as a face of a specific character, and a combination of values of components predefined for the object An image processing method and system capable of mapping a user's facial expression or a specific event to a character in real time by applying the event according to the virtual object, and a computer-readable recording in order to execute the image processing method in combination with a computer device. A computer program stored on the medium is started.

Description

Image processing method and system {METHOD AND SYSTEM FOR IMAGE PROCESSING}

The following description relates to an image processing method and system, and more specifically, a component that maps an object such as a user's face included in an input image to a virtual object such as a face of a specific character, and is predefined for the object. In order to execute an image processing method in combination with a computer device, an image processing method and system capable of mapping a user's facial expression or a specific event to a character in real time by applying an event according to the combination of values of the virtual object. It relates to a computer program stored in a computer-readable recording medium.

As a conventional technology for controlling a specific character such as an avatar, there is a technology for expressing an avatar more realistically by reflecting a user's facial expression on the avatar. For example, Korean Patent Application Publication No. 10-2012-0059994 relates to an apparatus and method for controlling an avatar using facial expression control points. By controlling the position of the feature point of the avatar according to the location of the feature point tracked in the user's facial expression, Can be applied to the avatar.

However, if the position of the feature point tracked from the user's face is applied to the avatar as it is, the user's facial expression may be well applied to the avatar, but there is a problem that an unnatural expression may appear on the avatar from the viewpoint of the avatar. For example, if a user's facial expression is mapped and applied to a character's face that does not resemble the user's face as it is, there is a problem that an expression that does not match the character's face at all can be obtained.

Determine values of predefined components (eg, eyes, nose, mouth, eyebrows, face angle, etc.) for an object (eg, a user's face) included in the input image, and By selecting one of the events set in advance for the virtual object according to the combination and applying it to the virtual object, changes in the object (for example, changes in facial expressions) are reflected in the virtual object, while at the same time more natural from the perspective of the virtual object It provides an image processing method and system capable of applying changes, and a computer program stored in a computer-readable recording medium to execute the image processing method in combination with a computer device.

A computer program combined with a computer and stored in a computer-readable recording medium to execute an image processing method, the image processing method comprising: extracting landmark information for an object included in an input image; Mapping an object included in the input image and a virtual object based on the landmark information; Determining a value of each of a plurality of components predefined for the object based on the landmark information; Determining an event corresponding to the determined combination of values of the plurality of components; And applying the determined event to a virtual object mapped to the object.

Extracting landmark information for an object included in the input image; Mapping an object included in the input image and a virtual object based on the landmark information; Determining a value of each of a plurality of components predefined for the object based on the landmark information; Determining an event corresponding to the determined combination of values of the plurality of components; And applying the determined event to a virtual object mapped to the object.

It provides a computer-readable recording medium, characterized in that a program for executing the image processing method on a computer is recorded.

Determine values of predefined components (eg, eyes, nose, mouth, eyebrows, face angle, etc.) for an object (eg, a user's face) included in the input image, and By selecting one of the events set in advance for the virtual object according to the combination and applying it to the virtual object, changes in the object (for example, changes in facial expressions) are reflected in the virtual object, and at the same time, more natural from the perspective of the virtual object. Change can be applied.

1 is a diagram showing an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention.
3 is a diagram illustrating an example of an image processing method according to an embodiment of the present invention.
4 is a diagram illustrating an example of a landmark component of a face object according to an embodiment of the present invention.
5 is a diagram illustrating an example of an angular component of a face object according to an embodiment of the present invention.
6 is a flowchart illustrating an example of a method of applying an event to a virtual object according to an embodiment of the present invention.
7 is a diagram illustrating an example of a plurality of motion frames included in a motion file according to an embodiment of the present invention.
8 is a diagram illustrating an example of normalizing landmark coordinates according to an embodiment of the present invention.
9 is a diagram illustrating an example of correcting landmark coordinates according to an embodiment of the present invention.
10 is a flowchart illustrating an example of a method for discriminating between blinking and winking of both eyes in an embodiment of the present invention.
11 is a flowchart illustrating an example of the overall service flow according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The image processing system according to the embodiments of the present invention may be implemented through an electronic device or a computer device such as a server, which will be described later, and the image processing method according to the embodiments of the present invention is performed through such an electronic device or a server. Can be. For example, an application implemented as a computer program according to an embodiment of the present invention may be installed and driven in an electronic device, and the electronic device processes an input image according to the control of the driven application, and an event is applied to a virtual object. Can be applied. The computer program may be combined with an electronic device and stored in a computer-readable recording medium to execute an image processing method. In another embodiment, the server may perform the image processing method according to the control of the above-described application.

In one embodiment, a computer device such as an electronic device or a server includes components (for example, eyes, nose, mouth, eyebrows, etc.) that are predefined for an object (for example, a user's face) included in an input image. Face angle, etc.), and by selecting one of the preset events for the virtual object and applying it to the virtual object according to the combination of the determined values, the change of the object (for example, change of facial expression) While reflecting on the virtual object, more natural changes can be applied from the perspective of the virtual object.

1 is a diagram showing an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, and 140, a plurality of servers 150 and 160, and a network 170. 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in FIG. 1.

The plurality of electronic devices 110, 120, 130, and 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smart phones, mobile phones, navigation, computers, notebook computers, digital broadcasting terminals, personal digital assistants (PDAs), and portable multimedia players (PMPs). ), tablet PC, etc. As an example, although the shape of a smartphone is shown as an example of the electronic device 1 110 in FIG. 1, in the embodiments of the present invention, other electronic devices 120 are substantially transmitted through the network 170 using a wireless or wired communication method. , 130, 140) and/or the servers 150 and 160 may mean one of various devices capable of communicating with each other.

The communication method is not limited, and not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network), but also a short-range wireless communication between devices may be included. For example, the network 170 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). And any one or more of networks such as the Internet. In addition, the network 170 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. Not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computers that communicates with a plurality of electronic devices 110, 120, 130, and 140 through a network 170 to provide commands, codes, files, contents, services, etc. It can be implemented with devices. For example, the server 150 may be a system that provides a first service to a plurality of electronic devices 110, 120, 130, and 140 accessed through the network 170, and the server 160 is also a network ( It may be a system that provides a second service to a plurality of electronic devices 110, 120, 130, and 140 accessed through 170). As a more specific example, the server 160 provides an installation file (for example, an application program package file) for installing an application (computer program) as a second service to a plurality of electronic devices 110, 120, 130, 140 It may be a system that does. In addition, the server 150 is a plurality of electronic devices (110, 120, 130, 140) providing services related to the above-described application as a first service (for example, a messaging service, a social network service, a content providing service, etc.) Can be provided as. These services are only one embodiment for describing a network environment, and in fact, services provided by each of the servers 150 and 160 in the network environment may be very diverse.

As described above, the image processing method according to embodiments of the present invention may be performed by an electronic device or a server communicating with the electronic device.

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention. . In FIG. 2, internal configurations of the electronic device 1 110 and the server 150 are described as examples of the electronic device. In addition, the other electronic devices 120, 130, 140 or the server 160 may also have the same or similar internal configuration as the electronic device 1 110 or the server 150 described above.

The electronic device 1 110 and the server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223, and input/output interfaces 214 and 224. The memories 211 and 221 are computer-readable recording media, and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the electronic device 1110 or the server 150 as a separate permanent storage device separated from the memories 211 and 221. In addition, the memories 211 and 221 contain an operating system and at least one program code (for example, a browser installed and driven in the electronic device 1 110 or an application installed in the electronic device 1 110 to provide a specific service). Code) can be saved. These software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memories 211 and 221 through communication modules 213 and 223 other than a computer-readable recording medium. For example, at least one program is a program installed by files provided by the file distribution system (for example, the server 150) for distributing the installation files of developers or applications (for example, the server 150). It may be loaded into the memories 211 and 221 based on the above-described application).

The processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223. For example, the processors 212 and 222 may be configured to execute a command received according to a program code stored in a recording device such as the memories 211 and 221.

The communication modules 213 and 223 may provide a function for the electronic device 1 110 and the server 150 to communicate with each other through the network 170, and the electronic device 1 110 and/or the server 150 May provide a function for communicating with another electronic device (eg, electronic device 2 120) or another server (eg, server 160 ). As an example, a request generated by the processor 212 of the electronic device 1 110 according to a program code stored in a recording device such as a memory 211 is transmitted through the network 170 under the control of the communication module 213. 150). Conversely, control signals, commands, contents, files, etc. provided under the control of the processor 222 of the server 150 are transmitted via the communication module 223 and the network 170 to the communication module of the electronic device 1 110 ( 213) through the electronic device 1 (110). For example, a control signal, command, content, file, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 or the memory 211, and the content or file is an electronic device 1 It may be stored in a storage medium (permanent storage device described above) that 110 may further include.

The input/output interface 214 may be a means for an interface with the input/output device 215. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display. As another example, the input/output interface 214 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. The input/output device 215 may be configured with the electronic device 1110 and one device. In addition, the input/output interface 224 of the server 150 may be a means for interface with a device (not shown) for input or output that is connected to the server 150 or that the server 150 may include. As a more specific example, the processor 212 of the electronic device 1 110 is configured by using data provided by the server 150 or the electronic device 2 120 in processing a command of a computer program loaded in the memory 211 The service screen or content to be displayed may be displayed on the display through the input/output interface 214.

In addition, in other embodiments, the electronic device 1 110 and the server 150 may include more components than those of FIG. 2. However, there is no need to clearly show most of the prior art components. For example, the electronic device 1 110 may be implemented to include at least some of the input/output devices 215 described above, or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. It may also contain more elements. As a more specific example, when the electronic device 1 110 is a smartphone, an acceleration sensor or a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and vibration Various components such as a vibrator may be implemented to be further included in the electronic device 1 110.

3 is a diagram illustrating an example of an image processing method according to an embodiment of the present invention. The image processing method according to the present exemplary embodiment may be performed by a computer device such as the electronic device 1 110 or the server 150 described above. In this embodiment, a process in which the electronic device 1 110 performs an image processing method will be described. The processor 212 of the electronic device 1 110 may be implemented to execute a control instruction according to a code of an operating system included in the memory 211 or a code of at least one computer program. Here, the processor 212 includes steps 310 to 350 in the image processing method according to the embodiment of FIG. 3 according to the control command provided by the code stored in the electronic device 1 110. The electronic device 1 110 may be controlled to perform the operation.

In step 310, the electronic device 1 110 may extract landmark information for an object included in the input image. Here, the object may include an object to be replaced with a virtual object such as a user's face included in the image, and the landmark information may include the position of the feature point (for example, the coordinates of the feature point in a virtual coordinate system related to the image) I can. A technique for controlling an avatar based on the location of a feature point has already been introduced, and a specific technique for extracting landmark information will be easily understood by those skilled in the art through known conventional techniques.

In step 320, the electronic device 1 110 may map an object included in an input image and a virtual object based on the landmark information. As in the above-described example, if the object is the user's face included in the image, the virtual object may mean a virtual object corresponding to the object included in the image, such as a user's avatar or a specific character's face.

For example, in the electronic device 1 110 replacing the face of a user included in the image with the face of a specific character, or providing the face of a specific character, the electronic device 1 110 may provide a facial expression of the user included in the image. It can be provided by applying it to the face of a specific character. As a more specific example, the electronic device 1 110 may provide a face of a user by replacing a face of a specific character in an image input through a camera for a video call. As another specific example, the electronic device 1 110 provides an alternative image including the face of a specific character instead of an image for a video call, and provides a user's facial expression in the image input through the camera on the face of the character in the alternative image. It can be applied and provided.

As another example, the electronic device 1 110 may generate and provide a recorded video file or a moving GIF format file by recording a screen of a character's face to which an event such as a user's facial expression or related effects is applied as a video. May be. In this case, the files created and provided may be shared with other users through a messaging service or a social network service. In this case, the electronic device 1 110 may add an additional effect (for example, a decoration effect) to the resulting image.

As already described, the image processing method may be performed through the server 150. For example, the electronic device 1 110 may sequentially transmit images input through a camera to the server 150 or may transmit a pre-stored image file to the server 150. In this case, the server 150 may perform the image processing method according to embodiments of the present invention on the transmitted and input image, and may provide the generated result to the electronic device 1 110 or other electronic devices.

In the following steps, a process in which a specific event is applied to a virtual object based on the extracted landmark information will be described.

In operation 330, the electronic device 1 110 may determine a value of each of a plurality of components predefined for the object based on the landmark information. Such a plurality of components may be predefined according to the type of object. For example, if the object is the user's face included in the image, the plurality of components may include items such as eyes, nose, mouth, eyebrows, and angle of the object. In this case, the value of a specific component may be a value indicating a shape or angle related to the component. For example, a component corresponding to'left eye' can have one of three preset values, such as'close eyes','open eyes', and'open half eyes (or close half eyes)'. have. In other words, when the'left eye' is open from the user's face included in the current frame of the input image, the value of the'left eye' component may be determined as'open eye'. These components and their values will be described in more detail later.

In operation 340, the electronic device 1 110 may determine an event corresponding to the determined combination of values of a plurality of components. For example, if the value of the'left eye' component is'open eyes' and the value of the'right eye' component is'close eyes', the electronic device 1 (110) can determine an event such as'right eye wink'. have. At this time, if the values of the'mouth' component are further combined, such as the shape of a mouth, the event can be more reliably determined. Also, the combination of values of a plurality of components need not be a combination of all values. For example, a value of one special component or a combination of values of two or more special components may be used except for general values.

In operation 350, the electronic device 1 110 may apply the determined event to the virtual object mapped to the object. Here, the event may be preset for a virtual object matching a specific object. In other words, for each virtual object, there may be events set in advance suitable for the corresponding virtual object, and since the event of the virtual object determined through the object is applied to the virtual object, more natural changes can be applied to the virtual object. For example, the facial expression of the user is not applied to the avatar as it is, but the facial expression (event) of the avatar corresponding to the facial expression of the user may be applied to the avatar. In this case, since the event may be predefined for each combination of values of a plurality of components, a very large number of events may be defined according to the number of the plurality of components and the number of values that can be predefined for one component. For example, if three values could be defined for the'left eye' component, three values for the'right eye' component, and three values for the'mouth', then 9(3 *3*3) events can be predefined for the corresponding virtual object.

A plurality of components predefined according to an embodiment may include a plurality of landmark components included in the object and a plurality of angular components for indicating rotation of the object. For example, landmark components may correspond to a landmark of an object such as eyes, nose, mouth, and eyebrows, and angle components may correspond to a plurality of rotation axes such as an X-axis, Y-axis, and Z-axis of the object. .

The electronic device 1 110 described in the embodiment of FIG. 3 may set and manage a plurality of shape values representing two or more different shapes for each of a plurality of landmark components prior to operation 310. As already explained, three shape values such as'close eyes','open eyes' and'open half eyes (or close half eyes)' corresponding to the landmark component corresponding to the'left eye' are pre-set and managed. Can be.

4 is a diagram illustrating an example of a landmark component of a face object according to an embodiment of the present invention. 4 shows landmark components included in the face object 410 as'eye Brow Left (BL)','eye Brow Right BR','Eye Left (EL)', and ' It represents the right eye (Eye Right, ER)' and'mouth (Mouth, MO)', respectively. In addition, Table 1 below shows examples of descriptions of identifiers and values for these landmark components.

Component ID Abbreviation Explanation Explanation Value: 1 Value: 2 Value: 3 BL eyeBrowLeft Left eyebrow shape Lower right Top right - BR eyeBrowRight Right eyebrow shape Lower right Top right - EL EyeLeft Left eye shape Eyes closed Open your eyes Half open eyes ER EyeRight Right eye shape Eyes closed Open your eyes Half open eyes MO Mouth Mouth shape Ah this Five

In Table 1, the values of components are expressed in text format to help understanding, but internally the system may be set and stored in a numerical value corresponding to the shape of the corresponding value or in other forms.

In this case, the electronic device 1 110 corresponds to the shape of the first landmark component determined based on the landmark information among a plurality of shape values managed for the first landmark component in step 330 of FIG. 3. The shape value may be determined as the value of the first landmark component. In the example of Table 1 above, if the first landmark component is'left eye' and the shape of the first landmark component determined based on the landmark information is'open eyes', the value of the first landmark component is' It can be determined as a value indicating left eye opening, such as EL1'.

In addition, the electronic device 1 110 described in the embodiment of FIG. 3 may set and manage a plurality of angle values representing two or more different angles for each of a plurality of rotation axes before operation 310.

5 is a diagram illustrating an example of an angular component of a face object according to an embodiment of the present invention. FIG. 5 shows the appearances 520 and 530 that can emerge when the face object 410 is rotated around the X-axis, the appearances 540 and 550 that can appear when the face object 410 is rotated around the Y-axis, and Z -Examples of the shapes 560 and 570 that can come out by rotating around an axis are shown, respectively. In addition, Table 2 below shows examples of descriptions of identifiers and values for these angular components.

Component ID Abbreviation Explanation Explanation Value: 1 Value: 2 PA PitchAngle Rotate around the X-axis Tilt forward Tilt back YA YawAngle Rotate around the Y-axis Right tilt Left tilt RA RollAngle Rotate around the Z-axis Lower right Lower left

The values of the components shown in Table 2 are also expressed in text format to aid understanding, but internally the system may be set and stored in a numerical value corresponding to the value or in other forms.

In this case, the electronic device 1 110 determines the angle of the object relative to the first rotation axis among the plurality of rotation axes in step 330 of FIG. 3 based on the landmark information, and Among the angle values, an angle value corresponding to the determined angle of the object may be determined as a value of an angle component corresponding to the first rotation axis. For example, if the first rotation axis is the Z-axis and the angle of the object with respect to the Z-axis determined based on the landmark information is'bottom-right', the value of the angle component is the Z-axis center, such as'RA1'. It can be set as a value to indicate the lower-right direction of rotation.

Table 3 below shows examples of events corresponding to combinations of values of components.

Combination meaning Event example EL1 + ER2 Left eye wink Left eye wink ER1 + EL2 Right eye wink Right eye wink EL1 + ER2 + MO1 Open your mouth and wink your left eye Left eye wink + Heart firing from left eye ER1 + EL2 + MO1 Open your mouth and wink your right eye Right eye wink + right eye firing heart BL1 + BR1 twist Anger effect BL1 + BR1 + MO1 Make your mouth'ah' and frown Anger Effect 2 … … …

For example,'EL1 + EL2' in Table 3 may represent a combination of'closed eyes' for'left eye' and'opened eyes' for'right eye' in Table 1, which means left eye wink. can do. In this case, a'left eye wink' event may be applied to a virtual object (for example, a character's face corresponding to a user's face). In addition, not only the user's facial expression, but also various effects such as an animation in which a heart-shaped object is fired depending on whether the user is open or not can be applied to the virtual object.

For example, the electronic device 1 110 described in the embodiment of FIG. 3 sets an event for each combination of values that can be set for a plurality of components before step 310 to manage a plurality of events for the combinations. I can. In this case, the electronic device 1 110 may extract an event corresponding to a combination of values determined from among a plurality of events managed in step 340, and apply the event extracted in step 350 to the virtual object. I can.

In this way, since events are determined according to the combination of the values of components, changes in the object (for example, the user's facial expression or the angle of the face) can be reflected in the image, and at the same time, events can be produced appropriately for the virtual object. Events that naturally match the virtual object can be applied.

6 is a flowchart illustrating an example of a method of applying an event to a virtual object according to an embodiment of the present invention. Steps 610 to 630 of FIG. 6 may be performed while being included in step 350 described with reference to FIG. 3.

In operation 610, the electronic device 1 110 may load a motion file corresponding to the determined event. As an event may be preset for combinations of component values, motion files may be generated and managed in advance in response to the event. These motion files may include at least one motion frame to apply animation for the virtual object. For example, a motion file corresponding to'heart firing' may include motion frames for applying an animation in which a heart is fired to a virtual object.

In operation 620, the electronic device 1 110 may execute a motion file loaded in units of frames. For example, the electronic device 1 110 may apply the first motion frame included in the motion file to the current frame for displaying the virtual object.

In step 630, the electronic device 1 110 may check the end of the motion file. In this case, when the motion file is terminated, application of the event may be terminated, and if the motion file is not terminated, step 620 may be performed again. For example, when a motion file includes three motion frames, three motion frames may be sequentially applied to frames for displaying a virtual object. Applying all of these motion frames may mean the end of the motion file.

In other words, while repeatedly performing steps 620 and 630, the electronic device 1 110 sequentially converts the motion frames of the animation included in the loaded motion file into frames for displaying the virtual object in units of frames. Can be applied as.

In addition, in the embodiment of FIG. 3, after step 310, the electronic device 1 110 may normalize the size and angle of the object using the extracted landmark information.

7 is a diagram illustrating an example of a plurality of motion frames included in a motion file according to an embodiment of the present invention. Since the speed at which a person actually blinks is too fast, as shown in FIG. 7, if only the eyes open state 710 and the eyes closed state 720 are mapped to the avatar as they are, an unnatural motion may be output. Accordingly, the electronic device 1 110 may provide a more natural motion of the virtual object using the plurality of motion frames 730 to 770 illustrated in FIG. 7.

The format of the motion file may be expressed in the form of (identifier of landmark component = change of value separated by','). For example, a motion file in which both eyes are closed in the embodiment of FIG. 7 may be expressed as (EL = 1.0, 0.8, 0.6, 0.2, 0.0, ER = 1.0, 0.8, 0.6, 0.2, 0.0). In this expression, a first motion frame in which the left and right eyes each have a value of 1.0, a second motion frame in which the left and right eyes have a value of 0.8, and the left and right eyes have a value of 0.6. A motion file containing five motion frames, such as a third motion frame, a fourth motion frame with a value of 0.2 for the left and right eyes, and a fifth motion frame with a value of 0.0 for the left and right eyes, respectively. It can mean.

8 is a diagram illustrating an example of normalizing landmark coordinates according to an embodiment of the present invention. The electronic device 1 110 may normalize the object 811 included in the input image frame 810 like the object 831 of the output frame 830 using the landmark coordinate normalization algorithm 820.

For example, when a picture is taken using a camera, if the size, position (relative position in the image) and/or angle of the face are different in the image, the size and position of the eyes are different. In this case, the electronic device 1 110 may adjust the face of the same person to have a similar size, position, and/or angle through the process of normalizing the landmark. For example, the landmark coordinate normalization algorithm 820 can normalize the landmark by adjusting the relative position of the pupil recognized on the face, but it is not limited thereto, and those skilled in the art know that various known normalization algorithms can be used. It will be easy to understand.

This normalization of landmark coordinates enables a relative value to be obtained based on the normalized landmark coordinates even if the size, position, and angle of the object (face) in the input image are different. To this end, the electronic device 1 110 converts the size of the face recognized through the scale factor into a standard size, and rotates the object's landmark by using pitch, roll, and yaw. You can normalize the coordinates. Normalized coordinate values may be two-dimensional coordinates.

In addition, in the embodiment of FIG. 3, the electronic device 1 110 is the landmark of the object recognized through the landmark information extracted through the current frame of the image after step 310 or after the above-described normalization of the landmark coordinates. The coordinates of the mark components may be corrected using landmark information extracted through the previous frame of the image. For example, an image captured and input through a user's mobile phone camera may be determined unevenly with landmark coordinates due to hand shake or the like. Accordingly, the electronic device 1 110 can make the result coordinate evenly by reflecting the weight function using the deviation of the face angle (Pitch, Roll, Yaw) of the previous frame during face recognition. .

9 is a diagram illustrating an example of correcting landmark coordinates according to an embodiment of the present invention. The electronic device 1 110 may detect 920 a landmark and an angle of the object 910. The detection of such a landmark and an angle 920 is to extract the landmark information in step 310 described with reference to FIG. 3, or after extracting the landmark information as in step 310, as described with reference to FIG. Likewise, it can correspond to normalizing the landmark coordinates.

In addition, the electronic device 1 110 may correct 930 landmark coordinates. For example, as described above, the electronic device 1 110 reflects the weight function using the deviation of the face angle (Pitch, Roll, Yaw) of the previous frame during face recognition. The resulting coordinates can be made even. In this case, the weight function may be defined as Equation 1 below as an example.

Here, y _i may be an output of an i -th frame, x _i may be an input of an i -th frame, and α may denote a reliability having a value between 0 and 1, respectively. In addition, α is a ratio to the difference between the face angle (Pitch, Roll, and yaw) of the previous frame, and may be calculated as in Equation 2 below.

Here, yaw _i is the yaw of the face angle in the i _{-th frame, roll i} is the roll of the face angle in the i _{-th frame, and pitch i} is the pitch in the i -th frame. ) Can each correspond. In addition, yaw _max , roll _max , and pitch _max may mean values of the maximum amount of change in pitch, roll, and yaw, respectively.

In addition, the electronic device 1 110 uses the landmark information whose coordinates are corrected to transfer the user's facial expression or event to a virtual object (a character's face corresponding to the user's face) as described through the embodiment of FIG. 3. Mapping 940 can be done.

Meanwhile, the electronic device 1 110 may use a separate method for discriminating between blinking eyes and a wink on the user's face.

10 is a flowchart illustrating an example of a method for discriminating between blinking of both eyes and a wink according to an embodiment of the present invention.

When the application is started 1010, the electronic device 1 110 may buffer information about blinking of an eye (1020). For example, the electronic device 1 110 may measure a relative value representing the degree of opening of two eyes included in the user's face for a predetermined number of consecutive frames of an image and store them in a buffer. Here, an example of the relative value has been described with reference to FIG. 7.

In this case, the electronic device 1 110 may check (1030) whether both eyes are currently open (open) or closed (close) in the process (1030). If both eyes are open or closed, both eye motion motion 1040 may be confirmed, and avatar motion motion 1050 may be determined accordingly.

On the other hand, when both eyes are not open or closed, the electronic device 1 110 may check the buffer. For example, the electronic device 1 110 may check 1070 whether both eyes are the same event (when both eyes are open or both eyes are closed) within a specific time in the buffer. The specific time may correspond to, for example, a preset number of frames. For example, the electronic device 1 110 analyzes the relative values of a preset number of consecutive frames in the buffer, and if the same event of both eyes exists in the corresponding frames, it can check the motion motion 1040 of both eyes. , If the same event of both eyes does not exist in the corresponding frames, one eye motion operation 1080 may be checked. In this case, the electronic device 1 110 may determine the avatar motion operation 1050 according to the one eye motion operation 1080.

11 is a flowchart illustrating an example of the overall service flow according to an embodiment of the present invention. 11 shows an example of an entire process of mapping a user's facial expressions to an avatar.

Process 1 (1111) may be a process in which the electronic device 1 (110) starts the camera to extract initial landmark information. Extracting information on landmarks such as eyes, nose, mouth, and eyebrows from an image including an object such as a user's face may be easily understood by those skilled in the art through well-known techniques as described above.

Step 2 1112 may be a process in which the electronic device 1 110 normalizes the landmark coordinates using the landmark coordinate normalization algorithm, as described with reference to FIG. 8.

Process 3 (1113) may be a process in which the electronic device 1 (110) extracts a value for an expression. For example, the electronic device 1 110 may extract a height of an open eye, a height of an open mouth, a distance between an eyebrow and an eye, an angle of an eyebrow, and the like.

Process 4 (1114) may be a process in which the electronic device 1 (110) generates a standard model based on the extracted value.

Step 5 (1115) may be a process in which the electronic device 1 (110) activates the camera and extracts landmark information during activation.

Step 6 (1116) may be a process in which the electronic device 1 (110) compares a face with a standard model through the extracted landmark information. The standard model can mean the state of the face extracted first when the service is started. For example, steps 1 to 4 may be information according to initial coordinates of landmarks (for example, eyes, nose, mouth, eyebrows, etc.) for the face of a user who intends to use the service. Step 6 may be a process for checking the changed state of the component by comparing the position and angle of the changed landmarks when the facial expression changes through the comparison with the standard model. Since the shape and size of landmarks such as eyes, nose, mouth, and eyebrows are different for each Sarah, it is difficult to characterize the state value with the absolute size of the component. For example, if there is a person A with a left eye of 2 cm and a person B with a left eye of 1 cm with the left eye wide open, a more accurate measurement is possible by calculating the change in the left eye proportionally using the standard model of each person. It becomes possible. In other words, if the size of the left eye of the person A whose left eye is 2 cm wide is changed to 1 cm, it can be determined that the left eye of the person A is half-closed.

Process 7 (1117) may be a process in which the electronic device 1 110 corrects the landmark coordinates based on the previous frame as described with reference to FIG. 9.

Step 8 (1118) may be a process in which the electronic device 1 110 extracts an attribute value of an action unit (AU). The attribute value of the AU may mean a change value calculated through comparison with the standard model described above. In other words, the standard model may refer to a set of initial values, and an attribute value of AU may refer to a set of changed values. For example, if the height of the left eye of the standard model is 2cm, it will be 1cm when the left eye is half closed, and the attribute value of AU for the left eye at this time may be 0.5.

Step 9 (1119) may be a process in which the electronic device 1 110 processes the blink synchronization of both eyes as described with reference to FIG. 10. In other words, the electronic device 1 110 may select one of the both eye motion operation 1040 and the user's one eye motion operation 1080 using the eye blink buffering 1020 or the like in step 9 (1119).

Process 10 (1120) may be a process in which the electronic device 1 (110) maps the user's face and the avatar's face based on landmark information, as described through step 320 of FIG. 3.

Process 11 (1121) may be a process in which the electronic device 1 (110) determines an event, as described through steps 330 and 340 of FIG. 3.

Process 12 (1122) may be a process in which the electronic device 1 110 determines an expression animation using the motion files described with reference to FIGS. 6 and 7.

The results of step 10 (1120), step 11 (1121), and step 12 (1122) may be applied to the avatar in parallel, respectively.

Process 13 (1123) may be a process in which the electronic device 1 (110) performs post-processing. Post-processing is to record a video of the avatar's screen to which the user's facial expressions or various effects are applied, edit the recorded video, save the edited video, convert the edited video to a moving GIF file, edited video or GIF file. May include processes such as sharing (for example, sharing through a messaging service or a social network service, etc.).

As described above, according to embodiments of the present invention, components (for example, eyes, nose, mouth, eyebrows, face angles, etc.) that are predefined for an object (for example, a user's face) included in an input image ), and by selecting one of the events preset for the virtual object according to the combination of the determined values and applying it to the virtual object, changes in the object (for example, changes in facial expressions) are applied to the virtual object. While reflecting, more natural changes can be applied from the perspective of the virtual object.

The system or device described above may be implemented as a hardware component, a software component, or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. It can be permanently or temporarily embody. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (17)

In a computer program stored in a computer-readable recording medium to execute an image processing method in combination with a computer,
The image processing method,
Extracting landmark information for an object included in the input image;
Mapping an object included in the input image and a virtual object based on the landmark information;
Determining a value of each of a plurality of components predefined for the object based on the landmark information;
Determining an event corresponding to the determined combination of values of the plurality of components; And
Applying the determined event to a virtual object mapped to the object
Including,
A plurality of preset events for each virtual object and a plurality of motion files corresponding to the plurality of events are generated in advance,
Applying the determined event to the virtual object mapped to the object,
Loading a motion file generated in advance in response to the determined event from among the plurality of previously generated motion files; And
Sequentially applying motion frames of animation included in the loaded motion file to frames for displaying the virtual object in frame units
Computer program comprising a. The method of claim 1,
The plurality of predefined components includes a plurality of landmark components included in the object and a plurality of angular components for indicating rotation of the object. The method of claim 2,
The image processing method,
Setting and storing a plurality of shape values representing two or more different shapes for each of the plurality of landmark components
Including more,
The step of determining a value of each of a plurality of components predefined for the object,
Determining a shape value corresponding to the shape of the first landmark component determined based on the landmark information among a plurality of shape values stored for the first landmark component as the value of the first landmark component A computer program characterized by. The method of claim 2,
The plurality of angular components correspond to a plurality of rotation axes settable for the object,
The image processing method,
Setting and storing a plurality of angle values representing two or more different angles for each of the plurality of rotation axes
Including more,
The step of determining a value of each of a plurality of components predefined for the object,
Determining an angle of the object with respect to a first rotation axis among the plurality of rotation axes based on the landmark information; And
Determining an angle value corresponding to the determined angle of the object among a plurality of angle values stored with respect to the first rotation axis as a value of an angle component corresponding to the first rotation axis
Computer program comprising a. The method of claim 1,
The image processing method,
Setting an event for each combination of values that can be set for the plurality of components and storing a plurality of events for the combinations
Including more,
The step of determining the event,
And extracting an event corresponding to a combination of the determined values from among the plurality of stored events. The method of claim 1,
The image processing method,
Normalizing the size, position, and angle of the object using the extracted landmark information
Computer program further comprising a. The method of claim 1,
The image processing method,
Correcting the coordinates of the landmark components of the object recognized through the landmark information extracted through the current frame of the image using the landmark information extracted through the previous frame of the image
Computer program further comprising a. The method of claim 7,
The step of correcting,
Coordinates of the landmark components of the object through the current frame using a reliability value calculated based on the angle of rotation of the object recognized in the current frame and the amount of change in the angle of rotation of the object recognized in the previous frame Computer program, characterized in that to calibrate. The method of claim 1,
The object includes a user's face included in the image input through a camera,
And the virtual object includes a face of a character to be mapped to the face of the user in real time. Extracting landmark information for an object included in the input image;
Mapping an object included in the input image and a virtual object based on the landmark information;
Determining a value of each of a plurality of components predefined for the object based on the landmark information;
Determining an event corresponding to the determined combination of values of the plurality of components; And
Applying the determined event to a virtual object mapped to the object
Including,
A plurality of preset events for each virtual object and a plurality of motion files corresponding to the plurality of events are generated in advance,
Applying the determined event to the virtual object mapped to the object,
Loading a pre-generated motion file corresponding to the determined event from among the plurality of pre-generated motion files; And
Sequentially applying motion frames of animation included in the loaded motion file to frames for displaying the virtual object in frame units
Image processing method comprising a. The method of claim 10,
The plurality of predefined components include a plurality of landmark components included in the object and a plurality of angular components for indicating rotation of the object. The method of claim 11,
Setting and storing a plurality of shape values representing two or more different shapes for each of the plurality of landmark components
Including more,
The step of determining a value of each of a plurality of components predefined for the object,
Determining a shape value corresponding to the shape of the first landmark component determined based on the landmark information among a plurality of shape values stored for the first landmark component as the value of the first landmark component Image processing method characterized by. The method of claim 11,
The plurality of angular components correspond to a plurality of rotation axes settable for the object,
Setting and storing a plurality of angle values representing two or more different angles for each of the plurality of rotation axes
Including more,
The step of determining a value of each of a plurality of components predefined for the object,
Determining an angle of the object with respect to a first rotation axis among the plurality of rotation axes based on the landmark information; And
Determining an angle value corresponding to the determined angle of the object among a plurality of angle values stored with respect to the first rotation axis as a value of an angle component corresponding to the first rotation axis
Image processing method comprising a. The method of claim 10,
Setting an event for each combination of values that can be set for the plurality of components and storing a plurality of events for the combinations
Including more,
The step of determining the event,
And extracting an event corresponding to a combination of the determined values from among the plurality of stored events. The method of claim 10,
Correcting the coordinates of the landmark components of the object recognized through the landmark information extracted through the current frame of the image using the landmark information extracted through the previous frame of the image
The image processing method further comprising a. The method of claim 15,
The step of correcting,
Coordinates of the landmark components of the object through the current frame using a reliability value calculated based on the angle of rotation of the object recognized in the current frame and the amount of change in the angle of rotation of the object recognized in the previous frame The image processing method, characterized in that to correct the. A computer-readable recording medium having a program recorded thereon for causing a computer to execute the method of any one of claims 10 to 16.

Images