KR102607789B1 - Methord for processing image and electronic device thereof - Google Patents

Abstract

An electronic device according to various embodiments includes at least one camera, a processor operatively connected to the at least one camera, and a memory operatively connected to the processor, wherein the memory is connected to the processor when executed. acquire an image including a visual object corresponding to a face using the at least one camera, identify a plurality of landmarks corresponding to a facial component from the visual object, and In response to identifying that a ratio of a distance of a first set of landmarks to a distance of a second set of landmarks of the plurality of landmarks is included in a first range, corrected based on the image Instructions for acquiring the first image may be stored. Other embodiments are also possible.

Description

Image processing method and electronic device thereof {METHORD FOR PROCESSING IMAGE AND ELECTRONIC DEVICE THEREOF}

Various embodiments of the present invention relate to an image processing method and electronic devices thereof.

As information and communication technology and semiconductor technology develop, the distribution and use of various electronic devices is gradually increasing. Electronic devices are reaching a stage of convergence that extends beyond their traditional domain and encompasses the domains of other electronic devices. For example, an electronic device may include a camera and provide images to a user through the camera.

The specifications of cameras included in electronic devices are gradually increasing, and technology for processing images captured through cameras is also becoming more advanced. For example, an electronic device may support image processing technology to correct distortion of visual objects (e.g., faces) included in images captured through a camera.

Conventional electronic devices provide image processing technology that corrects image distortion without considering various elements of visual objects included in the image, and accordingly, there are limitations in correcting image distortion. Therefore, a solution to more accurately correct image distortion may be required.

Various embodiments of the present invention can provide a method and electronic device for more accurately correcting image distortion by correcting image distortion by considering various elements of visual objects included in images acquired through a camera.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An electronic device according to various embodiments of the present invention includes at least one camera, a processor operatively connected to the at least one camera, and a memory operatively connected to the processor, wherein the memory is configured to operate when executed. , the processor acquires an image including a visual object corresponding to a face using the at least one camera, and identifies a plurality of landmarks corresponding to a facial component from the visual object, and In response to identifying that a ratio of a distance of a first set of landmarks of the plurality of landmarks and a distance of a second set of landmarks of the plurality of landmarks is included in a first range, based on the image Instructions for obtaining the corrected first image can be stored.

An image processing method in an electronic device according to various embodiments of the present invention includes an operation of acquiring an image including a visual object corresponding to a face, and a plurality of landmarks corresponding to a facial component from the visual object. an operation of identifying the plurality of landmarks, and in response to identifying that a ratio of a distance of a first set of landmarks among the plurality of landmarks and a distance of a second set of landmarks of the plurality of landmarks is included in a first range. Thus, an operation of acquiring a first image corrected based on the image may be included.

Electronic devices and methods thereof according to various embodiments can correct image distortion more accurately by correcting image distortion by considering various elements of visual objects included in images acquired through a camera.

Electronic devices and methods according to various embodiments can provide natural images to users by correcting image distortion.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

FIG. 1 is a diagram for explaining the cause of image distortion in an electronic device according to various embodiments.
Figure 2 is an example diagram for explaining a distorted image according to various embodiments.
Figure 3 is a block diagram of an electronic device in a network environment according to various embodiments.
4 is a block diagram of an electronic device according to various embodiments.
FIG. 5 is a flowchart illustrating a method for correcting a distorted image in an electronic device according to various embodiments.
FIG. 6 is an example diagram illustrating a method of correcting a distorted image in an electronic device according to various embodiments.
FIGS. 7A and 7B are flowcharts illustrating a method of correcting a distorted image based on a distance ratio between landmarks corresponding to facial elements in an electronic device according to various embodiments.
FIG. 8 is an example diagram illustrating a method of correcting a distorted image based on a distance ratio between landmarks corresponding to facial elements in an electronic device according to various embodiments.
FIG. 9 is a diagram illustrating a method of correcting a distorted image based on a position and length associated with a face area in an electronic device according to various embodiments.
FIG. 10 is an example diagram illustrating a method of correcting a distorted image based on a position and length associated with a face area in an electronic device according to various embodiments.
FIG. 11 is a diagram illustrating an algorithm for calculating image warping coordinates according to various embodiments.

FIG. 1 is a diagram for explaining the cause of image distortion in an electronic device according to various embodiments.

Hereinafter, with reference to FIG. 1, at least one of the relative position of the object (e.g., face) with respect to the camera of the electronic device, the distance of the object (e.g., focal length or depth), or the relative orientation of the object with respect to the camera. Based on this, explain the cause of image distortion.

According to various embodiments, the degree to which an image is distorted may vary depending on the relative position of the object with respect to the camera (or the relative position of the subject within the camera's field of view). For example, in the case of the object 110, the length of the path of the lights 113 and 115 passing through the lens 130 from the outline (or outline) of the object 110 (or the image from the outline of the object 110) The length of the path to the forming surface 140 may be the same. According to one embodiment, when the length of the path of light passing from the outline (or outline) of the object 110 through the lens 130 is the same, the object image 111 may not be distorted or the degree of distortion may be small. According to one embodiment, the object image 111 may be located at the center of the entire image acquired using a camera. According to one embodiment, in the case of the object 120, the length of the path of the lights 123 and 125 passing through the lens 130 from the outline (or outline) of the object 120 (or from the outline of the object 120) The length of the path to the image forming surface 140 may vary. If the path lengths of the lights 123 and 125 passing through the outline (or outline) of the object 120 are different, the object image 121 may be distorted. For example, if the object 110 and the object 120 are the same, the distance (x) between the object 110 and the surface 140 that forms the image and the distance (x) between the object 120 and the surface 140 that forms the image are Even if the distance (x') is the same, the length (d2) of the object image 121 may be different from the length (d1) of the object image 111. According to one embodiment, in the case of the object image 121, distortion may occur in which the length of the object image increases with respect to one axis of the object image and decreases the length with respect to the other axis. According to one embodiment, as the distance (y) between the positions of the object image and the center of the image acquired through the camera increases, the degree of distortion may increase.

According to various embodiments, the degree to which an object image is distorted may vary depending on the distance between the camera and the object. For example, as the distance between the camera and the object increases, the degree of distortion of the object image may increase. For example, comparing the object 120 and the object 141, the angle forming the light passing through the lens 130 from the outline of the object 141 is the angle from the outline of the object 120 passing through the lens 130. Because the angle is greater than the angle forming the light, greater distortion may occur in the object image 121 than in the object image for the object 141.

According to various embodiments, the degree to which an object image is distorted may vary depending on the relative direction (or angle) of the object with respect to the camera. For example, when the relative orientation of an object with respect to the camera changes, a plurality of landmarks (e.g., eyebrows, eyes, nose, or mouth) corresponding to facial feature components contained within the object are displayed from the camera. The distance between cameras may vary. For example, when the direction of the object 110 with respect to the camera changes (e.g., when the angle of the object 110 is rotated by θ), the lens 130 is moved from a point 155 of the outline of the object 110. The length of the light path passing through may be different from the length of the light path passing through the lens 130 from another point 151. The part of the object image centered on a point where the light path length decreases may be enlarged, and the part centered on the point where the light path length increases may be reduced.

Figure 2 is an example diagram for explaining a distorted image according to various embodiments.

According to various embodiments, the images 201 to 209 may be images taken of the same subject by changing the state between the subject and the camera.

According to various embodiments, the image 201 is in a state 201-1 (e.g., the axis 201-3 of the subject 201-2 and the axis 201-5 of the camera 201-4 are parallel and , the subject 201-2 and the camera 201-4 are arranged so that the face area 201-6 of the subject 201-2 is located in the center of the image 201. there is. According to one embodiment, as the image 201 is captured in state 201-1, the facial region 201-6 including at least a portion of the visual object included in the image 201 is It can be located in the center part. Accordingly, no distortion may occur in the face area 201-6 or the degree of distortion may be small. According to one embodiment, as in the image 201, when no distortion occurs in the face area 201-6, the first set of landmarks (e.g., eyebrows) included in the face area 201-6 and nose)) and the distance 201-8 of the second set of landmarks (e.g., eyes and mouth) are included in the first range (e.g., less than 0.95 or greater than 1.15). It may not work. According to one embodiment, when no distortion occurs for the face region 201-6, as in image 201, the distance 201-9 of the third set of landmarks (e.g., eyebrows and eyes) and the distance 201-10 of the fourth set of landmarks (e.g., eyes and nose) may not be included in the second range (e.g., less than 1). According to one embodiment, as in the image 201, when no distortion occurs in the face area 201-6, when no distortion occurs in the face area 201-6, the third set of lands The ratio of the distance of the marks 201-9 and the distance of the fourth set of landmarks 201-10 is the distance of the third set of landmarks 201-9 and the distance of the fifth set of landmarks (e.g. and mouth) may be smaller than the ratio of the distance (201-11).

According to various embodiments, the image 203 is in a state 203-1 (e.g., the axis 203-3 of the subject 203-2 and the axis 203-5 of the camera 203-4 are parallel and , the subject 203-2 and the camera 203-4 are positioned so that the face area 203-6 of the subject 203-2 is located at the bottom of the image 203. there is. According to one embodiment, as the image 203 is captured in state 203-1, the facial area 203-6 of the subject 203-2 included in the image 203 is located at the bottom of the image 203. It can be located in the part. Accordingly, distortion may occur in the face area 203-6. According to one embodiment, as in the image 203, when distortion occurs in the face area 203-6, the first set of landmarks (e.g., eyebrows and nose) included in the face area 203-6 ))'s distance 203-7 and the distance 203-8 of the second set of landmarks (e.g., eyes and mouth) may be included in the first range (e.g., less than 0.95 or greater than 1.15) .

According to various embodiments, the image 205 is in a state 205-1 (e.g., the axis 205-3 of the subject 205-2 and the axis 205-5 of the camera 205-4 are parallel). , the subject 205-2 and the camera 205-4 are positioned so that the face area 205-6 of the subject 205-2 is located at the top of the image 203. there is. According to one embodiment, as the image 205 is captured in state 205-1, the facial area 205-6 of the subject 205-2 included in the image 205 is positioned at the top of the image 205. It can be located in the part. Accordingly, distortion may occur in the face area 205-6. According to one embodiment, when no distortion occurs in the face area 205-6, the distance of the first set of landmarks (e.g., eyebrows and nose) included in the face area 205-6 ( The ratio of the distance 205-7) and the distance 205-8 of the second set of landmarks (e.g. eyes and mouth) may be included in the first range (e.g. less than 0.95 or greater than 1.15).

According to various embodiments, the image 207 is in a state 207-1 (e.g., the axis 207-3 of the subject 207-2 and the axis 207-5 of the camera 207-4 are not parallel). It is an image taken in a state in which the subject 207-2 and the camera 207-4 are arranged so that the face area 207-6 of the subject 207-2 is located at the upper part of the image 207. You can. According to one embodiment, as the image 207 is captured in state 207-1, the facial area 207-6 of the subject 207-2 included in the image 207 is positioned at the top of the image 207. It can be located in the part. Accordingly, distortion may occur in the face area 207-6. According to one embodiment, as in the image 207, when distortion occurs in the face area 207-6, the first set of landmarks (e.g., eyebrows and nose) included in the face area 207-6 ))'s distance 207-7 and the distance 207-8 of the second set of landmarks (e.g., eyes and mouth) will not fall within the first range (e.g., less than 0.95 or greater than 1.15). You can. According to one embodiment, when distortion occurs in the face area 207-6, as in the image 207, the distance 207-9 of the third set of landmarks (e.g., eyebrows and eyes) and the The ratio of distances 207 - 10 of the four sets of landmarks (e.g. eyes and nose) may not be included in the second range (e.g. less than 1). According to one embodiment, when distortion occurs in the face area 207-6, as in the image 207, the distance of the third set of landmarks 207-9 and the distance of the fourth set of landmarks (207-9) The ratio 207-10) may be greater than the ratio between the distance 207-9 of the third set of landmarks and the distance 207-11 of the fifth set of landmarks (eg, nose and mouth).

According to various embodiments, the image 209 is in a state 209-1 (e.g., the axis 209-3 of the subject 209-2 and the axis 209-5 of the camera 209-4 are not parallel). It is an image taken in a state in which the subject 209-2 and the camera 209-4 are arranged so that the face area 207-6 of the subject 207-2 is located at the bottom of the image 209. You can. According to one embodiment, as the image 209 is captured in state 209-1, the facial area 209-6 of the subject 209-2 included in the image 209 is located at the bottom of the image 209. It can be located in the part. Accordingly, distortion may occur in the face area 209-6. According to one embodiment, as in the image 209, when distortion occurs in the face area 209-6, the first set of landmarks (e.g., eyebrows and nose) included in the face area 209-6 ))'s distance 209-7 and the distance 209-8 of the second set of landmarks (e.g., eyes and mouth) will not fall within the first range (e.g., less than 0.95 or greater than 1.15). You can. According to one embodiment, when distortion occurs in the face area 209-6, as in the image 209, the distance 209-9 of the third set of landmarks (e.g., eyebrows and eyes) and the The ratio of distances 209-10 of the four sets of landmarks (e.g., eyes and nose) may be included in a second range (e.g., less than 1).

According to various embodiments, the distance between landmarks may include the distance (or difference) between the y-axis coordinates of the landmarks. For example, the distance of the first set of landmarks may include the distance (or difference) between the y-axis coordinate of the eyebrow identified in the image and the y-axis coordinate of the nose identified in the image. As another example, the distance of the second set of landmarks may include the distance (or difference) of the y-axis coordinate of the eye identified in the image and the y-axis coordinate of the mouth identified in the image. As another example, the distance of the third set of landmarks may include the distance (or difference) between the y-axis coordinate of the eyebrow identified in the image and the y-axis coordinate of the eye identified in the image. As another example, the distance of the fourth set of landmarks may include the distance (or difference) between the y-axis coordinate of the eye identified in the image and the y-axis coordinate of the nose identified in the image. As another example, the distance of the fifth set of landmarks may include the distance (or difference) between the y-axis coordinate of the nose identified in the image and the y-axis coordinate of the mouth identified in the image.

Figure 3 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 3 is a block diagram of an electronic device 301 in a network environment 300, according to various embodiments. Referring to FIG. 3, in the network environment 300, the electronic device 301 communicates with the electronic device 302 through a first network 398 (e.g., a short-range wireless communication network) or a second network 399. It is possible to communicate with the electronic device 304 or the server 308 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 301 may communicate with the electronic device 304 through the server 308. According to one embodiment, the electronic device 301 includes a processor 320, a memory 330, an input device 350, an audio output device 355, a display device 360, an audio module 370, and a sensor module ( 376), interface 377, haptic module 379, camera module 380, power management module 388, battery 389, communication module 390, subscriber identification module 396, or antenna module 397 ) may include. In some embodiments, at least one of these components (eg, the display device 360 or the camera module 380) may be omitted, or one or more other components may be added to the electronic device 301. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 376 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 360 (e.g., a display).

The processor 320, for example, executes software (e.g., program 340) to operate at least one other component (e.g., hardware or software component) of the electronic device 301 connected to the processor 320. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 320 stores commands or data received from another component (e.g., sensor module 376 or communication module 390) in volatile memory 332. The commands or data stored in the volatile memory 332 can be processed, and the resulting data can be stored in the non-volatile memory 334. According to one embodiment, the processor 320 includes a main processor 321 (e.g., a central processing unit or an application processor), and an auxiliary processor 323 (e.g., a graphics processing unit, an image signal processor) that can operate independently or together with the main processor 321. , sensor hub processor, or communication processor). Additionally or alternatively, the auxiliary processor 323 may be set to use less power than the main processor 321 or to specialize in a designated function. The auxiliary processor 323 may be implemented separately from the main processor 321 or as part of it.

The auxiliary processor 323 may, for example, act on behalf of the main processor 321 while the main processor 321 is in an inactive (e.g., sleep) state, or while the main processor 321 is in an active (e.g., application execution) state. ), along with the main processor 321, at least one of the components of the electronic device 301 (e.g., the display device 360, the sensor module 376, or the communication module 390) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 323 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 380 or communication module 390). there is.

The memory 330 may store various data used by at least one component (eg, the processor 320 or the sensor module 376) of the electronic device 301. Data may include, for example, input data or output data for software (e.g., program 340) and instructions related thereto. Memory 330 may include volatile memory 332 or non-volatile memory 334.

The program 340 may be stored as software in the memory 330 and may include, for example, an operating system 342, middleware 344, or application 346.

The input device 350 may receive commands or data to be used in a component of the electronic device 301 (e.g., the processor 320) from outside the electronic device 301 (e.g., a user). The input device 350 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, stylus pen).

The sound output device 355 may output sound signals to the outside of the electronic device 301. The sound output device 355 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 360 can visually provide information to the outside of the electronic device 301 (eg, a user). The display device 360 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display device 360 may include touch circuitry configured to detect a touch, or a sensor circuit configured to measure the intensity of force generated by the touch (e.g., a pressure sensor). there is.

The audio module 370 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 370 acquires sound through the input device 350, the audio output device 355, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 301). Sound may be output through an electronic device 302 (e.g., speaker or headphone).

The sensor module 376 detects the operating state (e.g., power or temperature) of the electronic device 301 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 376 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 377 may support one or more designated protocols that can be used to connect the electronic device 301 directly or wirelessly with an external electronic device (e.g., the electronic device 302). According to one embodiment, the interface 377 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 378 may include a connector through which the electronic device 301 can be physically connected to an external electronic device (eg, the electronic device 302). According to one embodiment, the connection terminal 378 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 379 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 379 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 380 can capture still images and moving images. According to one embodiment, the camera module 380 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 388 can manage power supplied to the electronic device 301. According to one embodiment, the power management module 388 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 389 may supply power to at least one component of the electronic device 301. According to one embodiment, the battery 389 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module 390 provides a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 301 and an external electronic device (e.g., the electronic device 302, the electronic device 304, or the server 308). It can support establishment and communication through established communication channels. Communication module 390 operates independently of processor 320 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 390 is a wireless communication module 392 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 394 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 398 (e.g., a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 399 (e.g., a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (e.g., a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 392 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 396 to communicate within a communication network such as the first network 398 or the second network 399. The electronic device 301 can be confirmed and authenticated.

The antenna module 397 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module may include one antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 397 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 398 or the second network 399 is, for example, connected to the plurality of antennas by the communication module 390. can be selected. Signals or power may be transmitted or received between the communication module 390 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) in addition to the radiator may be additionally formed as part of the antenna module 397.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 301 and the external electronic device 304 through the server 308 connected to the second network 399. Each of the electronic devices 302 and 304 may be the same or different type of device from the electronic device 301. According to one embodiment, all or part of the operations performed in the electronic device 301 may be executed in one or more of the external electronic devices 302, 304, or 308. For example, when the electronic device 301 must perform a function or service automatically or in response to a request from a user or another device, the electronic device 301 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 301. The electronic device 301 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technologies may be used.

4 is a block diagram of an electronic device according to various embodiments.

Referring to FIG. 4, the electronic device 400 (e.g., the electronic device 301 of FIG. 3) includes a processor 401 (e.g., the processor 320 of FIG. 3) and a memory 403 (e.g., the electronic device 301 of FIG. 3). It may include at least one of a memory 330), a camera 405 (e.g., the camera module 280 of FIG. 3), or a display 407 (e.g., the display device 360 of FIG. 3). However, it is not limited to this. For example, the electronic device 301 may further include a communication circuit (e.g., the communication module 390 of FIG. 3) and/or at least one sensor (e.g., the sensor module 376 of FIG. 3).

The processor 401 may control the overall operation of the electronic device 400. The processor 401 operates in conjunction with other components of the electronic device 400, such as the memory 403, the camera 405, or the display 407, to control the overall operation of the electronic device 400. can be connected

The processor 401 may receive instructions from other components, interpret the received instructions, and perform calculations or process data according to the interpreted instructions. For example, the processor 401 may request a command, data, or signal from the memory 403. The processor 401 may record (or store) or update instructions, data, or signals in the memory 403 to control the electronic device 400 or other components within the electronic device 400.

The processor 401 may interpret and process messages, data, commands, or signals received from the memory 403, camera 405, or display 407. Processor 401 may generate new messages, data, instructions, or signals based on received messages, data, instructions, or signals. The processor 401 may provide processed or generated messages, data, instructions, or signals to the memory 403, camera 405, or display 407.

The memory 403 may store commands for controlling the electronic device 400, control command codes, control information, or user data. For example, memory 403 may store one or more of an application, operating system, middleware, or device driver.

Camera 405 may include one or more lenses, image sensors, image signal processors, or flashes to acquire a still image or dynamic image. According to one embodiment, the camera is disposed to be exposed through at least a portion of the first side (e.g., front) of the housing of the electronic device 400, or is positioned to be exposed through a second side facing the first side of the housing of the electronic device 400. It may be arranged to be exposed through at least a portion of the surface (e.g., the back).

The display 407 can visually provide information to the outside of the electronic device 400. According to one embodiment, the display 407 may include a touch circuitry configured to detect a touch or a sensor circuit configured to measure the intensity of force generated by a touch (eg, a pressure sensor).

According to various embodiments, the processor 401 may acquire a plurality of landmarks corresponding to facial components from an image acquired through the camera 405. For example, the processor 401 performs a face detection function from an image acquired through the camera 405, and when one face area is detected in the image, a face component ( A plurality of landmarks corresponding to the facial component can be obtained. A plurality of landmarks corresponding to facial components may include at least one of eyebrows, eyes, nose, or mouth.

According to various embodiments, the processor 401 may identify whether distortion has occurred in the image based on distance information between a plurality of landmarks corresponding to facial components obtained from the image. Processor 401 determines the distance of a first set of landmarks (e.g., eyebrows and nose), the distance of a second set of landmarks (e.g., eyes and mouth), and the distance of a third set of landmarks (e.g., eyebrows). and eyes), a distance of a fourth set of landmarks (e.g., eyes and nose), or a distance of a fifth set of landmarks (e.g., nose and mouth), and identify Based on the distance, it can be determined whether distortion has occurred in the image. For example, the processor 401 may display the image of FIG. 2 (e.g., less than 0.95 or greater than 1.15) if the ratio of the distances of the first set of landmarks to the second set of landmarks falls within a first range (e.g., less than 0.95 or greater than 1.15). As shown in (203 or 205), it may be determined that distortion has occurred in the face area (203-6 or 205-6) of the image. For another example, processor 401 may determine that the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is not included in the first range, but the length of the third set of landmarks and the fourth set of landmarks are not included in the first range. The ratio of the distances of the landmarks of is included in a second range (e.g., less than or equal to 1), or the ratio of the distances of the landmarks of the third set to the length of the landmarks of the fourth set is within the range of the length of the landmarks of the third set and the second range (e.g., less than or equal to 1). If it is greater than the ratio of the distances of the five sets of landmarks, it can be determined that distortion has occurred in the face area (207-6 or 209-6) of the image, as in the image (207 or 209) of FIG. 2. As another example, the processor 401 determines that the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is included in the first range, and the length of the third set of landmarks and the fourth set of landmarks are included in the first range. The ratio of the distances of the landmarks is not included in the second range (e.g., less than or equal to 1), and the ratio of the distances of the landmarks of the third set to the length of the landmarks of the fourth set is greater than or equal to the length of the landmarks of the third set. If it is smaller than the ratio of the distances of the five sets of landmarks, it can be determined that no distortion has occurred in the face area 201-6 of the image, as in the image 201 of FIG. 2.

According to various embodiments, the processor 401 may correct the image when distortion occurs in the face area of the image. For example, processor 401 may determine a location associated with a facial region if the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks falls within a first range (e.g., less than 0.95 and greater than 1.15), and The image may be corrected based on the first correction method using length. For another example, processor 401 may determine that the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is not included in the first range, but the length of the third set of landmarks and the fourth set of landmarks are not included in the first range. The ratio of the distances of the landmarks of is included in a second range (e.g., less than or equal to 1), or the ratio of the distances of the landmarks of the third set to the length of the landmarks of the fourth set is within the range of the length of the landmarks of the third set and the second range (e.g., less than or equal to 1). If it is greater than the ratio of the distances of the five sets of landmarks, the image can be corrected based on a second correction method using the position and length associated with the facial area. According to one embodiment, the processor 401 may store the corrected image in the memory 403. According to one embodiment, the processor 401 may output the corrected image as a preview image through the display 407.

According to various embodiments, the processor 401 may determine the distance between landmarks based on the y-axis coordinates of the landmarks. For example, the processor 401 may determine the distance of the first set of landmarks by identifying the distance (or difference) between the y-axis coordinate of the eyebrow identified in the image and the y-axis coordinate of the nose identified in the image. As another example, processor 401 may determine the distances of the second set of landmarks by identifying the distance (or difference) between the y-axis coordinates of the eyes identified in the image and the y-axis coordinates of the mouth identified in the image. there is. As another example, processor 401 may determine the distances of the third set of landmarks by identifying the distance (or difference) between the y-axis coordinates of the eyebrows identified in the image and the y-axis coordinates of the eyes identified in the image. You can. As another example, processor 401 may determine the distance of the fourth set of landmarks by identifying the distance (or difference) between the y-axis coordinate of the eye identified in the image and the y-axis coordinate of the nose identified in the image. You can. As another example, processor 401 may determine the distance of the fifth set of landmarks by identifying the distance (or difference) between the y-axis coordinate of the nose identified in the image and the y-axis coordinate of the mouth identified in the image. You can.

According to various embodiments, the first range may refer to data for determining whether the image is distorted according to the relative position of the subject with respect to the camera (or the relative position of the subject within the camera's angle of view). For example, the first range can be used to determine how far the subject has moved in the vertical direction from the center of the camera's field of view. For example, when the distance of the first set of landmarks and the distance of the second set of landmarks are included in the first range, the processor 401 determines that the subject is at least a certain distance in the vertical direction from the center of the camera's field of view. By determining that it is located in the moved position, it can be determined that image distortion has occurred.

According to various embodiments, the second range may refer to data for determining whether the image is distorted according to the relative direction (or angle) of the subject with respect to the camera. For example, the second range can be used to determine how much the subject's orientation relative to the camera is rotated. For example, the processor 401 determines that the image was captured in state 209-1 when the ratio of the length of the third set of landmarks and the distance of the fourth set of landmarks is included in the second range. Therefore, it can be determined that distortion has occurred in the image.

According to various embodiments, the ratio of the length of the third set of landmarks to the distance of the fourth set of landmarks and the ratio of the length of the third set of landmarks to the distance of the fifth set of landmarks determine the subject relative to the camera. It can be used to determine whether the image is distorted according to the relative direction (or angle) of . For example, the processor 401 determines that the ratio of the length of the third set of landmarks to the distance of the fourth set of landmarks is greater than the ratio of the length of the third set of landmarks to the distance of the fifth set of landmarks. In this case, it may be determined that the image was captured in state 207-1 and that distortion has occurred in the image.

According to various embodiments, an electronic device (e.g., the electronic device 301 of FIG. 3 or the electronic device 401 of FIG. 4) includes at least one camera (e.g., the camera module 380 of FIG. 3 or the camera of FIG. 4). (405)), a processor operatively connected to the at least one camera (e.g., processor 320 of FIG. 3 or processor 401 of FIG. 4), and a memory operably connected to the processor (e.g., FIG. 3 memory 330, or memory 403 of FIG. 4), which, when executed, causes the processor to generate an image including a visual object corresponding to a face using the at least one camera. Obtaining, identifying a plurality of landmarks corresponding to a facial component from the visual object, and determining a distance of a first set of landmarks of the plurality of landmarks and a first set of landmarks of the plurality of landmarks. In response to identifying that the ratio of distances of the two sets of landmarks is included in the first range, instructions may be stored to obtain a first image corrected based on the image.

According to various embodiments, the first set of landmarks includes a first landmark and a second landmark, the second set of landmarks includes a third landmark and a fourth landmark, The instructions are, in response to the processor identifying that the ratio of the distances of the first set of landmarks to the distances of the second set of landmarks is not within the first range, the first landmark and determine whether the ratio of the distance of the third landmark and the distance between the second landmark and the third landmark is included in a second range, and between the first landmark and the third landmark When the distance and the ratio of the distances between the second landmark and the third landmark are within the second range, a second image corrected based on the image may be obtained.

According to various embodiments, the instructions are such that the processor determines that the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark are included in the second range. If not, the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark is the distance between the first landmark and the third landmark and the third landmark. 2 Determine whether the distance between the landmark and the fourth landmark is greater than the ratio, and the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark If this is greater than the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the fourth landmark, the second image corrected based on the image can be obtained. there is.

According to various embodiments, the instructions are such that the processor determines that the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark is the ratio of the distance between the first landmark and the third landmark. If the ratio of the distance between the third landmark and the distance between the second landmark and the fourth landmark is not greater than the ratio, the image may be maintained.

According to various embodiments, the instructions may cause the processor to determine a weight for image correction based on location information associated with a facial region that includes at least a portion of the visual object, and to determine a weight for image correction based on the weight and a length associated with the facial region. Determine a first correction coefficient for image correction based on the information, perform image warping based on the first correction coefficient, and obtain the first image based on at least a portion of the image on which the image warping was performed. You can do it.

According to various embodiments, the instructions are such that the processor determines the weight for image correction based on location information associated with the face region, and performs image correction based on the weight and length information associated with the face region. A second correction coefficient may be determined, image warping may be performed based on the second correction coefficient, and the second image may be acquired based on at least a portion of the image on which the image warping was performed.

According to various embodiments, location information associated with the face region may include coordinates of the center of the face region, and length information associated with the face region may include the length of one side of the face region.

According to various embodiments, the electronic device further includes a display (e.g., the display device 360 of FIG. 3 or the display 407 of FIG. 4) operatively connected to the processor, and the instructions are provided by the processor. , the image, the first image, or the second image may be displayed as a preview image through the display, and the image, the first image, or the second image may be stored in the memory.

According to various embodiments, the electronic device further includes a display operatively connected to the processor, and the instructions include: the first image, while the processor displays the image as a preview image through the display; Alternatively, the second image may be acquired, and the acquired image may be stored in the memory.

According to various embodiments, the instructions are such that, when a plurality of landmarks corresponding to facial components are not identified from the visual object, the processor identifies properties of the visual object and performs processing based on the properties of the visual object. to generate a distortion map, calculate a correction parameter for correcting the distortion map, and correct the image based on the correction parameter, and the properties of the visual object include the location of the visual object. Or it may include at least one of the sizes.

FIG. 5 is a flowchart illustrating a method for correcting a distorted image in an electronic device according to various embodiments. FIG. 6 is an example diagram illustrating a method of correcting a distorted image in an electronic device according to various embodiments.

Referring to FIGS. 5 and 6 , in operation 501, the processor (e.g., the processor 320 of FIG. 3 or the electronic device 400 of FIG. 3 ) of the electronic device (e.g., the electronic device 301 of FIG. 3 or the electronic device 400 of FIG. 4 The processor 401 of 4) may acquire an image including a visual object corresponding to a face through a camera (eg, the camera module 380 of FIG. 3 or the camera 405 of FIG. 4). For example, the processor may respond to identifying that input for image acquisition (e.g., input to a capture icon or capture button) is received while the camera application is executing, and generate a visual corresponding to a face through the camera, as shown in Figure 6. An image 601 including the object 601-1 may be obtained. Here, the image 601 is in a state 601-3 (e.g., the axis 601-4 of the camera and the axis 601-5 of the electronic device are parallel, and the visual object 601-1 is in the image 601). It may be an image taken in a state where the subject and camera are positioned at the bottom of . As another example, the processor may acquire an image to be used as a preview image displayed through a display (eg, the display device 360 of FIG. 3 or the display 407 of FIG. 4) while the camera application is running. According to one embodiment, the camera is disposed to be exposed through at least a portion of a first side (e.g., front) of the housing of the electronic device, or a second side (e.g., back) facing away from the first side of the housing of the electronic device. It may be arranged to be exposed through at least a portion of.

In operation 503, the processor may identify a plurality of landmarks corresponding to facial components from an image acquired through a camera. For example, as shown in FIG. 6, the processor performs a face detection function from an image acquired through a camera to include at least a part of a visual object (e.g., a face) included in the image through the face detection function. The face area 603-1 that is exposed to the facial area 603-1 may be identified, and a plurality of landmarks 603-3 to 603-9 corresponding to facial components may be identified from the identified face area. For example, the processor may select a first landmark 603-3 (eg, eyebrows), a second landmark 603-5 (eg, eyes), and a third landmark 603 from the face area 603-1. -7) (e.g., nose), or at least one of the fourth landmarks (e.g., 603-9) (e.g., mouth) can be identified. According to one embodiment, when a plurality of landmarks corresponding to a facial component are not identified from the face area, the processor may re-perform an operation of identifying a plurality of landmarks corresponding to the facial component. According to one embodiment, when a plurality of landmarks corresponding to facial components are not identified from the face area, the processor guides the user to retake an image that can identify a plurality of landmarks corresponding to the facial components. In order to do this, indicators (e.g., guidance messages, vibration, sound, or LED flashing) can be provided to the user. According to one embodiment, the shape of the face area 603-1 may include a square shape with the same horizontal and vertical lengths.

At operation 505, the processor, in response to identifying that the ratio of the distances of the first set of landmarks and the second set of landmarks among the plurality of landmarks is included in the first range, generates a corrected image based on the image. A first image may be acquired. The processor may identify distances between at least some of the plurality of landmarks. The processor determines the distance of the first set of landmarks (e.g., first landmark 603-3 and third landmark 603-7) and the second set of landmarks (e.g., 603-7) based on the identified distance. : The ratio of the distances of the second landmark (603-5) and the fourth landmark (603-9) can be identified. If the ratio of the distance of the landmarks of the first set to the distance of the landmarks of the second set is within the first range (e.g., less than 0.95 or more than 1.15), the processor determines that distortion has occurred in the visual object 601-1. By determining this, the image can be corrected. For example, the processor corrects the image using a first correction method using the position (center coordinates of the face area) and length (the length of one side of the face area) associated with the face area, so that the distortion is corrected, as shown in FIG. 6. The first image 605 can be obtained.

According to various embodiments, the processor determines whether the ratio of the distances of the landmarks of the first set to the distances of the landmarks of the second set is not within the first range (e.g., less than 0.95 or greater than 1.15), but The distance of the marks (e.g., the first landmark 603-3 and the second landmark 603-5) and the fourth set of landmarks (e.g., the second landmark 603-5 and the third The ratio of the distance of the landmark 603-7 is included in the second range (e.g., less than 1), or the ratio of the distance of the landmarks of the third set to the distance of the landmarks of the fourth set is within the range of the third set. Even when it is greater than the ratio of the distance of the landmarks and the distance of the fifth set of landmarks (e.g., the third landmark 603-7 and the fourth landmark 603-9), distortion occurs in the visual object. By determining this, the image can be corrected. For example, the processor may acquire a second image by correcting the image using a second correction method using a location (center coordinates of the face area) and length (length of one side of the face area) associated with the face area. .

According to various embodiments, the processor determines whether the ratio of the distances of the first set of landmarks to the second set of landmarks is not within the first range (e.g., less than 0.95 or greater than 1.15) and the third set of lands The distance of the marks (e.g., the first landmark 603-3 and the second landmark 603-5) and the fourth set of landmarks (e.g., the second landmark 603-5 and the third The ratio of the distance of the landmark 603-7 is not included in the second range (e.g., less than 1), and the ratio of the distance of the landmarks of the third set to the distance of the landmarks of the fourth set is within the third set. If it is less than the ratio of the distance of the landmarks of and the distance of the fifth set of landmarks (e.g., the third landmark 603-7 and the fourth landmark 603-9), distortion occurs in the visual object. The operation to correct the image may not be performed because it is determined that it was not done.

In the above, when a plurality of landmarks corresponding to facial elements are detected from the facial area of the image by the electronic device, an operation of correcting a distorted image based on the detected landmarks has been described. However, various embodiments of the present invention According to, the processor of the electronic device can correct a distorted image even when a plurality of landmarks corresponding to facial elements are not detected from the facial area of the image. For example, if a plurality of landmarks corresponding to facial elements are not detected from the facial area of the image, the processor identifies properties (e.g., location or size) of the visual object corresponding to the face, and based on the identified properties, You can also correct the image. For example, the processor determines a distortion map based on the properties of the visual object corresponding to the face, calculates correction parameters for correcting the determined distortion map, and performs image correction operations based on the calculated correction parameters. You can also correct a distorted image by performing . Here, the distortion map is a map that quantifies the degree to which the image is distorted, and the outline of the distortion map may have a shape that is curved in a direction toward the center of the image.

FIGS. 7A and 7B are flowcharts illustrating a method of correcting a distorted image based on a distance ratio between landmarks corresponding to facial elements in an electronic device according to various embodiments. FIG. 8 is an example diagram illustrating a method of correcting a distorted image based on a distance ratio between landmarks corresponding to facial elements in an electronic device according to various embodiments. The following description may be a detailed operation of acquiring the first image in operation 505 of FIG. 5 .

According to various embodiments, the objects 801-13, 803-13, and 805-13 included in the images 801 to 807 of FIG. 8 maintain a positional relationship between the subject and the camera when capturing each image. It only indicates information and does not mean information included in the image.

7A to 7B and 8, in operation 701, a processor (e.g., processor 320 of FIG. 3) of an electronic device (e.g., electronic device 301 of FIG. 3 or electronic device 400 of FIG. 4) ) or the processor 401 of FIG. 4) may identify distances between at least some landmarks among a plurality of landmarks identified from the image. For example, as shown in FIG. 8, the processor calculates the distance 801-3 of the first set of landmarks (e.g., eyebrows and nose) in the face area 801-1 included in the image 801, the second Distance 801-5 of a set of landmarks (e.g., eyes and mouth), a distance of a third set of landmarks (e.g., eyebrows and eyes) 801-7, a fourth set of landmarks (801-7), At least one of the distance 801-9 of (e.g., eyes and nose) or the distance (801-11) of the fifth set of landmarks (e.g., nose and mouth) may be identified. According to one embodiment, the shape of the face area may include a square shape with the same horizontal and vertical lengths.

At operation 703, the processor may determine whether the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is included in the first range. If the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is less than 0.95 or greater than 1.15, the processor determines the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks. It can be judged to be included in the first range. If the ratio of the distance of the first set of landmarks and the distance of the second set of landmarks is included in the first range, the processor performs operation 705 and determines the distance of the first set of landmarks and the second set of landmarks. If the ratio of the distances of the marks is not included in the first range, operation 707 may be performed.

At operation 705, the processor determines if the ratio of the distances of the first set of landmarks to the distances of the second set of landmarks is included in the first range, based on the first correction method using the position and length associated with the facial region. The first image can be obtained. For example, if the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is included in the first range, as in the image 801, the processor may select the face area 801 of the image 801. Determine a weight for image correction based on the center coordinate of -1), determine a correction coefficient for image correction based on the determined weight and the length of one side of the face area 801-1, and determine the correction coefficient based on the correction coefficient. By performing image warping and cropping and resizing at least a portion of the image on which the image warping has been performed, an image 807 with the distortion corrected can be obtained from the image 801. According to one embodiment, the processor may store the distortion-corrected image 807 in a memory (eg, memory 330 in FIG. 3 or memory 403 in FIG. 4). According to one embodiment, while generating and storing the image 807, the processor displays the image 801 as a preview image through a display (e.g., the display device 360 in FIG. 3 or the display 407 in FIG. 4). can be provided. According to one embodiment, the processor may provide a distortion-corrected image 807 as a preview image through a display.

In operation 707, if the ratio of the distance of the first set of landmarks to the distance of the second set of landmarks is not included in the first range, the processor determines the distance of the third set of landmarks and the fourth set of landmarks. It may be determined whether the ratio of the distances is included in the second range. If the ratio of the distance of the third set of landmarks to the distance of the fourth set of landmarks is less than 1, the processor determines that the ratio of the distance of the third set of landmarks to the distance of the fourth set of landmarks is in the second range. It can be judged to be included. If the ratio of the distance of the third set of landmarks and the distance of the fourth set of landmarks is included in the second range, the processor performs operation 711 and determines the distance of the third set of landmarks and the fourth set of landmarks. If the ratio of the distances of the marks is not included in the second range, operation 713 may be performed.

In operation 711, the processor determines if the ratio of the distances of the third set of landmarks to the fourth set of landmarks is included in the second range, based on a second correction method using the position and length associated with the facial region. A second image may be acquired. For example, if the ratio of the distance of the third set of landmarks to the distance of the fourth set of landmarks is included in the second range, as in the image 803, the processor may select the face area 803 of the image 803. Determine a weight for image correction based on the center coordinate of -1), determine a correction coefficient for image correction based on the determined weight and the length of one side of the face area 803-1, and determine the correction coefficient based on the correction coefficient. By performing image warping and cropping and resizing at least a portion of the image on which the image warping has been performed, an image 809 with the distortion corrected can be obtained from the image 803. According to one embodiment, the function used to determine the weight and the function used to determine the correction coefficient in operation 711 are at least partially different from the function used to determine the weight and the function used to determine the correction coefficient in operation 705. can do. According to one embodiment, the processor may store the distortion-corrected image 809 in memory. According to one embodiment, the processor may provide the image 801 as a preview image through a display while generating and storing the distortion-corrected image 809. According to one embodiment, the processor may provide a distortion-corrected image 809 as a preview image through a display.

In operation 713, if the ratio of the distances of the three sets of landmarks and the distances of the fourth set of landmarks is not included in the second range, the processor determines the ratio of the distances of the third set of landmarks and the fourth set of landmarks. It may be determined whether the ratio of distances is greater than the ratio of the distances of the third set of landmarks and the distances of the fifth set of landmarks. The processor performs operation 711 when the ratio of the distance of the third set of landmarks to the distance of the fourth set of landmarks is greater than the ratio of the distance of the third set of landmarks to the distance of the fifth set of landmarks, If the ratio of the distance of the third set of landmarks to the distance of the fourth set of landmarks is not greater than the ratio of the distance of the third set of landmarks to the distance of the fifth set of landmarks, the algorithm can be terminated. . For example, the processor determines that the ratio of the distances of the third set of landmarks to the distance of the fourth set of landmarks is the ratio of the distance of the third set of landmarks to the distance of the fifth set of landmarks, as shown in image 805. If it is greater than the ratio, a weight for image correction is determined based on the center coordinates of the face area 805-1 of the image 805, and the image is determined based on the determined weight and the length of one side of the face area 805-1. From the image 805, determine a correction coefficient for correction, perform image warping based on the correction coefficient, and crop and resize at least a portion of the image on which the image warping has been performed. An image 811 with distortion corrected can be obtained. According to one embodiment, the processor may provide the image 801 as a preview image through a display while generating and storing the distortion-corrected image 811. According to one embodiment, the processor may provide a distortion-corrected image 811 as a preview image through a display.

In the above, it has been described that the electronic device performs operation 713 after performing operation 709. However, according to various embodiments of the present invention, the electronic device performs operation 713 first and then performs operation 709, or operation 709. and operation 713 may be performed in parallel.

FIG. 9 is a diagram illustrating a method of correcting a distorted image based on a position and length associated with a face area in an electronic device according to various embodiments. FIG. 10 is an example diagram illustrating a method of correcting a distorted image based on a position and length associated with a face area in an electronic device according to various embodiments. Figure 11 illustrates an algorithm for calculating image warping coordinates according to various embodiments. The following description may be a detailed operation of acquiring the first image in operation 705 of FIG. 7A or the operation of acquiring the second image in operation 711 of FIG. 7B.

Referring to FIGS. 9 to 11 , in operation 901, the processor (e.g., the processor 320 of FIG. 3 or the electronic device 400 of FIG. 3) of the electronic device (e.g., the electronic device 301 of FIG. 3 or the electronic device 400 of FIG. 4) Processor 401 of 4) may determine weights for image correction based on the location associated with the face area. For example, the processor may determine the distances of a first set of landmarks (e.g., eyebrows and nose) and a second set of landmarks (e.g., eyes and mouth) included in facial region 1001-1 of image 1001. If the ratio of the distance is included in the first range (e.g., less than 0.95 or more than 1.15), the weight for image correction can be determined using <Equation 1> below.

In <Equation 1>, λ _v represents the correction weight based on the vertical position of the face area, f _y represents the weight value corresponding to the vertical coordinate of the center part of the face area, and height may represent the height of the image. . λ _h represents a correction weight based on the horizontal position of the face area, f _x represents a weight value corresponding to the horizontal coordinate of the center part of the face area, and width may represent the width of the image. Here, the values corresponding to f _y and f _x may be determined based on a table or map in which weight values are mapped for each horizontal or vertical coordinate of the image 1001. A table or map to which weight values are mapped may be updated based on information received from an external electronic device (eg, the server 308 or the electronic device 302 or 304 of FIG. 3). For another example, the processor determines that the ratio of the distances of the first set of landmarks included in the face area of the image 1001 to the distance of the second set of landmarks is not included in the first range, but is not included in the third set of lands. The ratio of the distance of the marks (e.g., eyebrows and eyes) to the distance of the fourth set of landmarks (e.g., eyes and nose) is included in the second range (e.g., less than 1) or If the ratio of the distance and the distance of the landmarks of the fourth set is greater than the ratio of the distance of the landmarks of the third set and the distance of the landmarks of the fifth set (e.g., nose and mouth), <Equation 1 above > can be used to determine weights for image correction. According to one embodiment, the shape of the face area may include a square shape with the same horizontal and vertical lengths.

At operation 903, the processor may determine a correction coefficient for image correction based on the weight and length associated with the facial region. For example, if the ratio of the distance of the first set of landmarks included in the face area 1001-1 of the image 1001 and the distance of the second set of landmarks is included in the first range, < By substituting the weight determined based on Equation 1 into <Equation 2> below, a correction coefficient for image correction can be determined.

In <Equation 2>, C _v represents a correction coefficient based on the vertical position of the face area, λ _c represents a constant value that determines the correction strength of the image, and f _width may represent the length of one side of the face area. . C _h may represent a correction coefficient based on the horizontal position of the face area. For another example, the processor determines that the ratio of the distances of the first set of landmarks and the second set of landmarks included in the face area is not included in the first range, but the ratio of the distances of the third set of landmarks and the fourth set of landmarks is not included in the first range. The ratio of the distances of the landmarks of the set is within a second range (e.g., less than 1), or the ratio of the distances of the landmarks of the third set to the distances of the landmarks of the fourth set is greater than the distance of the landmarks of the third set. If it is greater than the ratio of the distances of the fifth set of landmarks, a correction coefficient for image correction can be determined by substituting the weight determined based on <Equation 1> into <Equation 3> below.

In <Equation 3>, C _v represents the correction coefficient based on the vertical position of the face area, λ _c and λ g represent constant values that determine the correction strength of the image, and C _h represents the length of one side of the face area. You can.

At operation 905, the processor may perform image warping based on the correction coefficient. For example, the processor uses the algorithm shown in FIG. 11 below to determine coordinates for transforming at least a portion of the shape of the image 1001 based on the correction coefficient value determined based on <Equation 2> or <Equation 3>. It can be decided through The processor may transform at least part of the shape of the image based on the determined coordinates. For example, the processor determines, through the algorithm shown in FIG. 11, that the first part (P _tl ) of the image 1001 is located in the third part (P' _tl ) calculated by the algorithm shown in FIG. 11, and the image To obtain an image 1003 in which at least a portion of the image 1001 is expanded so that the second part (P _tr ) of (1001) is located in the fourth part (P' _tr ) calculated by the algorithm shown in FIG. 11. You can.

In operation 907, the processor may obtain another image in which image distortion has been corrected based on at least a portion of the image on which image warping has been performed. For example, the processor crops a partial area (1003-1) corresponding to the image 1001 in the image 1003 on which image warping has been performed, and resizes the cropped partial area to correspond to the size of the image 1001 ( By resizing, an image 1005 with distortion corrected can be obtained.

According to various embodiments, an image processing method in an electronic device includes acquiring an image including a visual object corresponding to a face, and identifying a plurality of landmarks corresponding to a facial component from the visual object. In response to identifying that a ratio of a distance of a first set of landmarks among the plurality of landmarks and a distance of a second set of landmarks of the plurality of landmarks is included in a first range, It may include an operation of acquiring a first image corrected based on the image.

According to various embodiments, the first set of landmarks includes a first landmark and a second landmark, the second set of landmarks includes a third landmark and a fourth landmark, The image processing method, in response to identifying that a ratio of a distance of the first set of landmarks to a distance of the second set of landmarks is not included in the first range, sets the first landmark and the An operation of determining whether the distance of a third landmark and the ratio of the distance between the second landmark and the third landmark are included in a second range, and the distance between the first landmark and the third landmark and When the ratio of the distance between the second landmark and the third landmark is within the second range, the method may further include obtaining a second image corrected based on the image.

According to various embodiments, the image processing method is such that the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark is not included in the second range. In this case, the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark is the distance between the first landmark and the third landmark and the second land. An operation of determining whether the ratio of the distance between the mark and the fourth landmark is greater than the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark. If it is greater than the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the fourth landmark, further comprising acquiring the second image corrected based on the image. can do.

According to various embodiments, the image processing method is such that the ratio of the distance between the first landmark and the third landmark and the distance between the second landmark and the third landmark is the ratio of the distance between the first landmark and the third landmark. If the distance between the three landmarks and the distance between the second landmark and the fourth landmark are not greater than the ratio, the operation of maintaining the image may be further included.

According to various embodiments, the operation of acquiring the first image includes determining a weight for image correction based on location information associated with a facial area including at least a portion of the visual object, the weight and the face Determining a first correction coefficient for image correction based on length information associated with an area, performing image warping based on the first correction coefficient, and based on at least a portion of the image on which the image warping was performed. This may include an operation of acquiring the first image.

According to various embodiments, the operation of acquiring the second image includes determining the weight for image correction based on location information associated with the face region, and determining the weight for image correction based on the weight and length information associated with the face region. determining a second correction coefficient for image correction, performing image warping based on the second correction coefficient, and obtaining the second image based on at least a portion of the image on which the image warping was performed. It may include actions such as:

According to various embodiments, location information associated with the face region may include coordinates of the center of the face region, and length information associated with the face region may include the length of one side of the face region.

According to various embodiments, the image processing method includes displaying the image, the first image, or the second image as a preview image and storing the image, the first image, or the second image. It may further include.

According to various embodiments, the first image or the second image is acquired and stored while the image is displayed as a preview image.

According to various embodiments, the image processing method includes, when a plurality of landmarks corresponding to facial components are not identified from the visual object, identifying properties of the visual object, and identifying properties of the visual object. It further includes generating a distortion map based on the distortion map, calculating a correction parameter for correcting the distortion map, and correcting the image based on the correction parameter, wherein the visual object The attribute may include at least one of the location or size of the visual object.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “A. Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be “coupled” or “connected” to another (e.g. second) component, with or without the terms “functionally” or “communicatively”. Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document are one or more instructions stored in a storage medium (e.g., built-in memory 336 or external memory 330) that can be read by a machine (e.g., electronic device 301). It may be implemented as software (e.g., program 340) including these. For example, a processor (e.g., processor 320) of a device (e.g., electronic device 301) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. Device-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or , or one or more other operations may be added.

Claims (20)

In electronic devices,
At least one camera;
a processor operatively connected to the at least one camera; and
comprising a memory operatively coupled to the processor,
The memory, when executed, the processor,
Obtaining an image including a visual object corresponding to a face using the at least one camera,
Identifying a plurality of landmarks corresponding to facial components from the visual object, and
Identifying that a first ratio of a first distance between landmarks of a first set of the plurality of landmarks and a second distance between landmarks of the second set of the plurality of landmarks is included in the first range In response to, storing instructions for obtaining a first image corrected based on the image and a first correction coefficient;
The first distance between the landmarks of the first set and the second distance between the landmarks of the second set are determined based on the vertical coordinates of the image,
the first correction coefficient is determined based on a location within the image of a facial area including the plurality of landmarks,
Whether the first ratio is included in the first range is determined based on a third distance between the center of the image and the center of the face area, and the third distance is determined based on the vertical coordinate of the image. Device. According to paragraph 1,
The first set of landmarks are,
Comprising a first landmark and a second landmark,
The second set of landmarks are,
Includes a third landmark and a fourth landmark,
The instructions are such that the processor,
In response to identifying that the first ratio of the first distance between the first set of landmarks and the second distance between the second set of landmarks is not included in the first range, the first Determine whether a second ratio of a fourth distance between a 1 landmark and the third landmark and a fifth distance between the second landmark and the third landmark is included in a second range, and
When the second ratio of the fourth distance between the first landmark and the third landmark and the fifth distance between the second landmark and the third landmark are included in the second range, the image and obtain a second image corrected based on the second correction coefficient,
the second correction coefficient is determined based on a location within the image of the facial area including the plurality of landmarks,
Whether the second ratio is included in the second range is determined based on a rotation angle of the visual object with respect to the axis of the at least one camera.
According to paragraph 2,
The instructions are such that the processor,
If the second ratio of the fourth distance between the first landmark and the third landmark and the fifth distance between the second landmark and the third landmark are not included in the second range, the Determine whether the fifth distance between the second landmark and the third landmark is greater than the sixth distance between the second landmark and the fourth landmark, and
When the fifth distance between the second landmark and the third landmark is greater than the sixth distance between the second landmark and the fourth landmark, the corrected image based on the image and the second correction coefficient An electronic device for acquiring a second image.
According to paragraph 3,
The instructions are such that the processor,
An electronic device that maintains the image when the fifth distance between the second landmark and the third landmark is not greater than the sixth distance between the second landmark and the fourth landmark.
According to paragraph 3,
The instructions are such that the processor,
Determine weights for image correction based on location information associated with the facial region containing at least a portion of the visual object,
determine the first correction coefficient for image correction based on the weight and length information associated with the facial region;
Perform image warping based on the first correction coefficient, and
An electronic device configured to acquire the first image based on at least a portion of the image on which the image warping was performed.
According to clause 5,
The instructions are such that the processor,
Determine the weights for image correction based on location information associated with the face area,
determine the second correction coefficient for image correction based on the weight and length information associated with the facial region;
Perform image warping based on the second correction coefficient, and
An electronic device configured to obtain the second image based on at least a portion of the image on which the image warping was performed.
◈Claim 7 was abandoned upon payment of the setup registration fee.◈ According to clause 6,
Location information associated with the face area includes coordinates of the center of the face area,
The length information associated with the face area includes the length of one side of the face area.
According to paragraph 3,
further comprising a display operatively connected to the processor,
The instructions are such that the processor,
Displaying the image, the first image, or the second image as a preview image through the display, and
An electronic device configured to store the image, the first image, or the second image in the memory.
According to paragraph 3,
further comprising a display operatively connected to the processor,
The instructions are such that the processor,
acquire the first image or the second image while displaying the image as a preview image through the display, and
An electronic device that stores the acquired image in the memory.
According to paragraph 1,
The instructions are such that the processor,
If a plurality of landmarks corresponding to facial components are not identified from the visual object, identify properties of the visual object,
Generate a distortion map based on the properties of the visual object,
Calculate correction parameters for correcting the distortion map, and
Correcting the image based on the correction parameters,
The properties of the visual object include at least one of the location or size of the visual object.
In an image processing method in an electronic device,
An operation of acquiring an image including a visual object corresponding to a face;
Identifying a plurality of landmarks corresponding to a facial component from the visual object; and
Identifying that a first ratio of a first distance between landmarks of a first set of the plurality of landmarks and a second distance between landmarks of the second set of the plurality of landmarks is included in the first range In response to, obtaining a first image corrected based on the image and the first correction coefficient,
The first distance between the landmarks of the first set and the second distance between the landmarks of the second set are determined based on the vertical coordinates of the image,
the first correction coefficient is determined based on a location within the image of a facial area including the plurality of landmarks,
Whether the first ratio is included in the first range is determined based on a third distance between the center of the image and the center of the facial area, and the third distance is determined based on the vertical coordinate of the image. .
◈Claim 12 was abandoned upon payment of the setup registration fee.◈ According to clause 11,
The first set of landmarks are,
Comprising a first landmark and a second landmark,
The second set of landmarks are,
Includes a third landmark and a fourth landmark,
In response to identifying that the first ratio of the first distance between the first set of landmarks and the second distance between the second set of landmarks is not included in the first range, the first determining whether a second ratio of a fourth distance between a first landmark and the third landmark and a fifth distance between the second landmark and the third landmark is included in a second range; and
When the second ratio of the fourth distance between the first landmark and the third landmark and the fifth distance between the second landmark and the third landmark are included in the second range, the image and acquiring a second image corrected based on the second correction coefficient,
the second correction coefficient is determined based on a location within the image of the facial area including the plurality of landmarks,
Whether the second ratio is included in the second range is determined based on a rotation angle of the visual object with respect to the axis of at least one camera of the electronic device.
◈Claim 13 was abandoned upon payment of the setup registration fee.◈ According to clause 12,
If the second ratio of the fourth distance between the first landmark and the third landmark and the fifth distance between the second landmark and the third landmark are not included in the second range, the determining whether the fifth distance between the second landmark and the third landmark is greater than the sixth distance between the second landmark and the fourth landmark; and
When the fifth distance between the second landmark and the third landmark is greater than the sixth distance between the second landmark and the fourth landmark, the corrected image based on the image and the second correction coefficient The method further includes acquiring a second image.
◈Claim 14 was abandoned upon payment of the setup registration fee.◈ According to clause 13,
The method further includes maintaining the image when the fifth distance between the second landmark and the third landmark is not greater than the sixth distance between the second landmark and the fourth landmark.
◈Claim 15 was abandoned upon payment of the setup registration fee.◈ According to clause 13,
The operation of acquiring the first image is,
determining weights for image correction based on location information associated with the facial area including at least a portion of the visual object;
determining the first correction coefficient for image correction based on the weight and length information associated with the face region;
An operation of performing image warping based on the first correction coefficient; and
A method comprising acquiring the first image based on at least a portion of the image on which the image warping was performed.
◈Claim 16 was abandoned upon payment of the setup registration fee.◈ According to clause 15,
The operation of acquiring the second image is,
determining the weight for image correction based on location information associated with the face area;
determining the second correction coefficient for image correction based on the weight and length information associated with the facial region;
An operation of performing image warping based on the second correction coefficient; and
A method comprising acquiring the second image based on at least a portion of the image on which the image warping was performed.
◈Claim 17 was abandoned upon payment of the setup registration fee.◈ According to clause 16,
Location information associated with the face area includes coordinates of the center of the face area,
The length information associated with the face area includes the length of one side of the face area.
◈Claim 18 was abandoned upon payment of the setup registration fee.◈ According to clause 13,
Displaying the image, the first image, or the second image as a preview image; and
The method further includes storing the image, the first image, or the second image.
◈Claim 19 was abandoned upon payment of the setup registration fee.◈ According to clause 13,
A method wherein the first image or the second image is acquired and stored while the image is displayed as a preview image.
◈Claim 20 was abandoned upon payment of the setup registration fee.◈ According to clause 11,
When a plurality of landmarks corresponding to facial components are not identified from the visual object, identifying attributes of the visual object;
An operation of generating a distortion map based on properties of the visual object;
calculating correction parameters for correcting the distortion map; and
Further comprising the operation of correcting the image based on the correction parameter,
A method wherein the properties of the visual object include at least one of a location or a size of the visual object.

Images