JP2017059235A - Apparatus and method for adjusting brightness of image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus and method for adjusting brightness of an image.SOLUTION: A method of adjusting a brightness of an image includes: matching an object model to an object on the basis of one or more feature points of the object extracted from an input image exhibiting the object S110; mapping a surface normal map in a two-dimensional image form to the input image on the basis of the matched object model S120; and generating shadow information for the input image on the basis of the mapped surface normal map and a virtual light source S130.SELECTED DRAWING: Figure 1

Description

  Hereinafter, a technique for adjusting the brightness of an image is provided.

  Smart devices such as smartphones are widely used, and such smart devices generally have a built-in camera so that users can create content. Smart devices take pictures and videos using a built-in camera. In addition, the smart device provides a function of displaying a photograph taken and reproducing a moving image.

  Furthermore, the smart device provides a function that allows a user to edit an image or the like. For example, the user can change the color and size of the image taken by the smart device according to the user's own preference. The smart device can provide an editing function so that the user can edit the image more conveniently and naturally.

  An object of the present invention is to provide a technique for adjusting the brightness of an image.

  According to one embodiment, a method for adjusting the brightness of an image includes matching an object model to the object based on one or more feature points of the object extracted from an input image in which the object is shown; Mapping a surface normal map in a two-dimensional image format to the input image based on the mapped object model, and generating shadow information for the input image based on the mapped surface normal map and virtual light source Including.

  The step of mapping the surface normal map to the input image includes a step of generating the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located.

  The step of mapping the surface normal map to the input image may convert a surface normal model previously stored in a database with respect to the object model into the surface normal map.

  Matching the object model to the input image includes determining a transformation function that converts the coordinates of the feature points of the object model into the coordinates of the feature points of the input image.

  The step of mapping the surface normal map to the input image includes a step of converting each coordinate of the surface normal model using the determined conversion function to generate a surface normal map in a two-dimensional image format.

  The step of generating the shadow information includes a step of calculating a dot product between a normal vector at each coordinate of the surface normal map and a light vector by the virtual light source at the corresponding coordinate in the input image.

  The step of performing the dot product calculation is a step of determining the light vector incident on the corresponding pixel based on a relationship between the position of the virtual light source and the corresponding position of each pixel in the input image or an illumination direction of the virtual light source. including.

  A method of adjusting the brightness of an image includes generating a reilluminated image by reflecting the shading information in the input image, and a tone of the reilluminated image based on a brightness histogram of the input image. Adjusting the step.

  Adjusting the tone of the re-illuminated image comprises at least the re-illuminated image based on a ratio of the number of pixels in a brightness range predetermined in the input image to the total number of pixels of the input image. Adding weights to pixel tones in some areas.

  The method for adjusting the brightness of the image may further include receiving a control signal indicating at least one of the intensity, position, color, illumination direction, and type of the virtual light source.

  The object may be a human face.

  An apparatus for adjusting brightness of an image according to an embodiment matches an object model to the object based on a feature point of the object extracted from an input image on which the object is shown, and based on the matched object model A processor that maps a surface normal map in a two-dimensional image format to the input image and generates shadow information for the input image based on the mapped surface normal map and a virtual light source;

  The processor may generate the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located.

  The apparatus may further include a memory that stores a surface normal model related to the object model in a database, and the processor may convert the surface normal model to the surface normal map to generate the surface normal map.

  The processor may determine a conversion function that converts the coordinates of the feature points of the object model into the coordinates of the feature points of the input image.

  The processor may convert each coordinate of the surface normal model using the determined conversion function to generate a surface normal map in a two-dimensional image format.

  The processor performs a dot product operation between a normal vector at each coordinate of the surface normal map and a light vector by the virtual light source at the corresponding coordinate in the input image.

  The processor determines the light vector at the corresponding pixel of the input image based on one of a relationship between a position of the virtual light source and a position of each pixel in the input image or an illumination direction of the virtual light source. obtain.

  The processor may generate a reilluminated image reflecting the shading information in the input image, and adjust a tone of the reilluminated image based on a brightness histogram of the input image.

  The processor weights pixel tones in at least some regions of the re-illuminated image based on a ratio of the number of pixels in the brightness range predetermined in the input image to the total number of pixels in the input image. Can be added.

  An apparatus for adjusting the brightness of an image according to another embodiment includes an image receiving unit that receives an input image indicating an object, and an object model based on a feature point of the object extracted from the input image. Matching, mapping a surface normal map in a two-dimensional image format to the input image based on the matched object model, and generating shading information for the input image based on the mapped surface normal map and a virtual light source Processor.

  The image receiving unit may be a camera.

  The apparatus may further include a display that reflects the shading information in the input image.

  The processor may generate the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located.

  The apparatus may further include a memory that stores a surface normal model related to the object model in a database, and the processor may convert the surface normal model to the surface normal map.

  The processor may generate the shading information based on a view vector indicating a direction in which a user views the object and a reflection vector indicating a direction in which light projected from the virtual light source is reflected by the object.

  According to one embodiment, a method for adjusting brightness of an image includes generating a surface normal map of an input image based on an object model and an object feature point in the image, and a virtual light source and the input based on the surface normal map. Generating shading information for the video; and applying the shading information to the input video to adjust the brightness of the input video.

  The object model includes a feature point corresponding to the feature point of the object and a normal vector corresponding to the feature point of the object model, and the step of generating the surface normal map includes the normal vector in the coordinates of the input image. Interpolating the normal vector of the object model to obtain.

  The step of generating the surface normal map may include the step of converting the surface normal model into the surface normal map based on the object model and the feature points of the object.

  The method further includes matching the object model with the object based on the feature points of the object model and the feature points of the object, and generating the surface normal map from the other direction. Generating a surface normal map by combining a result of matching a plurality of surface normal models corresponding to the view with the feature points of the object model matched to the object.

  The step of generating the shadow information may include calculating a dot product between the normal vector at each coordinate of the surface normal map and the light vector of the virtual light source at the corresponding coordinate of the input image as the shadow information.

  The step of generating the shadow information includes: a reflection vector at each coordinate of the input video that indicates a direction in which light projected from the virtual light source on the object is reflected by the object; and a direction in which the user views the object And calculating a dot product between view vectors indicating the shadow information as the shadow information.

  The step of generating the shadow information may include the step of calculating the reflection vector based on the virtual light vector and the surface normal map.

  Applying the shading information to the input image may generate a reilluminated image, and the method may further include adjusting a tone of the reilluminated image based on a brightness histogram of the input image. .

  Adjusting the tone of the re-illuminated image includes selecting a tone adjustment style from a plurality of tone adjustment styles based on a brightness histogram of the input image, and adjusting the tone of the re-illuminated image Applying the selected tone adjustment style to the re-illuminated video image.

  According to the present invention, a technique for adjusting the brightness of an image can be provided.

5 is a flowchart schematically illustrating a method for adjusting image brightness according to an exemplary embodiment; FIG. 4 is a detailed diagram illustrating an example of a method for adjusting the brightness of an image according to an embodiment. FIG. 4 is a detailed diagram illustrating an example of a method for adjusting the brightness of an image according to an embodiment. FIG. 6 is a detailed diagram illustrating another example of a method for adjusting the brightness of an image according to an embodiment; FIG. 6 is a detailed diagram illustrating another example of a method for adjusting the brightness of an image according to an embodiment; FIG. 6 is a detailed diagram illustrating another example of a method for adjusting the brightness of an image according to an embodiment; It is a figure explaining the illustration of the light vector according to the kind of light source by one Embodiment. It is a figure explaining the illustration of the light vector according to the kind of light source by one Embodiment. It is a figure explaining the production | generation of the shadow information based on the view vector and reflection vector by one Embodiment. It is a figure which shows the image re-illuminated according to the intensity | strength of the light source by one Embodiment. It is a figure which shows the image re-illuminated according to the position of the light source by one Embodiment. FIG. 2 illustrates an exemplary graphic user interface of an apparatus for adjusting image brightness according to one embodiment. FIG. 6 is a diagram illustrating an example in which a tone is adjusted to a re-illuminated image according to an embodiment. FIG. 6 illustrates tone adjustment based on an image brightness histogram according to one embodiment. FIG. 6 illustrates tone adjustment based on an image brightness histogram according to one embodiment. FIG. 6 illustrates tone adjustment based on an image brightness histogram according to one embodiment. FIG. 6 illustrates tone adjustment based on an image brightness histogram according to one embodiment. It is a block diagram which shows the structure of the apparatus which adjusts the brightness of the image by one Embodiment. It is a block diagram which shows the structure of the apparatus which adjusts the brightness of the image by one Embodiment. It is a block diagram which shows the structure of the apparatus which adjusts the brightness of the image by one Embodiment. It is a block diagram which shows the structure of the apparatus which adjusts the brightness of the image by one Embodiment.

  Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail with reference to the drawings. The same reference numerals provided in the drawings indicate the same members.

  The terminology used in this embodiment is merely used to describe a specific embodiment, and is not intended to limit the embodiment. An expression used in the singular encompasses the expression of the plural, unless it has a clearly different meaning in the context. In this specification, terms such as “including” or “having” indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It should be understood as not excluding the possibility of the presence or addition of one or more other features or numbers, steps, actions, components, parts, or combinations thereof.

  Unless defined differently, all terms used herein, including technical or scientific terms, are the same as commonly understood by one of ordinary skill in the art to which this embodiment belongs. Has meaning. Commonly used predefined terms should be construed as having a meaning consistent with the meaning possessed in the context of the related art and are ideal or excessive unless explicitly defined herein. It is not interpreted as a formal meaning.

  Moreover, when it demonstrates with reference to drawings, the same referential mark is provided to the same component irrespective of drawing code | symbol, and the duplication description with respect to it is abbreviate | omitted. In the case where it is determined that the specific description of the related art in the description of the present embodiment unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

  FIG. 1 is a flowchart schematically illustrating a method for adjusting image brightness according to an exemplary embodiment.

  First, in step S110, the processor of the apparatus that adjusts the brightness of an image matches an object model to an object based on the feature points of the object extracted from the input image in which the object is shown. For example, the processor may match the object model to an object in the input image by matching the coordinates of each feature point in the object model with the coordinates of the corresponding feature point in the input image. Even if not all feature points designated for the general form of the object have been extracted, the processor can extract a predetermined feature point or extract a predetermined number or more of feature points. Match the object model to the object in the input image. The input image may be a two-dimensional image as an image showing the object. For example, the input image may be each frame of a stopped photo or movie.

  The object is distinguished from the background and indicates the object shown in the input image. There are various types of objects, for example, the types of objects may be human faces, human bodies, animals, things, and so on. The feature point is a point indicating the feature of the object. For example, the feature point of the human face type object is specified as eyes, nose, mouth and the like.

  The object model indicates a general-purpose model in which feature points for a specific type of object are designated in advance. For example, the object model is defined as an image having a predetermined size (for example, the same size as the input image) and each feature point is located at a specific coordinate. An object model describes the general form of a particular type of object (for example, the outline of an object, the geometric structure between object feature points, etc.) and the position at which the feature point is specified on such general form. It can be a 2D image format or 3D image format model. If the object model is in 3D image format, the processor can match the object model to an input image that is 2D.

  According to one embodiment, the processor may match the contour of the object model to the contour of the object. In the following description, it is assumed that the object is a human face type, but the present invention is not limited to this, but can be applied to other types of objects. For example, an object model having a characteristic point of the head of the corresponding animal and a general form may be used for the head of the specific animal.

  For example, the processor may apply a conventional face contour extraction algorithm to an input image in which a human face is shown for accurate matching. As another example, when real-time operation is prioritized, the processor assumes a face as an ellipse (a figure corresponding to the general outline of the object according to the object type), and deforms the ellipse according to the proportion of each feature point. . The face contour extraction algorithm described above requires additional computation time so that the processor provides a preview to the user without storing the real-time preview mode (eg, the image shown in FIG. 10 below in memory). Mode), apply a method of deforming assuming that the face is an ellipse, and store the reilluminated image (for example, store the reilluminated image when the user clicks the capture button) An extraction algorithm can be applied to increase the precision. If the facial contours match exactly, errors that occur at the boundary between the facial contour and the background can be reduced when applying the shadow effect.

  In step S120, the processor maps the surface normal map in the two-dimensional image format to the input image based on the matched object model. The mapping of the surface normal map is described in detail below.

  The surface normal map is a map in which normal vectors corresponding to the surface of the object are mapped to the coordinates of each pixel of the input image. A normal vector mapped to a specific pixel on the object shown in the input image may have a vector value on the normal of the surface of the object at the corresponding pixel. According to an embodiment, the surface normal map is in a two-dimensional image format, for example, a normal vector is mapped to each of two-dimensional coordinates. When the resolution of the input image is m * n, a map in which a normal vector is mapped to the corresponding input image can be expressed as the following formula (1). (Where n and m are integers greater than or equal to 1)

In the above formula (1), the normal vector mapped to the coordinates of each pixel of the input image is represented by three-dimensional vectors w _xy , u _xy , v _xy , and w _xy is the pixel at the (x, y) position. X direction component of the normal vector corresponding to, u _xy represents the y direction component of the normal vector corresponding to the pixel at the (x, y) position, and v _xy represents the z direction component corresponding to the pixel at the (x, y) position. (X is an integer from 1 to n, and y is an integer from 1 to m). For example, the x direction component may be expressed as an R color value, the y direction component as a G color value, and the z direction component as a B color value. However, various coordinate systems can be applied depending on the design without limiting the coordinate system to an orthogonal coordinate system in this specification. In addition, although the normal vector is also expressed as an RGB color value, the present invention is not limited to this and can be expressed in various colors such as a YCbCr color value.

  Next, in step S130, the processor generates shadow information for the input image based on the mapped surface normal map and the virtual light source. For example, the processor generates shadow information indicating a shadow effect when light from a virtual light source is projected onto an object based on the surface normal map. According to one embodiment, the processor performs a dot product operation between the normal vector at each coordinate of the surface normal map and the light vector by the virtual light source at the corresponding coordinate, and generates shadow information based on the result of the dot product operation. The processor determines the light vector by the virtual light source based on the relationship between the position of the virtual light source and the position of each feature point in the input image, and calculates the light vector and calculates the dot product with reference to FIGS. 6A and 6B below. Will be described in detail.

  The shadow effect indicates an effect that pixels in at least a partial region of the input image become brighter or darker. The shadow information is information indicating the degree to which the specific pixel of the input image becomes brighter or darker.

  2 and 3 are diagrams illustrating in detail an example of a method for adjusting the brightness of an image according to an embodiment.

  FIG. 2 is a flowchart of a method for adjusting the brightness of an image using a surface normal map generated by interpolating an object model in which a normal vector is assigned to each feature point. FIG. The process processed by the method shown in 2 is shown.

  First, in step S <b> 210, the image receiving unit of the device that adjusts the brightness of the image receives the input image 301. For example, the image receiving unit may be a camera, and the camera captures the input image 301. When the image receiving unit is a communication unit, the communication unit can receive the input image 301 from outside by wired / wireless communication.

  In step S <b> 220, the processor of the device that adjusts the brightness of the image extracts the feature point 302 of the object indicated in the input image 301. The processor may extract the feature points 302 of the object by a method specialized for the corresponding type according to the type of the object. For example, when the type of the object is a human face, the processor uses an algorithm (for example, PCA (Principal Component Analysis), LDA (Linear Discrimination Analysis), etc.) to extract a feature point 302 of the human face. 302 is extracted. For example, as shown in FIG. 3, the processor extracts feature points 302 corresponding to eyes, nose, and mouth in a human face. The processor extracts at least a part of all feature points set for the general form of the object. However, the algorithm for extracting the feature points is not limited to the above-described content, and the above-described feature points are merely examples, and are not limited to the eyes, nose, and mouth.

  Next, in step S110, the processor matches the object model 201 to the object as shown in FIG. 3, as described above with reference to FIG. As shown in FIG. 3, the object model 201 is stored in the database in advance.

  According to one embodiment, the processor determines a transformation function for matching the object model 201 to the object. The conversion function is a function for converting the coordinates of the feature points of the object model 201 into the coordinates of the feature points 302 of the input image, for example, and shows the relationship between the coordinates of the feature points of the object model and the coordinates of the feature points 302 of the input image. .

  In step S120, the processor maps the surface normal map 303 to the input image 301 as described above with reference to FIG. More specifically, in step S230, the processor generates a surface normal map 303 by interpolating normal vectors at points where the feature points of the matched object model 201 are located. For example, in the object model 201 shown in FIG. 2, a normal vector is designated only at feature points, and the processor can interpolate the normal vector with respect to the space between the feature points. Therefore, the processor can generate the surface normal map 303 that is adaptive to the object in real time. The quality of the surface normal map 303 is determined according to the precision of the geometric structure of the object model 201.

  Next, in step S130, the processor generates the shadow information 304 as described above with reference to FIG. The processor sets the virtual light source by the virtual light source setting 202. The virtual light source setting 202 is set or automatically set by the user as a control signal for setting at least one of the intensity, position, color, illumination direction, and type of the virtual light source. The shadow information 304 includes information on a weight value indicating the degree to which the brightness should increase or decrease at each coordinate of the object, and the area brightly displayed in the shadow information 304 shown in FIG. Indicates an area having a weight value that increases, and an area having a weight value that decreases brightness in a darkly displayed area.

  According to one embodiment, the processor applies the surface normal map 303 aligned based on the facial feature points on the input image 301 and the face based on the direction / position, intensity, type, etc. of the virtual light source specified by the user. Shading information 304 is generated. For example, the processor performs the dot product operation between the normal vector assigned to the surface normal map 303 for each pixel in the face area and the light vector specified for the corresponding pixel, for each pixel in the face area. The shadow information 304 can be calculated.

  In step S240, the processor reflects the shadow information 304 in the input image 301 and adjusts the tone. For example, the processor generates the re-illuminated image 309 reflecting the shadow information 304 in the input image 301. The processor adjusts the tone of the re-illuminated image 309 based on the brightness histogram of the input image 301. According to one embodiment, the processor weights pixel tones corresponding to at least some regions of the re-illuminated image 309 based on a ratio of pixels in a predetermined brightness range to total pixels of the input image 301. Append.

  According to one embodiment, the processor can update the shading information 304 for each pixel in the face region to the final shading information according to the tone adjustment style specified or automatically set by the user. The processor reflects the final shadow information in the input image 301 to express the shadow effect in the face area. In addition, when the processor adjusts the tone of the input image 301 and the re-illuminated image 309, in order to minimize the mismatch in the contour portion of the object (for example, the face), a blur effect or the like is applied to the boundary region between the object and the background. Apply.

  4, 5A, and 5B are diagrams illustrating in detail another example of a method for adjusting the brightness of an image according to an embodiment.

  FIG. 4 is a flowchart illustrating a method for adjusting the brightness of an image using the stored surface normal model 401 with respect to the object model 201 and the object model 201 stored in advance. 5A and 5B show a process in which the input image 301 is processed by the method shown in FIG.

  First, in step S210, the image receiving unit of the apparatus that adjusts the brightness of the image receives the input image 301 as described above with reference to FIG. In step S220, the processor extracts feature points 302 of the object indicated in the input image 301 as described above with reference to FIG. Next, in step S110, the processor matches the object model 201 to the object as shown in FIG. 5A as described above with reference to FIG. In step S120, the processor maps the surface normal map 501 to the input image 301. Here, unlike the surface normal map 303 shown in FIG. 3, the surface normal map 501 shown in FIG. 5A is obtained by converting the surface normal model 401. Next, in step S130, the processor generates the shadow information 304 based on the surface normal map 501 and the virtual light source as described above with reference to FIG. Then, in step S240, the processor adjusts the tone as described above with reference to FIG. Steps S210, 220, 110, 130, and 240 shown in FIGS. 4, 5A, and 5B have been described above with reference to FIGS. However, step S120 shown in FIG. 4 will be described in detail through step S430 in FIG.

  According to one embodiment, in step S430, the processor generates a surface normal map 501 by converting the pre-stored surface normal model 401 shown in FIG. For example, the processor converts the surface normal model 401 by matching each feature point of the surface normal model 401 with the feature point of the object model 201 matched with the object in step S110. According to one embodiment, the processor transforms each coordinate of the surface normal model 401 stored in advance in step S430 using the transformation function determined in step S110 described above, as shown in FIG. 5A. An image-type surface normal map 501 is generated. The surface normal map 501 can be aligned with the position of the object of the input image 301.

  The surface normal model 401 indicates a model in which a normal vector corresponding to each coordinate is mapped for each coordinate of the two-dimensional image. For example, the surface normal model 401 may be a two-dimensional image format model stored in advance with respect to the object model 201. In the surface normal model 401, as shown in FIG. 5A, normal vectors are assigned to the coordinates of each pixel on the general form of a specific type of object, and feature points can be designated in advance.

  Further, according to another embodiment, as shown in FIG. 5B, a plurality of surface normal models 401 to 404 may be stored in advance at various angles. In this case, in step S430, the processor generates a surface normal map 501 by combining the results of matching each of the plurality of surface normal models 401 to 404 with the feature points of the object model 201 matched with the object. For example, the processor calculates the similarity between the feature points of the object model 201 matched with the object and the feature points of the plurality of surface normal models 401 to 404, and sets each of the plurality of surface normal models 401 to 404 as the object. The results matched with the feature points of the model 201 can be combined according to the calculated similarity ratio.

  For example, when the object is looking in a specific direction, the feature points of the object model 201 corresponding to the object and the front surface normal model 401, the upper end surface normal model 402, the lower end surface normal model 403, and the left surface normal model 404 The ratio of the similarity of each feature point is calculated as a: b: c: d (where a, b, c, d are real numbers between 0 and 1, and a, b, c , D is 1). The processor weights a for the result of matching the front surface normal model 401 to the feature point of the object model 201, and weights b for the result of matching the top surface normal model 402 to the feature point of the object model 201. Value, the weight value of c for the result of matching the lower end surface normal model 403 to the feature point of the object model 201, and the weight value of d for the result of matching the left surface normal model 404 to the feature point of the object model 201 Can be generated and the surface normal map 501 can be generated.

  However, in FIG. 5B, only four surface normal models 401 to 404 are shown. However, the surface normal model 501 is stored in advance by storing surface normal models of various angles according to the design without being limited thereto. Can be used to generate. The processor can also generate and utilize a surface normal model for the right side by inverting the left surface normal model 404 horizontally.

  6A and 6B are diagrams illustrating examples of light vectors according to the types of light sources according to an embodiment.

  The type of virtual light source used to adjust the brightness of the image is determined by the user or automatically. The virtual light source may be a spot light source or a directional light source, for example, as a light source set to exert an illumination effect on the input image. However, the type of the virtual light source is not limited to this.

  In this specification, the spot light source indicates a virtual light source set so that light is emitted in all directions from the virtual light source set at a specific position. The directional light source indicates a virtual light source set so that light is emitted in a specific direction. For example, in the case of a light source located at a very far distance such as the sun, light having substantially the same direction reaches.

  The light vector indicates the direction and magnitude of light emitted from the virtual light source with respect to a specific point. The light vector indicates the degree and direction of the light source effect by the virtual light source. Hereinafter, the present specification will be described based on a light vector having a normalized size, but the present invention is not limited to this.

  For example, for a spot light source, the processor determines a light vector incident on the pixel based on the relationship between the position of the virtual light source and the position of each pixel in the input image. Therefore, the light vector for the spot light source can change depending on the position of the spot light source. The light vector of the spot light source will be described with reference to FIG. 6A below.

  As another example, for a directional light source, the processor determines a light vector incident on the corresponding pixel based on the illumination direction of the virtual light source. Therefore, the light vector for the directional light source is set to a constant vector value in advance. The light vector of the directional light source will be described with reference to FIG. 6B below.

  FIG. 6A is a diagram for explaining a light vector by a spot light source.

According to one embodiment, the light emitted from the spot light source 601 is projected onto an arbitrary pixel 630 on the object 620 shown in the input image 610. The position information of the spot light source 601 (for example, the coordinates of the spot light source 601) is designated or automatically designated by the user. The light vector 640 by the spot light source 601 is calculated only with the position information of the spot light source 601. For example, light projected from the spot light source 601 to the corresponding pixel 630 is indicated by the light vector 640, and the light vector 640 can be indicated as a normalized positional difference between the spot light source 601 and the corresponding pixel 630. For example, the position of the spot light source 601 is S _x , S _y , S _z (for example, specified by the user or automatically), and the position of any pixel 630 on the input image 610 is x, y, 0 (where , Assuming that the z-coordinate of all pixels on the input image 610 is 0), the processor determines that the light vector 640 projected to that pixel 630 is L = [x−S _x , y−S _y , −S ^z ] and can be calculated to have a normalized value. For example, the light vector 640 is normalized by dividing x−S _x , y−S _y , and −S _z by the length of the light vector 640.

For example, in the surface normal map, when the normal vector values on the x and y positions on the input image 610 are w _xy , u _xy , and v _xy , the processor uses the light vector L by the spot light source 601 for the x and y positions. W _xy (x−S _x ) + u _xy (y−S _y ) + v _xy (−S _z ) can be calculated as a result value of the dot product operation between the normal vector and the normal vector.

  FIG. 6B is a diagram for explaining a light vector 680 by a directional light source.

According to one embodiment, assuming that the directional light source 609 is located at a very far distance from the input image 650, the same light is projected from the directional light source to all the pixels 670 on the object 660 of the input image 650. obtain. The direction of the directional light source 609 is designated or automatically designated by the user. The directional light source 609 is assumed to be located at a very far distance from the object 660 on the input image 650, and when only the light projection direction by the directional light source 609 is specified, each pixel 670 of the object 660 is designated by the directional light source 609. The light vectors 680 projected onto are all the same. For example, all light vectors 680 projected to each pixel 670 by the directional light source 609 are shown as L = [L _x , L _y , L _z ]. L _x represents the direction of light with respect to the x-axis direction, L _y represents the direction of light with respect to the y-axis direction, and L _z represents the direction of light with respect to the z-axis direction.

For example, when the normal vector values on the x and y positions on the input image 650 in the surface normal map are w _xy , u _xy , and v _xy , the processor uses the light vector L by the directional light source 609 for the x and y positions. W _xy Lx + u _xy Ly + v _xy Lz can be calculated as a result value of the dot product operation between the normal vector and the normal vector.

  In FIGS. 6A and 6B described above, the processor generates the shadow information based on the dot product operation between the light vector and the normal vector, but the present invention is not limited to this. In the following, referring to FIG. 7, another method of generating shadow information will be described.

  FIG. 7 is a diagram illustrating generation of shadow information based on a view vector and a reflection vector according to an embodiment.

  View vector 750 indicates a vector that indicates the direction in which the user views input image 710 or object 720. A reflection vector 742 indicates a vector indicating light reflected after the light projected from the light source 701 reaches any pixel 730 of the object 720. Accordingly, the reflection vector 742 may be a vector in which the light vector 741 is reflected from the object 720. For example, the processor calculates the reflection vector 742 by the following equation.

Here, L indicates an optical vector 741 and n ^ indicates a normal vector 743.

  According to one embodiment, the processor may generate shadow information by dot product 790 of view vector 750 and reflection vector 742. For example, in the general case where the user observes the device from the front, since the view vector 750 is 0, 0, 1, the processor can select only the z-direction component of the reflection vector 742 and generate it as shadow information.

  According to one embodiment, the method of generating shadow information based on the view vector 750 and the reflection vector 742 shown in FIG. 7 is applied to all of the spot light source and the directional light source shown in FIGS. 6A and 6B. The For example, when the spot light source shown in FIG. 6A is set to emit light in a specific direction like the directional light source shown in FIG. 6B, the direction of the reflected light is also shown in the specific direction. The shadow information generated based on the vector 750 can be expressed as bright as it matches the viewpoint.

  FIG. 8 is a diagram illustrating an image re-illuminated according to the intensity of the light source according to an embodiment.

  For example, FIG. 8 shows reilluminated images 810, 820, 830 in which the brightness of the input image 301 has been adjusted by the method shown in FIGS.

  Re-illuminated images 810, 820, and 830 shown in FIG. 8 show examples in which the intensity of the virtual light source is set to be different from each other. When the first re-illuminated image 810 has a low intensity of the virtual light source, the second re-illuminated image 820 has an intermediate intensity of the virtual light source, and the third re-illuminated image 830 has the intensity of the virtual light source. Indicates a high value.

  As shown in FIG. 8, the brightness of the partial areas 821 and 822 of the second re-illuminated image 820 is adjusted to be relatively high compared to the partial areas 811 and 812 of the first re-illuminated image 810. The Further, the brightness of the partial areas 831 and 832 of the third re-illuminated image 830 is adjusted to be relatively higher than the brightness of the partial areas 821 and 822 of the second re-illuminated image 820.

  However, in FIG. 8, only the brightness increase in a partial area of the input image 301 has been described. However, the lighting effect by the virtual light source affects the general area of the object, and is not limited to this. The brightness of the area may be reduced (for example, a shadow effect).

  According to one embodiment, an apparatus that adjusts the brightness of an image according to the intensity of a virtual light source can naturally increase or decrease the brightness of an object shown in the input image 301 without feeling uncomfortable.

  FIG. 9 is a diagram illustrating an image re-illuminated according to the position of the light source according to an embodiment.

  For example, FIG. 9 shows reilluminated images 910, 920, 930 in which the brightness of the input image 301 is adjusted by the method shown in FIGS. In FIG. 9, the virtual light source 909 may be a directional light source.

  Re-illuminated images 910, 920, and 930 shown in FIG. 9 show examples in which the directions of the virtual light sources 909 are set to be different from each other. The fourth re-illuminated image 910 shows a result of applying a virtual light source 909 that is set so that light is projected in the −45 degree direction from the upper left end toward the object. The fifth re-illuminated image 920 shows a result of applying a virtual light source 909 that is set so that light is projected in a 0-degree direction from the front upper end toward the object. The sixth re-illuminated image 930 shows a result of applying a virtual light source 909 that is set so that light is projected in the +45 degree direction from the upper right end toward the object.

  As shown in FIG. 9, in the fourth re-illuminated image 910, the brightness of the lower right end region 911 of the object (human face in FIG. 9) is generated by the virtual light source 909 that projects light in the −45 degree direction. Can be reduced. In the fifth re-illuminated image 920, the brightness of the lower end region 921 of the face can be reduced by the virtual light source 909 that projects light from the upper end of the front toward the object. In the sixth re-illuminated image 930, the brightness of the left lower end region 931 of the face can be reduced by the virtual light source 909 in which light is projected from the right upper end toward the object. In FIG. 9, the shadow effect generated on the object by the virtual light source 909 may be shown.

  However, in FIG. 9, only the reduction in brightness (for example, the shadow effect) of a partial region of the input image 301 has been described. However, the lighting effect by the virtual light source 909 is not limited to this, and the general region of the object May also affect the brightness of some areas.

  According to an embodiment, an apparatus for adjusting the brightness of an image can increase or decrease the brightness of an object shown in an input image naturally without a sense of incongruity depending on the position and direction of the virtual light source 909 and the like.

  FIG. 10 is a diagram illustrating an exemplary graphical user interface (GUI) of an apparatus for adjusting image brightness according to an embodiment.

  The apparatus 1000 that adjusts the brightness of an image can be, for example, a smartphone. However, the present invention is not limited to this. The apparatus 1000 for adjusting the brightness of an image receives a user operation required in the process of performing the method shown in FIGS. For example, FIG. 10 shows a graphic user interface for receiving user operations for virtual light source settings.

  For example, the graphic user interface shown in FIG. 10 includes a light source type interface 1010, a light source position / orientation interface 1020, a light source intensity interface 1030, and the like. The interface type and design are not limited to this, and various interfaces may be used depending on the design.

  The light source type interface 1010 is set to receive an operation related to designation of a virtual light source type from the user. For example, the apparatus 1000 illustrated in FIG. 10 receives an operation for designating any one of a spot light source and a directional light source from the user through the light source type interface 1010.

  The light source position / direction interface 1020 is set to receive an operation related to designation of at least one of the position and direction of the virtual light source from the user according to the type of the virtual light source.

  When the spot light source is selected in FIG. 10, the light source position / direction interface 1020 receives an operation for designating the position of the virtual light source. Since the spot light source projects light in all directions of the light source, if only the position of the spot light source is specified, the light vector from the spot light source is derived for each pixel of the input image (the light vector at any pixel is It is calculated as a positional difference between the coordinates of the corresponding pixel and the coordinates of the spot light source, and the light source effect can be expressed based on the light vector even in an interface for designating only the position. According to one embodiment, the processor provides an interface (not shown) for a plurality of predesignated positions to the user, and upon receiving a selection for the position of the spot light source from the user, sets the corresponding position as the position of the spot light source. To do. For example, if the intermediate icon is activated by the user, the apparatus 1000 can set the position of the virtual light source at a preset point on the front from the object.

  When the directional light source is selected in FIG. 10, the light source position / direction interface 1020 receives an operation for designating the direction of the virtual light source. Since a directional light source can be assumed to project light having the same direction to all pixels on the image, if only the direction of the directional light source is specified, the light vector from the directional light source Is derived for each pixel of the input image, and the light source effect based on the light vector can also be expressed in the interface for designating the direction. According to one embodiment, the processor provides the user with an interface for a plurality of predesignated directions, and if the user receives a selection for the direction of the directional light source, sets the corresponding direction as the direction of the directional light source. For example, when the right icon is activated by the user, the apparatus 1000 can set the direction of the light vector projected from the virtual light source so as to be directed from the upper right end to the object.

  The light source intensity interface 1030 is set to receive an operation related to designation of the intensity of the virtual light source from the user. In FIG. 10, the intensity of the virtual light source is divided from level 0 (for example, virtual light source off) to level 5 (for example, virtual light source intensity maximum). For example, if an icon corresponding to level 3 is activated by the user The apparatus 1000 sets the intensity of the virtual light source to level 3 (intermediate). However, the intensity classification of the virtual light source can be set to be classified into n levels (where n is an integer of 1 or more) without being limited to the above-described content.

  The preview interface provides a re-illuminated image 1009. The preview interface can also provide a result of adjusting the tone of the reilluminated image 1009. For example, the preview interface displays a re-illuminated image 1009 generated by reflecting a user operation received by a light source setting interface (1010 to 1030 in FIG. 10) and a tone adjustment interface (not shown) in real time. provide. For example, if the light source intensity is changed by the light source intensity interface 1030, the apparatus 1000 can display the re-illuminated image 1009 reflecting the changed light source intensity in real time.

  However, the tone of the re-illuminated image 1009 is adjusted without limiting the graphic user interface of the apparatus 1000 for adjusting the brightness of the image to the content shown in FIG. 10 (for example, step S240 in FIGS. 2 and 4). Various interfaces can be provided, such as an interface for taking pictures, an interface for taking pictures, and an interface for storing reilluminated images 1009.

  FIG. 11 is a diagram illustrating an example in which a tone is adjusted to a re-illuminated image according to an embodiment.

  According to one embodiment, the processor of the apparatus adjusts the tone by reflecting the shadow information in the input image 301 in step S240 shown in FIGS. For example, the processor generates a re-illuminated image reflecting shadow information in the input image 301 and adjusts the tone of the re-illuminated image. Although four types of tone adjustment styles are illustrated in FIG. 11, the tone adjustment styles are not limited to this.

  The first style is a style in which weight values are added to the tone of pixels in a light area and a dark area in the image. According to the first style, for example, the processor adds the same weight to the tone of the pixels corresponding to the bright areas and the pixels corresponding to the dark areas. In this case, the re-illuminated image 1110 whose tone is adjusted according to the first style has the same ratio in which the bright area is brightened by the weight value and the dark area is darkened by the weight value. Becomes clear. The first style is denoted as “dark & bright” style.

  A pixel corresponding to a bright region indicates a pixel whose pixel brightness is larger than a predetermined threshold value, and a pixel corresponding to a dark region indicates a pixel whose pixel brightness is equal to or lower than a predetermined threshold value.

  The second style is a style in which the same weight value as that of the first style described above is added, and a specific shadow color is further added to pixels corresponding to dark areas. The re-illuminated image 1120 with the tone adjusted according to the second style may add a shadow color set similar to a human skin tone (eg, brown) to make the shadow effect more natural. The second style is referred to as a “shadow color mixing” style.

  The third style is a style in which a pixel corresponding to a bright region is retained and a weight is added to the tone of a pixel corresponding to a dark region. The re-illuminated image 1130 with the tone adjusted according to the third style is suitable when the dark area is more emphasized and the input image 301 is photographed generally brightly. The third style is denoted as “dark tone” style.

  The fourth style is a style in which pixels corresponding to dark areas are retained and weights are added to the tones of pixels corresponding to light areas. The re-illuminated image 1140 with the tone adjusted according to the fourth style is suitable when the bright area is more emphasized and the input image 301 is photographed generally dark. The fourth style is referred to as “bright tone” style.

  The tone adjustment style described above is determined or automatically determined according to a user operation received using the graphic user interface. The following FIGS. 12 to 15 illustrate an example in which the tone adjustment style is automatically determined.

  12-15 illustrate tone adjustment based on an image brightness histogram according to one embodiment.

  FIG. 12 shows a re-illuminated image 1210 with a tone adjusted according to the first style shown in FIG. 11, FIG. 13 shows a re-illuminated image 1310 with a tone adjusted according to the second style, and FIG. A re-illuminated image 1410 with a tone adjusted by style, FIG. 15 shows a re-illuminated image 1510 with a tone adjusted by a fourth style.

  The processor determines a tone adjustment style based on the input image brightness histograms 1220, 1320, 1420, 1520 and adjusts the tones of the re-illuminated images 1210, 1310, 1410, 1510 according to the determined style. For example, the processor weights the tones of pixels corresponding to at least some regions of the reilluminated image 1210, 1310, 1410, 1510 based on the ratio of pixels in a predetermined brightness range to the total pixels of the input image. Add a value. The brightness histograms 1220, 1320, 1420, and 1520 shown in FIGS. 12 to 15 indicate the brightness distribution of the pixels of the input image, for example, the horizontal axis indicates the brightness, and the vertical axis indicates the horizontal in the input image. Indicates the number of pixels with axis brightness. However, although the brightness histograms 1220, 1320, 1420, and 1520 are related to the input image, the brightness histograms of the re-illuminated images 1210, 1310, 1410, and 1510 are used without being limited thereto. Also good.

  According to one embodiment, the processor may have a pixel number of the input image that is less than or equal to a second threshold brightness but has a pixel number greater than the first threshold brightness that is greater than or equal to a predetermined first ratio of all pixels. In response to the case of the number, the weight value can be added to the dark area and the bright area with the shadow information. For example, the processor may adjust the tone according to the first style when the brightness of the input image is intermediate, such as the brightness histogram 1220 shown in FIG. 12 of the input image.

  According to one embodiment, the processor has a brightness of pixels of the input image that is less than or equal to the brightness of the third threshold, and the number of pixels that is greater than the brightness of the second threshold is greater than or equal to a predetermined second ratio of the total pixels. In response to the number, the weight value is added to the dark area and the bright area by the shadow information, and a predetermined shadow color is added to the dark area. For example, the processor may adjust the tone according to the second style when the brightness of the input image is intermediate, such as the brightness histogram 1320 shown in FIG. 13 of the input image.

  According to an embodiment, the processor is responsive to the shadowing in response to a case where the number of pixels in which the pixel brightness of the input image is greater than the third threshold brightness is equal to or greater than a predetermined third ratio of the total pixels. Add weights to dark areas with information. For example, the processor may adjust the tone according to the third style when the brightness of the input image is strong, such as the brightness histogram 1420 shown in FIG.

  According to one embodiment, the processor is responsive to a case where the number of pixels in which the brightness of the pixels of the input image is less than or equal to the brightness of the first threshold is a number greater than or equal to a predetermined fourth ratio of the total pixels, A weight value is added to a bright area by shading information. For example, the processor may adjust the tone according to the fourth style when the brightness of the input image is weak, such as the brightness histogram 1520 shown in FIG.

  However, the application of each style for tone adjustment is not limited by the above-described contents, and various style application standards can be changed according to user preferences, background image conditions, user operations, designs, and the like. .

  16 to 19 are block diagrams illustrating the configuration of an apparatus for adjusting the brightness of an image according to an embodiment.

  An apparatus 1600 for adjusting the brightness of an image according to one embodiment includes a processor 1610 as shown in FIG. In addition, as illustrated in FIGS. 17 to 19, a memory 1720, an image receiving unit 1830, and a display 1940 may be further included.

  The processor 1610 performs the method of adjusting the brightness of the image described with reference to FIGS. For example, the processor 1610 matches the object model to the object based on the feature point of the object extracted from the input image in which the object is shown, and inputs the surface normal map in the two-dimensional image format based on the matched object model. And shadow information for the input image is generated based on the mapped surface normal map and the virtual light source.

  The memory 1720 includes a computer program stored on a medium for performing the method of adjusting the brightness of an image according to an embodiment in combination with hardware. In addition, the memory 1720 previously stores an object model and a surface normal model related to the object model in a database. For example, the memory 1720 stores the input image, the re-illuminated image, and the re-illuminated image with the tone adjusted, and performs the method shown in FIGS. 1 to 5 such as shadow information and surface normal map. It stores data that is needed or generated in the course of execution.

  The image receiving unit 1830 receives an input image from the outside. For example, the image receiving unit 1830 receives an input image taken by a camera. The image receiving unit 1830 receives an input image through wired / wireless communication.

  Display 1940 displays an image. According to one embodiment, the display 1940 may represent shadow information in the input image. For example, the display 1940 displays an input image, a re-illuminated image, an image with an adjusted tone, and the like. In the present specification, a photograph in which an image is stopped has been described as an example. However, the present invention is not limited to this, and the apparatus 1600 for adjusting the brightness of an image is also applied to a moving image. When the image is a moving image, the processor 1610 can execute the above-described method shown in FIGS. 2 to 15 on each frame of the moving image in real time to apply a natural shadow effect to the object indicated on each frame. . The display 1940 includes a screen, a projector, an LCD, an OLED, and the like.

  The device 1600 according to an embodiment is an electronic device 1600 that includes a processor 1610 capable of processing images, and may be, for example, a smartphone and a camera.

  The apparatus 1600 according to an exemplary embodiment may apply a low-complexity graphic rendering technique (for example, lighting effect by a light source and Soft Shading) to a face taken by a camera to additionally express a natural shadow effect. . The performance is limited in mobile smart devices such as smartphones, but the performance is not degraded while the natural shading effect is reflected in the input image in real time on small smart devices. For example, the apparatus 1600 utilizes a CPU, GPU, and standard graphics S / W built in a mobile smart device such as a smartphone or a camera, and a shadow effect by a virtual light source on an object without complicated physical calculation regarding illumination. Can be expressed in real time.

  In addition, the apparatus 1600 can more naturally reflect the shadow effect when a human face is photographed in front (for example, a selfie camera). The apparatus 1600 can overlay the shadow effect of the virtual light source in real time on the face area when taking a picture (for example, when previewing and storing the taken picture), and the image taken by the above-described method can be displayed. Easy to edit.

  According to one embodiment, the apparatus 1600 can obtain high quality shading effects while utilizing a simple geometric model (eg, object model) without using a complex 3D geometric model. Further, the apparatus 1600 can express a fine shadow effect in real time without performing a complicated calculation by simplifying the lighting calculation into a dot product with a light vector using a surface normal map.

  Further, the apparatus 1600 can manually or automatically select the tone adjustment style according to the shooting situation when applying the shadow effect, so that a more natural shadow effect can be expressed. The device 1600 can obtain a re-illuminated image that is more natural and clearer than the simple filtering of the image.

  The apparatus described above is implemented by hardware components, software components, or a combination of hardware components and software components. For example, the apparatus and components described in the present embodiment include, for example, a processor, a controller, an ALU (arithmetic logic unit), a digital signal processor (digital signal processor), a microcomputer, an FPA (field programmable array), and a PLU (programmable logarithm). unit), a microprocessor, or a different device that executes and responds to instructions, and is implemented using one or more general purpose or special purpose computers. The processing device executes an operating system (OS) and one or more software applications running on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to software execution. For convenience of understanding, one processing device may be described as being used, but those having ordinary knowledge in the art may recognize that the processing device has multiple processing elements and / or multiple types of processing. It can be seen that it contains elements. For example, the processing device includes a plurality of processors or one processor and one controller. Other processing configurations such as parallel processors are also possible.

  The software includes computer programs, code, instructions, or a combination of one or more of these, configures the processing device to operate as desired, and instructs the processing device independently or in combination. Software and / or data is interpreted by the processing device and any type of machine, component, physical device, virtual device, computer storage medium or device for providing instructions or data to the processing device, or transmitted signal It can be realized permanently or temporarily via waves. The software is distributed on computer systems connected to the network and stored or executed in a distributed manner. Software and data are stored on one or more computer-readable recording media.

  The method according to the present embodiment is embodied in the form of program instructions executed through various computer means, and is recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc. alone or in combination. The recording medium and the program instructions may be specially designed and configured for the purpose of the present invention, and may be known and usable by those skilled in the art having computer software technology. . Examples of computer-readable recording media include magnetic media such as hard disks, floppy (registered trademark) disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-lights such as floppy disks. Media and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language code generated by a compiler but also high-level language code executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

  As mentioned above, although embodiment of this invention was described in detail, referring drawings, this invention is not limited to the above-mentioned embodiment, In the range which does not deviate from the technical scope of this invention, it changes variously. It is possible to implement.

1600 Apparatus for Adjusting Image Brightness 1610 Processor 1720 Memory 1830 Image Receiving Unit 1940 Display

Claims (36)

Matching an object model to the object based on one or more feature points of the object extracted from an input image in which the object is shown;
Mapping a normal surface map in the form of a two-dimensional image to the input image based on the matched object model;
Generating shadow information for the input image based on the mapped surface normal map and virtual light source;
A method for adjusting the brightness of an image characterized by comprising:   The step of mapping the surface normal map to the input image includes a step of generating the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located. The method of adjusting the brightness of an image according to claim 1.   The image of claim 1, wherein mapping the surface normal map to the input image includes converting a surface normal model previously stored in a database with respect to the object model to the surface normal map. To adjust the brightness.   The mapping of the object model to the input image includes determining a transformation function that converts the coordinates of the feature points of the object model into the coordinates of the feature points of the input image. How to adjust the brightness of the image described.   The step of mapping the surface normal map to the input image includes the step of converting each coordinate of the surface normal model using the determined conversion function to generate a surface normal map in a two-dimensional image format. The method of adjusting the brightness of an image according to claim 4.   The step of generating the shadow information includes a step of performing a dot product operation between a normal vector at each coordinate of the surface normal map and a light vector by the virtual light source at the corresponding coordinate in the input image. Item 6. A method for adjusting the brightness of an image according to any one of Items 1 to 5.   The step of performing the dot product calculation is a step of determining the light vector incident on the corresponding pixel based on a relationship between the position of the virtual light source and the corresponding position of each pixel in the input image or an illumination direction of the virtual light source. The method of adjusting the brightness of an image according to claim 6, further comprising: Generating a re-illuminated image reflecting the shading information in the input image;
Adjusting the tone of the reilluminated image based on a brightness histogram of the input image;
The method of adjusting the brightness of an image according to claim 1, further comprising:   Adjusting the tone of the re-illuminated image comprises at least the re-illuminated image based on a ratio of the number of pixels in a brightness range predetermined in the input image to the total number of pixels of the input image. 9. The method of adjusting the brightness of an image according to claim 8, further comprising the step of adding a weight value to the tone of a pixel in a partial area.   The method according to claim 1, further comprising receiving a control signal indicating at least one of the intensity, position, color, illumination direction, and type of the virtual light source. How to adjust the brightness of the image.   The method according to claim 1, wherein the object is a human face.   The computer program which makes a processor perform the method as described in any one of Claims 1 thru | or 11.   An object model is matched with the object based on the feature points of the object extracted from the input image in which the object is shown, and a surface normal map in a two-dimensional image format is mapped to the input image based on the matched object model And a processor for generating shading information for the input image based on the mapped surface normal map and a virtual light source.   The image brightness of claim 13, wherein the processor generates the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located. apparatus. A memory for storing a surface normal model related to the object model in a database;
The apparatus of claim 13, wherein the processor converts the surface normal model into the surface normal map.   The apparatus of claim 13, wherein the processor determines a conversion function for converting the coordinates of the feature points of the object model into the coordinates of the feature points of the input image.   The brightness of an image according to claim 16, wherein the processor generates a surface normal map of a two-dimensional image format by converting each coordinate of the surface normal model using the determined conversion function. Device to adjust.   The said processor performs the dot product calculation between the light vector by the said virtual light source in the said coordinate with the normal vector of each coordinate of the said surface normal map, and the said input image, It is any one of Claim 13 thru | or 17 characterized by the above-mentioned. The apparatus which adjusts the brightness of the image as described in a term.   The processor determines the light vector at a pixel of the input image based on any one of a relationship between a position of the virtual light source and a position of each pixel in the input image or an illumination direction of the virtual light source. The apparatus for adjusting the brightness of an image according to claim 18.   The processor generates a re-illuminated image by reflecting the shadow information on the input image, and adjusts the tone of the re-illuminated image based on a brightness histogram of the input image. The apparatus for adjusting the brightness of an image according to any one of claims 13 to 19.   The processor weights pixel tones in at least some regions of the re-illuminated image based on a ratio of the number of pixels in the brightness range predetermined in the input image to the total number of pixels in the input image. 21. The apparatus according to claim 20, wherein the brightness of the image is adjusted. An image receiver for receiving an input image showing the object;
Matching an object model to the object based on feature points of the object extracted from the input image, mapping a surface normal map in a two-dimensional image format to the input image based on the matched object model, and A processor for generating shadow information for the input image based on the mapped surface normal map and the virtual light source;
The apparatus characterized by including.   The apparatus of claim 22, wherein the image receiving unit is a camera. The processor is configured to apply the shading information to the input image;
24. The apparatus according to claim 22 or 23, further comprising a display that reflects the shading information in the input image.   The apparatus according to claim 22 or 23, wherein the processor generates the surface normal map by interpolating a normal vector at a point where each feature point of the matched object model is located. A memory for storing a surface normal model related to the object model in a database;
The apparatus according to claim 22 or 23, wherein the processor converts the surface normal model into the surface normal map.   The processor generates the shadow information based on a view vector indicating a direction in which a user views the object and a reflection vector indicating a direction in which light projected from the virtual light source is reflected by the object. 27. Apparatus according to any one of claims 22 to 26, characterized in that In the method of adjusting the brightness of the image,
Generating a surface normal map of the input video based on the object model and the feature points of the object in the video;
Generating shadow information for the input image based on a virtual light source and the surface normal map;
Applying the shading information to the input video to adjust the brightness of the input video;
A method comprising the steps of: The object model includes a feature point corresponding to the feature point of the object and a normal vector corresponding to the feature point of the object model,
29. The method of claim 28, wherein generating the surface normal map includes interpolating the normal vector of the object model to obtain a normal vector at the coordinates of the input image.   29. The method of claim 28, wherein generating the surface normal map comprises transforming a surface normal model into the surface normal map based on the object model and the feature points of the object. Matching the object model with the object based on the feature points of the object model and the feature points of the object;
The step of generating the surface normal map combines results obtained by matching a plurality of surface normal models corresponding to the object view from other directions with the feature points of the object model matched to the object. 29. The method of claim 28, comprising generating the surface normal map by:   The step of generating the shadow information includes a step of calculating a dot product between the normal vector at each coordinate of the surface normal map and the light vector of the virtual light source at the corresponding coordinate of the input image as the shadow information. 32. A method according to any one of claims 28 to 31.   The step of generating the shadow information includes: a reflection vector at each coordinate of the input video that indicates a direction in which light projected from the virtual light source on the object is reflected by the object; and a direction in which the user views the object 32. The method according to claim 28, further comprising a step of calculating a dot product between view vectors instructing as the shading information.   The method of claim 33, wherein generating the shadow information includes calculating the reflection vector based on a light vector of the virtual light source and the surface normal map.   Applying the shading information to the input image generates a re-illuminated image, and the method further includes adjusting a tone of the re-illuminated image based on a brightness histogram of the input image. 35. A method according to any one of claims 28 to 34, characterized in that Adjusting the tone of the re-illuminated image comprises:
Selecting a tone adjustment style from a plurality of tone adjustment styles based on a brightness histogram of the input video;
Applying the selected tone adjustment style to the re-illuminated video to adjust the tone of the re-illuminated video;
36. The method of claim 35, comprising: