KR20160026281A - Method and apparatus for grid pattern noise removal - Google Patents

Abstract

According to various embodiments of the present invention, a method for removing grid pattern noise comprises the following steps: comparing sample images associated with grid pattern noise according to each of a plurality of displays with an input image obtained by an electronic device so as to infer the grid pattern noise included in the input image; and removing the inferred grid pattern noise from the input image.

Description

METHOD AND APPARATUS FOR GRID PATTERN NOISE REMOVAL BACKGROUND OF THE INVENTION Field of the Invention [0001]

Various embodiments of the invention relate to a method and an electronic device for eliminating grid pattern noise appearing in an image of a display using a camera mounted on an electronic device.

2. Description of the Related Art In recent years, with the development of cameras mounted on portable electronic devices, users often take photographs of displays such as LCD monitors and TVs using cameras of portable electronic devices for the purpose of simple memos and the like. In such a case, when the captured image is viewed, a grid pattern in the digital display overlaps with a grid pattern of the camera mounted on the portable electronic device, and grid pattern noise is generated in the image captured by the user.

In order to eliminate such grid pattern noise, conventionally, a frequency component corresponding to a regular grid pattern noise appearing on the entire image is detected by converting the entire image taken by the display using a camera of an electronic device into a frequency domain, And a method of correcting the grid pattern noise appearing in the photographed image by removing the detected frequency component from the frequency domain of the image is widely provided.

Further, when a display is photographed through a camera mounted on an electronic device, application of a method of changing the hardware for correcting the grid pattern noise by attaching a film having the characteristics of a low-pass filter to the display device has been expanded.

The conventional technique for correcting the grid pattern noise has a problem that the accuracy is not high and thus a satisfactory image can not be provided to the users who take the display with the camera attached to the electronic device.

In addition, if the grid pattern noise appears in various shapes and shapes, it may be difficult to accurately detect and correct the grid pattern noise that appears in various shapes and shapes.

In addition, since portions not corresponding to the grid pattern noise can be removed together, a phenomenon occurs in which the detail of the image seen by the user is deteriorated.

On the other hand, there is a method of attaching a film having a filter characteristic to the display in another way of correcting the grid pattern noise. However, such a change of hardware can be costly, and when the user uses the display device, There is a disadvantage that it becomes inconvenient to do.

Accordingly, an object of the present invention is to provide a grid pattern noise removing method and an electronic apparatus for removing a grid pattern noise accurately deduced from an image of a display of a camera of an electronic device, so that a user can see a clean image will be.

According to one aspect of the present invention, there is provided a method of removing grid pattern noise, comprising the steps of: comparing sample images relating to grid pattern noise according to each of a plurality of displays with an input image acquired by an electronic device, Deducing noise; And removing the deduced grid pattern noise from the input image.

And displaying the input image from which the deduced grid pattern noise is removed.

The inferring operation may include dividing the sample images and the input image into blocks of predetermined sizes.

The inferring operation may include converting each of the sample images and the input image block into a frequency domain.

The inferring operation may include calculating a basis vector from each of the sample images and the block of the input image.

The reasoning operation may include selecting a sample image having the smallest difference between the sample images and the input image.

The inferring operation may include combining a basis vector between the sample images and the input image.

According to another aspect of the present invention, there is provided a grid pattern noise removing apparatus comprising: a storage unit for storing sample images of grid pattern noise according to each of a plurality of displays and an input image acquired by an electronic apparatus; And a control unit for comparing the sample images with the input image to infer the grid pattern noise included in the input image and removing the inferred grid pattern noise from the input image.

And a display unit for displaying an input image from which the inferred grid pattern noise is removed.

The storage unit may include a sample image storage unit for storing sample images of grid pattern noise according to each of the plurality of displays, and an input image storage unit for storing an input image acquired by the electronic device.

The control unit may include a reasoning unit for comparing the sample images with the input image to infer the grid pattern noise included in the input image and a removing unit for removing the inferred grid pattern noise from the input image .

The control unit may divide the sample images and the input image into blocks of a predetermined size.

The control unit may convert each of the sample images and the block of the input image into a frequency domain.

The control unit may calculate a basis vector from each of the sample images and the block of the input image.

The control unit may select a sample image having the smallest difference between the sample images and the input image.

The control unit may combine the basis vectors between the sample images and the input image.

A grid pattern noise canceling method and an electronic apparatus thereof according to various embodiments allow a user to accurately deduce grid pattern noise included in an image of a display of a display with a camera of an electronic device. On the other hand, precisely deduced grid pattern noise is removed from the image, allowing the user to view the clean image.

1 is a schematic block diagram illustrating an electronic device for performing grid pattern noise removal according to an embodiment of the present invention.
2 is a block diagram illustrating a main operation of an electronic device control unit for performing grid pattern noise removal according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating an operation for inferring grid pattern noise according to an embodiment of the present invention.
4 is a flowchart showing an operation of removing grid pattern noise according to an embodiment of the present invention.
5 is an exemplary diagram showing a state in which a modeled input image and a sample image of the present invention are divided into blocks.
FIG. 6 is a diagram illustrating an exemplary image representing a shape obtained by transforming each block of a modeled input image and a sample image into a frequency domain
7 is an exemplary diagram illustrating a state in which a basis vector is calculated in a frequency domain of blocks of a modeled input image and a sample image according to the present invention.
FIG. 8 is a graph showing an example image showing the inferred grid pattern noise when performing speculative operations
9 illustrates an exemplary image representing the inferred grid pattern noise when performing removal operations and an exemplary image from which the inferred grid pattern noise is removed
FIG. 10 illustrates an exemplary image representing grid pattern noise when performing removal operations and an exemplary image in which the inferred grid pattern noise is removed through a band-
11 shows an example image showing the operation of generating a band-stop filter from inferred grid pattern noise when performing removal operations
Figure 12 is a block diagram of an electronic device according to various embodiments of the present invention.

Best Mode for Carrying Out the Invention Various embodiments of the present invention will be described below with reference to the accompanying drawings. The various embodiments of the present invention are capable of various changes and may have various embodiments, and specific embodiments are illustrated in the drawings and the detailed description is described with reference to the drawings. It should be understood, however, that it is not intended to limit the various embodiments of the invention to the specific embodiments, but includes all changes and / or equivalents and alternatives falling within the spirit and scope of the various embodiments of the invention. In connection with the description of the drawings, like reference numerals have been used for like elements.

The use of "including" or "including" in various embodiments of the present invention can be used to refer to the presence of a corresponding function, operation or component, etc., which is disclosed, Components and the like. Also, in various embodiments of the invention, the terms "comprise" or "having" are intended to specify the presence of stated features, integers, steps, operations, components, parts or combinations thereof, But do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

The expression "or" or "at least one of A and / or B" in various embodiments of the present invention includes any and all combinations of words listed together. For example, each of "A or B" or "at least one of A and / or B" may comprise A, comprise B, or both A and B.

&Quot; first, "" second," " first, "or" second, " etc. used in various embodiments of the present invention may modify various elements of various embodiments, I never do that. For example, the representations do not limit the order and / or importance of the components. The representations may be used to distinguish one component from another. For example, both the first user equipment and the second user equipment are user equipment and represent different user equipment. For example, without departing from the scope of the various embodiments of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it is to be understood that the element may be directly connected or connected to the other element, It should be understood that there may be other new components between the different components. On the other hand, when it is mentioned that an element is "directly connected" or "directly connected" to another element, it is understood that there is no other element between the element and the other element It should be possible.

The terminology used in the various embodiments of the present invention is used only to describe a specific embodiment and is not intended to limit the various embodiments of the present invention. The singular expressions include plural expressions unless the context clearly dictates otherwise.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the present invention belong. Terms such as those defined in commonly used dictionaries should be interpreted to have the meanings consistent with the contextual meanings of the related art and, unless expressly defined in the various embodiments of the present invention, It is not interpreted as meaning.

An electronic device according to various embodiments of the present invention may be a device including a communication function. For example, the electronic device can be a smartphone, a tablet personal computer, a mobile phone, a videophone, an e-book reader, a desktop personal computer, a laptop Such as a laptop personal computer (PC), a netbook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device Such as a head-mounted device (HMD) such as electronic glasses, an electronic garment, an electronic bracelet, an electronic necklace, an electronic app apparel, an electronic tattoo, or a smart watch.

According to some embodiments, the electronic device may be a smart home appliance with communication capabilities. [0003] Smart household appliances, such as electronic devices, are widely used in the fields of television, digital video disk (DVD) player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air cleaner, set- And may include at least one of a box (e.g., Samsung HomeSync ™, Apple TV ™, or Google TV ™), game consoles, an electronic dictionary, an electronic key, a camcorder,

According to some embodiments, the electronic device may be a variety of medical devices (e.g., magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT) (global positioning system receiver), EDR (event data recorder), flight data recorder (FDR), automotive infotainment device, marine electronic equipment (eg marine navigation device and gyro compass), avionics, A security device, a head unit for a vehicle, an industrial or home robot, an ATM (automatic teller's machine) of a financial institution, or a POS (point of sale) of a shop.

According to some embodiments, the electronic device may be a piece of furniture or a structure / structure including a communication function, an electronic board, an electronic signature receiving device, a projector, (E.g., water, electricity, gas, or radio wave measuring instruments, etc.). An electronic device according to various embodiments of the present invention may be one or more of the various devices described above. Further, the electronic device according to various embodiments of the present invention may be a flexible device. It should also be apparent to those skilled in the art that the electronic device according to various embodiments of the present invention is not limited to the above-described devices.

Hereinafter, an electronic device according to various embodiments will be described with reference to the accompanying drawings. The term user as used in various embodiments may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, specific items such as a specific category, a handwriting image, and the like are shown. However, this is provided only for the better understanding of the present invention. It is to be understood that certain changes and modifications may be made without departing from the scope of the present invention. And will be apparent to those skilled in the art.

1 is a block diagram of an electronic device 100 that performs a grid pattern noise removal operation in accordance with an embodiment of the present invention. 1, the electronic device 100 may include a control unit 110, a storage unit 120, a display unit 130, a camera unit 140, and an input unit 150. Hereinafter, the storage unit 120 will be described.

The storage unit 120 may store signals or data input / output corresponding to the operations of the display unit 130, the input unit 150, and the camera unit 140 under the control of the controller 110. The storage unit 150 may store control programs, applications, or contents for controlling the electronic device 100 or the control unit 110. [

According to one embodiment, the term "storage unit" includes a storage unit 120, a ROM (not shown) in the control unit 110, a RAM (not shown) (E.g., SD card, memory stick). The storage unit 120 may include a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD).

According to an embodiment of the present invention, the storage unit 120 may store sample images related to grid pattern noise according to each of the plurality of displays and an input image acquired by the electronic device. The storage unit 120 includes a sample image storage unit 121 for storing sample images related to grid pattern noise according to each of a plurality of displays, an input image storage unit 122 for storing an input image acquired by the electronic device 100 ). Here, the plurality of sample images related to the grid pattern noise may be images in which a blank state display is photographed through the camera unit 140 of the electronic device 100. In this case, the manufacturer of the sample images relating to the grid pattern noise can take a plurality of displays in consideration of the pixel, resolution, and the like of the display, and can determine the distance between the display and the camera unit 140 of the electronic device, So that sample images can be produced. In the embodiment of the present invention, the manufacturer performs the operation of producing the above-described sample images in the electronic device 100 of the present invention including the sample image storage unit 121. However, in the electronic device 100 It is also possible to transmit the sample images produced by performing the operation of producing the sample images in a separate electronic device to the electronic device 100 of the present invention without performing the above operation.

The input image storage unit 122 may store images obtained through the camera unit 140 of the electronic device 100. [ The input images stored in the input image storage unit 122 may be images obtained through the camera unit 140 of the electronic device 100 of the present invention including the input image storage unit 122, The image obtained through the camera unit of the electronic device may be an image transmitted to the electronic device 100 of the present invention.

In addition, the input image storage unit 122 may selectively store an image captured by the display among the images acquired through the camera unit 140 of the electronic device 100. In this case, the user may manually select and store the photographed image of the display, and the control unit 110, which will be described later, displays the image obtained through the camera unit 140, or the display of the image input from the separate electronic device 100 The display may automatically select the photographed image and store the photographed image in the input image storage unit 122.

The display unit 130 (e.g., a touch screen) receives a user's operation and can display an execution image, an operation state, and a menu state of the application program. For example, the display unit 130 may provide a user interface corresponding to various services (e.g., call, data transmission, broadcasting, photographing, etc.) to the user. The display unit 130 may transmit an analog signal corresponding to at least one touch input to the user interface to a touch screen controller (not shown). The display unit 130 can receive at least one touch through touchable input means such as a hand touch or an electronic pen (e.g., a stylus pen, hereinafter referred to as an electronic pen). Also, the display unit 130 can receive a continuous movement of one touch among at least one touch. The display unit 130 may transmit an analog signal corresponding to a continuous movement of the input touch to the touch screen controller.

According to one embodiment, the display unit 130 may be implemented by, for example, a resistive method, a capacitive method, an electromagnetic induction (EMR) method, an infrared method, or an acoustic wave method .

According to one embodiment, the touch is not limited to a direct touch with the electronic pen or a hand touch to the display unit 130, and may include non-contact. For example, the display unit 130 may display a touch event caused by touching with a hand or an electronic pen and a touch event caused by a touch with the electronic pen, (E.g., a current value, etc.) recognized by the touch event and the hovering event may be output differently so that the touch event and the hovering event can be separately recognized. In addition, the display unit 130 may output a recognized value (e.g., a current value or the like) differently depending on the distance between the space where the hovering event is generated and the display unit 130. [

According to one embodiment, the touch screen controller may convert an analog signal received from the display unit 130 into a digital signal (e.g., X and Y coordinates) and transmit the digital signal to the controller 110. [ The control unit 110 may control the display unit 130 using the digital signal received from the touch screen controller. For example, the control unit 110 may execute a shortcut icon (not shown) or a shortcut icon (not shown) displayed on the display unit 130 in response to a touch event or a hovering event. According to one embodiment, the touch screen controller may be included in the controller 110.

According to one embodiment, the touch screen controller recognizes a value (e.g., a current value) output through the display unit 130 and can confirm the distance between the space where the hovering event is generated and the display unit 130, The distance value may be converted into a digital signal (for example, Z coordinate) and provided to the control unit 110.

According to one embodiment, the display unit 130 includes at least two touch screen panels capable of recognizing the touch or proximity of the hand touch and the electronic pen, respectively, so that the hand touch and the input by the electronic pen can be input simultaneously . The at least two touch screen panels provide different output values to the touch screen controller, and the touch screen controller recognizes different values input from the at least two touch screen panels, And the input by the electronic pen can be distinguished.

The camera module 140 may include at least one of a first camera (not shown) and a second camera (not shown) You can perform the same general digital camera functions. For example, the first camera or the second camera may be operated to acquire an image through an image sensor (not shown) included in the camera.

The input unit 150 receives a user's operation and may include at least one of a button (not shown), a keypad (not shown), and a microphone (not shown). According to one embodiment, the button may be formed on the front, side, or rear surface of the housing of the electronic device 100, and may be at least one of a power / lock button and a menu button.

According to one embodiment, the keypad may receive a keystroke from a user for control of the electronic device 100. [ The keypad may include a physical keypad formed in the electronic device 100. According to one example, the physical keypad formed on the electronic device 100 may be excluded depending on the performance or structure of the electronic device 100. [ According to one embodiment, the microphone may receive voice or sound according to the control of the controller 110 to generate an electrical signal.

The control unit 110 stores a ROM (not shown) in which a control program for controlling the electronic device 100 is stored, a signal or data input from the outside of the electronic device 100, (RAM) (not shown) used as a storage area for operations performed in the apparatus 100. [ The CPU may include a single core, dual core, triple core, or quad core. The CPU, ROM and RAM may be interconnected via an internal bus (not shown). The control unit 110 may be functionally connected to at least one of the display unit 130, the input unit 150, the camera module 140, and the storage unit 120 to function as a display unit 130, an input unit 150, a camera module 140 ) Or the storage unit 120 in accordance with the present invention.

 The control unit 110 according to an embodiment of the present invention includes a reasoning unit 111 for comparing sample images and an input image to deduce a grid pattern noise included in the input image and a deduction unit 111 for removing the deduced grid pattern noise from the input image (Not shown).

The inference unit 111 may compare the sample images with the input image to deduce the grid pattern noise included in the input image. According to an embodiment of the present invention, an operation of calculating a basis vector of sample images and an input image to calculate a combination that minimizes a basis vector difference between sample images and an input image, And calculating a combination in which the correlation of the basis vectors of the plurality of base vectors is maximized. According to another embodiment of the present invention, a sample image having the smallest difference between the sample images and the input image is selected using a plurality of images without calculating the basis vectors of the sample images and the input image I can reason. In performing the inference using the above-described various embodiments, the shape of the sample images and the input image may be a spatial domain or a frequency domain. At this time, the region shape of the sample images and the input image may be the same region type, and one of the spatial region or the frequency region used in the reasoning unit 111 should be commonly used by the removal unit 112.

The removal unit 112 may remove the grid pattern noise deduced by the reasoning unit 140 from the input image. According to an embodiment of the present invention, a method of calculating and removing the difference between the input image and the inferred grid pattern noise may be used. According to another embodiment of the present invention, a method of generating a band-stop filter having a frequency value higher than a reference value based on the inferred grid pattern noise and applying the generated band-stop filter to an input image may be used have. In performing the cancellation using the above-described various embodiments, the inferred grid pattern noise and the region shape of the input image may be a spatial domain or a frequency domain. This can be the same as the inference performed by the reasoning unit 140. For example, if the inference unit 140 deduces the grid pattern noise from the sample images and the input image in the spatial domain, the removing unit 150 removes the inferred grid pattern noise and the input image The shape may be a space region. However, in the case of the method using the above-described band-stop filter, the inferred grid pattern noise and the shape of the input image may be in the frequency domain. That is, in order to use the band-stop filter as a method of performing the removal, the frequency domain can be used as the form of the sample image and the input image that have previously performed inference.

In addition, the controller 110 checks the display characteristics of the image acquired through the camera unit 140 or the image captured through the separate electronic device 100, and automatically selects the captured image And may be stored in the input image storage unit 122. The operation of dividing the sample images and the input image into blocks of predetermined sizes, the operation of converting the sample images and the input image blocks into the frequency domain, the operation of calculating the basis vectors from the sample images and the input image blocks, It is a matter of course that the control can be selectively performed.

2 is a block diagram illustrating the main operation of the controller 110 of the electronic device 100 for performing grid pattern noise removal according to an embodiment of the present invention.

2, the frequency domain converter 210 may convert the sample images stored in the sample image storage unit 111 from a spatial domain type to a frequency domain type.

The basis vector calculator 220 may calculate a basis vector of the transformed frequency domain type sample images. Here, in an embodiment of the present invention, the basis vectors of the sample images transformed into the frequency domain have been calculated. However, in another embodiment, operations performed by the frequency domain transformer 210 may be omitted, A vector may be calculated. In the preferred embodiment of the present invention, the basis vectors of the sample images are calculated. However, the basis vectors may not be calculated if necessary.

The frequency domain converter 230 may convert the input image stored in the input image storage unit 112 from a spatial domain format to a frequency domain format. This conversion operation can be performed when the sample images are transformed from the spatial domain form to the frequency domain form by the frequency domain transformer 210 of the sample images. If the sample images are not transformed from the spatial domain to the frequency domain by the frequency domain transformer 210 of the sample images, the transform operation performed by the frequency domain transformer 230 of the input image may be omitted.

If the base vector calculator 220 of the sample images is used to calculate the basis vectors of the sample images, the process of calculating the basis vectors of the input image and the input image may be easily performed by a person skilled in the art You can do it. If the operation of calculating the basis vectors of the sample images is omitted, the operation of calculating the basis vectors of the input images may be omitted.

The grid pattern noise inference unit 240 can infer the grid pattern noise included in the input image using the sample images and the basis vector calculated from the input image. If the base vector is not calculated, the grid pattern noise included in the input image may be inferred without using the basis vector.

The grid pattern noise eliminator 250 may remove the inferred grid pattern noise from the input image.

If the spatial domain transformer 260 transforms the sample images and the input image from the spatial domain to the frequency domain in the frequency domain transformer 210 of the sample images and the frequency domain transformer 230 of the input image, . ≪ / RTI > By converting to the spatial domain, we can obtain the input image from which the grid pattern noise is removed. Thus, it is possible to accurately deduce the grid pattern noise included in the image captured by the camera of the electronic device, and to remove the deduced grid pattern noise from the image, thereby allowing the user to view a clean image.

Hereinafter, with reference to FIG. 3, the inference unit 111 of the main part of the control unit 110 of the electronic device 100 performing grid pattern noise removal shown in FIG. 2 will be described in detail.

As shown in FIG. 3, the input image and the sample images can be modeled to deduce the grid pattern noise. First, the input image X stored in the input image storage unit 112 can be defined as Equation 1 below.

Where X ₀ represents an ideal image without grid pattern noise, and N _M Represents a grid pattern noise, and W can represent a weight value. In the embodiment of the present invention, an ideal image without grid pattern noise and a grid pattern noise applied with a weight are modeled as a sum, but it is needless to say that it can be modeled by various methods.

In order to substantially infer the grid pattern noise N _M to be inferred according to the above modeling, in the operation 310 shown in FIG. 3, as shown in FIG. 5, the input image X modeled as Equation Divided into blocks of a set size, and the sample images can also be divided into blocks of predetermined sizes. The number of input image blocks is R can be. According to one embodiment, as shown in FIG. 6, a block of the input image and a block of the sample image can be transformed from the spatial domain form to the frequency domain form. At this time, it is possible to convert from the spatial domain to the frequency domain using discrete Fourier transform or discrete cosine transform. However, it is also possible to perform the inference of the grid pattern noise N _M in the form of the spatial domain, without converting it into the frequency domain.

Next, in operation 320, as shown in FIG. 7, a block of the input image and the sample images divided into blocks can be calculated as a vector of a predetermined size (base vector) of 1xK size. If a vector of a predetermined size (base vector) of a magnitude in the frequency domain is calculated, the value of K can be expressed as 1/2 because the value can be transformed into a symmetric form about the origin in the frequency domain . Also, when the base vector calculation is applied to R images in the input image and the sample images, a vector of a predetermined size in the frequency domain can be represented by RxK. At this time, as a method of calculating the basis vector, a dimension reduction may be performed using a principle component analysis (PCA) or the like to calculate a basis vector from the eigenvalues of the input image and the sample images.

In operation 320, if the base vector is calculated, operation 330 can be continued, and operation 340 can be performed in the case where the base vector is not calculated.

In operation 330, the difference between the input image and the sample images may be minimized or the correlation between the input image and the sample images may be maximized using the calculated basis vectors, A method of inferring grid pattern noise by calculating a combination can be used. As shown in Equation (2), an example of calculating a combination between base vectors having a minimum difference between an input image and sample images can be shown.

Here, X denotes a vector obtained by transforming the input image into 1 * K in the frequency domain, and W _P is a matrix composed of basis vectors of the calculated P sample images, and may have a P * K size. ( W _{P = [E 1 ,, e P]} ). f ^* ( W _P ) is the grid pattern noise calculated by combining the basis vectors of the sample images, and either linear combination or non-linear combination may be possible. Therefore, the grid pattern noise can be deduced within the sample images.

In another embodiment, a scheme of inferring grid pattern noise using an importance combination of basis vectors can be used. For example, when an inner product is used, the combination of importance can be calculated as shown in Equation 3 below.

The grid pattern noise X ^* can be inferred as shown in Equation (4) using the combination of importance calculated as above.

If the input images and the sample images divided into blocks are not changed to the basis vectors, the operation 340 can be continuously performed.

In operation 340, when grid pattern noise is inferred as a linear combination using R sample image data and a corresponding importance, the least squares are used to calculate the grid pattern noise by linear combination of the respective sample images as shown in Equation (5) Pattern noise can be deduced.

Here, T _i represents each sample image converted into a 1 × K vector, and C can represent importance in each sample image. Using this importance, the grid pattern noise can be deduced as shown in Equation (6) below.

Also, in case of using R sample images as they are, grid pattern noise can be inferred by selecting a sample image having the smallest difference from the input image in the sample images as shown in Equation (7) below.

Using the above-described methods, operation 350 can deduce the operation of deducing the grid pattern noise.

Here, the grid pattern noise may exist in whole or in part in the spatial domain or the frequency domain. Therefore, when the grid pattern noise is inferred, the entire input image can be inferred. On the other hand, it can be deduced from a part in the spatial domain. In the case of the frequency domain, it can be inferred not only in the low frequency domain but also in the whole.

FIG. 8 shows an exemplary image of the inferred grid pattern noise when performing inference operation using the inference methods described above.

8, (a) is an input image obtained by a user through a camera module (not shown) of the electronic device 100, (b) is a grid pattern noise included in the input image and a grid of sample images (C) can be converted into a frequency domain, and (d) can be transformed into a frequency domain. In this case, (b) can be deduced in the form that the input image and the sample images are in the spatial domain as shown in (a), and (d) can be deduced in the state where the input image and the sample images are in the frequency domain as shown in (c) have.

Hereinafter, with reference to FIG. 4, a description will be made in detail of the main operation 112 of the control unit 110 of the electronic device 100 performing the grid pattern noise removal shown in FIG.

As shown in FIG. 4, in operation 420, it can be confirmed whether the deduced grid pattern noise is in the form of a frequency domain or a spatial domain. Accordingly, if the deduced grid pattern noise is confirmed to be in the spatial domain form, the 430 operation can perform the removal operation in the spatial domain. In this case, in operation 430, the grid pattern noise deduced in the spatial domain may be removed from the input image as it is. Next, the 480 operation can be performed continuously.

 In the 480 operation, the result of the input image from which the grid pattern noise is removed can be obtained.

9 is an exemplary image in which inferred grid pattern noise is removed when performing removal operations and the inferred grid pattern noise is removed.

Referring to FIGS. 9A to 9C, an image performed in operation 430 can be confirmed. (a) is an input image in a spatial domain, (b) is a grid pattern noise inferred in a spatial domain, and (c) is an example of a result obtained by removing the inferred grid pattern noise shown in the above- It can be a video.

In operation 420, if it is confirmed that the deduced grid pattern noise exists in the frequency domain or the spatial domain, it is possible to perform the removal operation in the frequency domain if it is confirmed that the deduced grid pattern noise is in the frequency domain form. And then perform the operation 440 or 450 operation.

9 (d) to 9 (f), in operation 440, an image obtained by removing inferred grid pattern noise from an input image can be confirmed. At this time, it is possible to remove the deduced grid pattern noise from the input image by using a method of calculating the difference between the input image and the deduced grid pattern noise. (d) is an image obtained by converting an input image in a spatial domain into a frequency domain, (e) is an image obtained by converting the inferred grid pattern noise in the spatial domain into a frequency domain, (f) And may be an image exemplarily showing a result of removing the inferred grid pattern noise shown in the figure. At this time, in (470) operation, (f) can be converted into the spatial domain again.

In another embodiment, if the grid pattern noise deduced in operation 420 is in the frequency domain, operation 450 may continue.

450 operation, a band-stop filter can be formed at a frequency value corresponding to a specific frequency value confirmed in the deduced grid pattern noise. The generated band-stop filter may be applied to an input image to reduce a specific frequency value to eliminate grid pattern noise.

In operation 450, a bandstop filter may be formed to reduce this particular frequency value.

In operation 460, a frequency point corresponding to a specific frequency point identified in the frequency domain of the deduced grid pattern noise may be identified in the frequency domain of the input image, and then the band stop filter may be applied to the point. At this time, autocorrelation or a Gaussian mixture model can be used as a method of constructing the band-stop filter. For example, when a Gaussian mixture model is used, a Gaussian distribution can be predicted by calculating a point where a peak appears in each scan line of a frequency domain image, a size, and an area where the peak occurs . Here, the Gaussian distribution can be changed to a Gaussian distribution centered on a point at the apex, using a change amount (dispersion degree) of the Gaussian distribution in a table calculated in advance according to the area where the apex occurs. If the above process is repeated for each of the vertices, a Gaussian mixture model dispersion can be generated in one scan line. At this time, the sizes of the respective vertices included in the scan line may have the same height. Or may have different heights depending on the size of each of the vertices. Since the values of the points having high values for generating the grid pattern noise must be changed to a low value, the Gaussian mixture distribution is normalized to a value of 0 to 1, and then the normalized value is calculated inversely, The point of emergence can be lowered to a low value to form a band-stop filter.

In the 470 operation, the input image from which the grid pattern noise obtained by applying the band-stop filter is removed can be transformed from the frequency domain to the spatial domain again.

In the 480 operation, an image in which the grid pattern noise is removed can be obtained by using the above-described method.

Figure 10 is an exemplary image showing grid pattern noise when performing removal operations and an exemplary image showing the result of removal of the inferred background pattern noise through a band-stop filter. Referring to FIG. 10, an embodiment in which the above-described band-stop filter is applied can be confirmed. (a), (b), (c), (c), (d) and (d) show the input image in the spatial domain, (b) the inferred grid pattern noise in the spatial domain, (F) may be an image obtained by applying a band-stop filter based on (e), and (c) may be an image obtained by transforming the illustrated (f) into a spatial domain. (C) may be an input image from which grid pattern noise is removed, and may be an input image to be viewed by a user. Accordingly, the user can see a clearer image by removing the grid pattern noise from the image of the display taken by the camera of the electronic device.

Referring to FIG. 11, a process of forming a band-stop filter using the above-described Gaussian mixture model and passing it through an input image to obtain a resultant image can be confirmed. The inferred grid pattern noise may be represented by a waveform, and then a waveform opposite to the inferred grid pattern noise waveform may be generated to form a band-stop filter. It is possible to form such a band-stop filter that does not pass a specific frequency in the frequency domain. As a result, when a frequency band of the input image passes through the band-stop filter, only a specific frequency of the deduced grid pattern noise can be removed.

12 illustrates a block diagram 1200 of an electronic device 1201 in accordance with various embodiments. 12, the electronic device 1201 includes at least one application processor (AP) 1220, a communication module 1220, a subscriber identification module (SIM) card 1224, a memory An input module 1250, a display 1260, an interface 1270, an audio module 1280, a camera module 1291, a power management module 1295, a battery 1296, An indicator 1297, and a motor 1298.

The AP 1210 may control a plurality of hardware or software components connected to the AP 1210 by driving an operating system or an application program, and may perform various data processing and operations including multimedia data. The AP 1210 may be implemented as a system on chip (SoC), for example. According to one embodiment, the AP 1210 may further include a graphics processing unit (GPU) (not shown).

The communication module 1220 can perform data transmission and reception in communication between the electronic device 1201 and other electronic devices (e.g., an electronic device or a server) connected via a network. According to one embodiment, the communication module 1220 includes a cellular module 1221, a Wifi module 1223, a BT module 1225, a GPS module 1227, an NFC module 1228, and a radio frequency (RF) module 1229).

The cellular module 1221 may provide voice, video, text, or Internet services over a communication network (e.g., LTE, LTE-A, CDMA, WCDMA, UMTS, WiBro or GSM). The cellular module 1221 may also perform identification and authentication of electronic devices within the communication network using, for example, a subscriber identity module (e.g., SIM card 1224). According to one embodiment, the cellular module 1221 may perform at least some of the functions that the AP 1210 may provide. For example, the cellular module 1221 may perform at least some of the multimedia control functions.

According to one embodiment, the cellular module 1221_ may include a communication processor (CP), and the cellular module 1221 may be implemented with, for example, SoC. Components such as the cellular module 1221 (e.g., communication processor), the memory 1230 or the power management module 1295 are shown as separate components from the AP 1210, The AP 1210 may be implemented to include at least a portion of the aforementioned components (e.g., cellular module 1221).

According to one embodiment, the AP 1210 or the cellular module 1221 (e.g., a communications processor) may load commands or data received from at least one of non-volatile memory or other components connected to each other into volatile memory, . In addition, the AP 1210 or the cellular module 1221 may store data generated by at least one of the other components or received from at least one of the other components in the non-volatile memory.

Each of the Wifi module 1223, the BT module 1225, the GPS module 1227 or the NFC module 1228 includes a processor for processing data transmitted and received through a corresponding module . Although the cellular module 1221, the Wifi module 1223, the BT module 1225, the GPS module 1227 or the NFC module 1228 are shown as separate blocks in FIG. 12, according to one embodiment, At least some (e.g., two or more) of the wireless module 1221, the Wifi module 1223, the BT module 1225, the GPS module 1227 or the NFC module 1228 may be included in one integrated chip (IC) have. At least some of the processors (e.g., cellular module 1221) corresponding to each of cellular module 1221, Wifi module 1223, BT module 1225, GPS module 1227 or NFC module 1228, And a Wifi processor corresponding to the Wifi module 1223) may be implemented in one SoC.

The RF module 1229 can transmit and receive data, for example, transmit and receive RF signals. The RF module 1229 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module 1229 may further include a component, for example, a conductor or a lead, for transmitting and receiving electromagnetic waves in free space in wireless communication. 12 shows that the cellular module 1221, the Wifi module 1223, the BT module 1225, the GPS module 12270 and the NFC module 1228 share one RF module 1229, According to an example, at least one of the cellular module 1221, the Wifi module 1223, the BT module 1225, the GPS module 1227, or the NFC module 1228 performs transmission and reception of an RF signal through a separate RF module .

The SIM card 1224 may be a card containing a subscriber identity module and may be inserted into a slot formed at a specific location in the electronic device. The SIM card 1224 may include unique identification information (e.g., an integrated circuit card identifier (ICCID)) or subscriber information (e.g., international mobile subscriber identity (IMSI)).

The memory 1230 may include an internal memory 1212 or an external memory 1234. The built-in memory 1212 may be a nonvolatile memory such as a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), or the like, For example, one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, NAND flash memory, And may include at least one.

According to one embodiment, the internal memory 1212 may be a solid state drive (SSD). The external memory 1234 may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro secure digital (SD), a mini secure digital (SD), an extreme digital A Memory Stick, and the like. The external memory 1234 may be functionally connected to the electronic device 1201 through various interfaces. According to one embodiment, the electronic device 1201 may further include a storage device (or storage medium) such as a hard drive.

The sensor module 1240 may measure a physical quantity or sense an operating state of the electronic device 1201, and convert the measured or sensed information into an electrical signal. The sensor module 1240 includes a gesture sensor 1240A, a gyro sensor 1240B, an air pressure sensor 1240C, a magnetic sensor 1240D, an acceleration sensor 1240E, a grip sensor 1240F, A light sensor 1240G and a color sensor 1240H such as an RGB (red, green and blue) sensor, a biosensor 1240I, a temperature sensor 1240J, an illuminance sensor 1240K or an ultra violet sensor 1240M. Or the like. Additionally or alternatively, the sensor module 1240 may include, for example, an E-nose sensor (not shown), an EMG sensor (not shown), an EEG sensor (not shown) (Not shown), an iris sensor (not shown), or a fingerprint sensor (not shown). The sensor module 1240 may further include a control circuit for controlling at least one sensor included in the sensor module 1240.

The input device 1250 may include a touch panel 1252, a (digital) pen sensor 1254, a key 1256, or an ultrasonic input device 1258. The touch panel 1252 can recognize a touch input by at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type, for example. In addition, the touch panel 1252 may further include a control circuit. In electrostatic mode, physical contact or proximity recognition is possible. The touch panel 1252 may further include a tactile layer. In this case, the touch panel 1252 may provide a tactile response to the user.

The (digital) pen sensor 1254 may be implemented using the same or similar method as receiving the touch input of the user, or using a separate recognition sheet. The key 1256 may include, for example, a physical button, an optical key or a keypad. The ultrasonic input device 1258 is an apparatus that can confirm data by sensing a sound wave from the electronic device 1201 to a microphone (e.g., a microphone 1288) through an input tool for generating an ultrasonic signal, Recognition is possible. According to one embodiment, the electronic device 1201 may use the communication module 1220 to receive user input from an external device (e.g., a computer or a server) connected thereto.

The display 1260 may include a panel 1262, a hologram device 1264, or a projector 1266. The panel 1262 may be, for example, a liquid crystal display (LCD) or an active matrix organic light-emitting diode (AM-OLED). The panel 1262 can be implemented, for example, flexible, transparent or wearable. The panel 1262 may be formed of a single module with the touch panel 1252. The hologram device 1264 can display stereoscopic images in the air using interference of light. The projector 1266 can display an image by projecting light onto a screen. The screen may, for example, be located inside or outside the electronic device 1201. According to one embodiment, the display 1260 may further include control circuitry for controlling the panel 1262, the hologram device 1264, or the projector 1266.

The interface 1270 may include, for example, a high-definition multimedia interface 1272, a universal serial bus 1274, an optical interface 1276, or a D-sub (D-subminiature 1278). have. The interface 1270 may include, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card / multi- media card (MMC) interface or an infrared data association (IrDA) interface.

The audio module 1280 can convert sound and electrical signals into both directions. The audio module 1280 may process sound information input or output through, for example, a speaker 1282, a receiver 1284, an earphone 1286, a microphone 1288, or the like.

The camera module 1291 can capture still images and moving images. According to one embodiment, the camera module 1291 may include one or more image sensors such as a front sensor or a rear sensor, a lens (not shown), an image signal processor (ISP) (Not shown) or a flash (not shown), such as an LED or xenon lamp.

The power management module 1295 may manage the power of the electronic device 1201. Although not shown, the power management module 1295 may include, for example, a power management integrated circuit (PMIC), a charger integrated circuit (PMIC), or a battery or fuel gauge.

The PMIC can be mounted, for example, in an integrated circuit or a SoC semiconductor. The charging method can be classified into wired and wireless. The charging IC can charge the battery, and can prevent an overvoltage or an overcurrent from the charger. According to one embodiment, the charging IC may comprise a charging IC for at least one of a wired charging scheme or a wireless charging scheme. The wireless charging system may be, for example, a magnetic resonance system, a magnetic induction system or an electromagnetic wave system, and additional circuits for wireless charging may be added, such as a coil loop, a resonant circuit or a rectifier have.

The battery gauge can measure, for example, the remaining amount of the battery 1296, the voltage during charging, the current or the temperature. The battery 1296 may store or generate electricity, and may supply power to the electronic device 1201 using the stored or generated electricity. The battery 1296 may include, for example, a rechargeable battery or a solar battery.

The indicator 1297 may indicate a specific state of the electronic device 1201 or a portion thereof (e.g., the AP 1210), for example, a boot state, a message state, or a charged state. The motor 1298 can convert an electrical signal into a mechanical vibration. Although not shown, the electronic device 1201 may include a processing unit (e.g., a GPU) for mobile TV support. The processing device for supporting the mobile TV can process media data conforming to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow.

Each of the above-described components of the electronic device according to various embodiments of the present invention may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. The electronic device according to various embodiments of the present invention may be configured to include at least one of the above-described components, and some components may be omitted or further include other additional components. In addition, some of the components of the electronic device according to various embodiments of the present invention may be combined into one entity, so that the functions of the components before being combined can be performed in the same manner.

The term "module" as used in various embodiments of the present invention may mean a unit including, for example, one or a combination of two or more of hardware, software or firmware. A "module" may be interchangeably used with terms such as, for example, unit, logic, logical block, component or circuit. A "module" may be a minimum unit or a portion of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" may be implemented either mechanically or electronically. For example, a "module" in accordance with various embodiments of the present invention may be implemented as an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) And a programmable-logic device.

According to various embodiments, at least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments of the present invention may be, for example, a computer readable And may be implemented with instructions stored on a computer-readable storage medium. The instructions, when executed by one or more processors, may cause the one or more processors to perform functions corresponding to the instructions. The computer readable storage medium may be, for example, a memory. At least some of the programming modules may be implemented (e.g., executed) by, for example, the processor 210. At least some of the programming modules may include, for example, modules, programs, routines, sets of instructions or processes, etc. to perform one or more functions.

The computer-readable recording medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc) A magneto-optical medium such as a floppy disk, and a program command such as a read only memory (ROM), a random access memory (RAM), a flash memory, Module) that is configured to store and perform the functions described herein. The program instructions may also include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the various embodiments of the present invention, and vice versa.

Modules or programming modules according to various embodiments of the present invention may include at least one or more of the elements described above, some of which may be omitted, or may further include other additional elements. Operations performed by modules, programming modules, or other components in accordance with various embodiments of the invention may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. And the like. Accordingly, the scope of various embodiments of the present invention should be construed as being included in the scope of various embodiments of the present invention without departing from the scope of the present invention, all changes or modifications derived from the technical idea of various embodiments of the present invention .

It will also be appreciated that embodiments of the present invention may be implemented in hardware, software, or a combination of hardware and software. Any such software may be stored in a non-volatile storage such as, for example, a storage device such as a ROM or a memory such as, for example, a RAM, a memory chip, a device, or an integrated circuit, whether or not erasable or re- (E.g., a computer), as well as being optically or magnetically recordable, such as, for example, a CD, DVD, magnetic disk or magnetic tape. It will be appreciated that the memory that may be included in the electronic device is an example of a machine-readable storage medium suitable for storing programs or programs containing instructions embodying the embodiments of the present invention. Accordingly, the present invention includes a program including a code for implementing the apparatus or method described in any of the claims, and a machine-readable recording medium storing the program. In addition, such a program may be electronically transported through any medium such as a communication signal transmitted via a wired or wireless connection, and the present invention appropriately includes the same.

Claims (16)

In the grid pattern noise removing method,
Comparing the sample images of the grid pattern noise according to each of the plurality of displays with the input image acquired by the electronic device to infer the grid pattern noise included in the input image;
And removing the deduced grid pattern noise from the input image.
The method according to claim 1,
And displaying the input image with the deduced grid pattern noise removed.
2. The method of claim 1,
And dividing the sample images and the input image into blocks of a preset size.
4. The method of claim 3,
And converting each of the blocks of the sample images and the input image into a frequency domain.
5. The method according to claim 3 or 4,
And calculating a basis vector from each of the sample images and the block of the input image.
5. The method according to claim 3 or 4,
And selecting a sample image having the smallest difference between the sample images and the input image.
6. The method of claim 5,
And combining a basis vector between the sample images and the input image.
A grid pattern noise removing apparatus comprising:
A storage unit for storing sample images of grid pattern noise according to each of the plurality of displays and an input image acquired by the electronic apparatus;
And a control unit for controlling the operation of inferring the grid pattern noise included in the input image and the operation of removing the inferred grid pattern noise from the input image by comparing the sample images with the input image, Removal device.
9. The method of claim 8,
And a display unit for displaying an input image from which the deduced grid pattern noise is removed.
9. The apparatus according to claim 8,
A sample image storage unit for storing sample images of grid pattern noise according to each of the plurality of displays;
And an input image storage unit in which an input image acquired by the electronic device is stored.
9. The apparatus according to claim 8,
A reasoning unit for comparing the sample images with the input image to infer the grid pattern noise included in the input image;
And a removing unit for removing the deduced grid pattern noise from the input image.
9. The apparatus according to claim 8,
And dividing the sample images and the input image into blocks of a preset size.
13. The apparatus according to claim 12,
And converting each of the sample images and the block of the input image into a frequency domain.
The method as claimed in claim 12 or 13,
And calculates a basis vector from each of the sample images and the block of the input image.
The method as claimed in claim 12 or 13,
And selects a sample image having the smallest difference between the sample images and the input image.
15. The apparatus of claim 14,
And a basis vector between the sample images and the input image is combined.

Images