US20180114294A1 - Image enhancement device and method for convolutional network apparatus - Google Patents
Image enhancement device and method for convolutional network apparatus Download PDFInfo
- Publication number
- US20180114294A1 US20180114294A1 US15/716,490 US201715716490A US2018114294A1 US 20180114294 A1 US20180114294 A1 US 20180114294A1 US 201715716490 A US201715716490 A US 201715716490A US 2018114294 A1 US2018114294 A1 US 2018114294A1
- Authority
- US
- United States
- Prior art keywords
- image
- correction
- image enhancement
- enhancement device
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 22
- 238000012937 correction Methods 0.000 claims abstract description 126
- 238000005070 sampling Methods 0.000 claims abstract description 42
- 238000012549 training Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 description 7
- 238000010200 validation analysis Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 238000003709 image segmentation Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000003702 image correction Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G06T5/003—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the present disclosure relates to an image-processing technology. More particularly, the present disclosure relates to an image enhancement device and image enhancement method for a convolutional network apparatus.
- Image segmentation is a challenging problem in computer graphics, particularly with respect to digitally separating a desired object (foreground) from a background of the digital image. Such separation is useful during digital editing of still images and video.
- deep learning model-based segmentation method is used to perform segmentation.
- the segmentation result of such a method is often affected by image scale, lighting and location changes.
- An aspect of the present disclosure is to provide an image enhancement device that includes a down-sampling module, correction modules and an up-sampling module is provided.
- the down-sampling module is configured to down-sample an input image to generate down-sampled images having different down-sampled resolutions.
- Each of the correction modules is coupled to the down-sampling module, and is configured to perform correction on one of the down-sampled images according to at least one correction model based on at least one correction parameter to generate one of corrected images.
- the up-sampling module is coupled to the correction modules, and is configured to up-sample the corrected images to generate up-sampled images, wherein each of the up-sampled images is of a same up-sampled resolution.
- the concatenating module is coupled to the up-sampling module, and concatenates the up-sampled images into an output image.
- Another aspect of the present disclosure is to provide a convolutional network apparatus that includes operation layers and an image enhancement device.
- the operation layers are operated in series and are configured to perform convolution on an under-processed input image to generate a processed output image.
- the image enhancement device includes a down-sampling module, correction modules and an up-sampling module is provided.
- the down-sampling module is configured to down-sample an input image to generate down-sampled images having different down-sampled resolutions, wherein the input image is either received from a first one of the operation layers or is either the under-processed input image.
- Each of the correction modules is coupled to the down-sampling module, and is configured to perform correction on one of the down-sampled images according to at least one correction model based on at least one correction parameter to generate one of corrected images.
- the up-sampling module is coupled to the correction modules, and is configured to up-sample the corrected images to generate up-sampled images, wherein each of the up-sampled images is of a same up-sampled resolution.
- the concatenating module is coupled to the up-sampling module, and concatenates the up-sampled images into an output image.
- Yet another aspect of the present disclosure is to provide an image enhancement method used in an image enhancement device that includes the steps outlined below.
- An input image is down-sampled to generate a plurality of down-sampled images.
- Correction is performed on the down-sampled images each according to at least one correction model based on at least one correction parameter to generate a plurality of corrected images.
- the corrected images are up-sampled to generate a plurality of up-sampled images, wherein each of the up-sampled images is of a same resolution.
- the up-sampled images are concatenated into an output image.
- FIG. 1 is a diagram illustrating a convolutional network apparatus in an embodiment of the present disclosure
- FIG. 2 is a block diagram of the image enhancement device in an embodiment of the present invention.
- FIG. 3 is an image enhancement method in an embodiment of the present invention.
- FIG. 1 is a diagram illustrating a convolutional network apparatus 1 in an embodiment of the present disclosure.
- the convolutional network apparatus 1 includes operation layers L 1 , L 2 , L 3 , . . . , Ln and an image enhancement device 10 .
- the operation layers L 1 , L 2 , L 3 , . . . , Ln may include such as, but not limited to a convolution layer, a pooling layer, a dropout layer, a batch normalization layer, a dilation convolution layer, an active function layer or a combination thereof.
- the convolutional network apparatus 1 is a fully convolutional neural network for performing image segmentation.
- the operation layers L 1 , L 2 , L 3 , . . . , Ln included therein are operated in series along the direction A and are configured to perform convolution on an under-processed input image F in to generate a processed output image F out .
- the processed output image F out may include such as, but not limited to at least one segmented feature (not illustrated).
- each of the operation layers L 1 , L 2 , L 3 , . . . , Ln is illustrated as a cubic and is operated to perform convolution on an input image processed by a previous operation layer to generate an output image to a subsequent operation layer, except that the first operation layer L 1 performs convolution on the under-processed input image F in and the last operation layer Ln generates the processed output image F out .
- the operation layer L 2 performs convolution on an image (not illustrated) processed by the operation layer L 1 and generates a processed image (not illustrated) to the operation layer L 3 .
- the image enhancement device 10 is disposed either before the operation layer L 1 or between any two of the operation layers, e.g. the operation layers L 2 and L 3 .
- the image enhancement device 10 is disposed before the operation layer L 1 the image enhancement device 10 is configured to perform image enhancement on the under-processed input image F in such that the operation layer L 1 substantially receives the enhanced result, e.g. an enhanced image F in ′ illustrated in FIG. 1 .
- the operation layers disposed after the image enhancement device 10 can perform convolution by using an enhanced image.
- the convolutional network apparatus 1 can accomplish a better segmentation result.
- the embodiment of the image enhancement device 10 disposed before the operation layer L 1 is merely an example.
- the image enhancement device 10 can be disposed between such as, but not limited to the operation layers L 2 and L 3 to perform image enhancement on the image finished processing by the operation layer L 2 such that the operation layer L 3 substantially receives the enhanced result.
- the convolutional network apparatus 1 may include more than one image enhancement devices each disposed between different pair of operation layers.
- FIG. 2 is a block diagram of the image enhancement device 10 in an embodiment of the present invention.
- the image enhancement device 10 includes a down-sampling module 100 , correction modules 102 A, 102 B and 102 C, an up-sampling module 104 , a concatenating module 106 and a comparison module 108 .
- the down-sampling module 100 , the correction modules 102 A, 102 B and 102 C, the up-sampling module 104 , the concatenating module 106 and the comparison module 108 can be realized by, for example, one or more processors, such as central processors and/or microprocessors, but are not limited in this regard.
- the down-sampling module 100 , the correction modules 102 A, 102 B and 102 C, the up-sampling module 104 , the concatenating module 106 and the comparison module 108 include one or more memory devices, each of which includes, or a plurality of which collectively include a computer readable storage medium.
- the computer readable storage medium may include a read-only memory (ROM), a flash memory, a floppy disk, a hard disk, an optical disc, a flash disk, a flash drive, a tape, a database accessible from a network, and/or any storage medium with the same functionality that can be contemplated by persons of ordinary skill in the art to which this disclosure pertains.
- the down-sampling module 100 , the correction modules 102 A, 102 B and 102 C, the up-sampling module 104 , the concatenating module 106 and the comparison module 108 may run or execute various software programs and/or sets of instructions stored in memory (not illustrated) to perform various functions for the image enhancement device 10 and to process data.
- the down-sampling module 100 is configured to down-sample an input image, such as but not limited to the under-processed input image F in , to generate down-sampled images D 1 , D 2 and D 3 having different down-sampled resolutions.
- the resolutions of the down-sampled images D 1 , D 2 and D 3 can be such as, but not limited to 200 ⁇ 200 (down-sampled by 1), 100 ⁇ 100 (down-sampled by 1 ⁇ 2) and 50 ⁇ 50 (down-sampled by 1 ⁇ 4) respectively.
- Each of the correction modules 102 A, 102 B and 102 C is coupled to the down-sampling module 100 , and is configured to perform correction on one of the down-sampled images D 1 , D 2 and D 3 according to at least one correction model based on at least one correction parameter to generate one of corrected images C 1 , C 2 and C 3 .
- the correction model can be any appropriate mathematics model used to perform image correction.
- each of the correction modules 102 A, 102 B and 102 C includes a blur correction model BL, a color correction model CO and a luminous correction model LU.
- the blur correction model BL is configured to perform blur correction on an image, e.g. the under-processed input image F in , according to at least one correction parameter related to the blur correction operation.
- the blur region in the image processed by the blur correction model BL would become clear when the at least one correction parameter is well-selected.
- the color correction model CO is configured to perform color correction on an image, e.g. the under-processed input image F in , according to at least one correction parameter related to the color correction operation.
- the region having false colors in the image processed by the color correction model CO would be corrected when the at least one correction parameter is well-selected.
- the luminous correction model LU is configured to perform luminous correction on an image, e.g. the under-processed input image F in , according to at least one correction parameter related to the luminous correction operation.
- the region having low brightness in the image processed by the luminous correction model LU would become brighter when the at least one correction parameter is well-selected.
- the up-sampling module 104 is coupled to the correction modules 102 A, 102 B and 102 C, and is configured to up-sample the corrected images C 1 , C 2 and C 3 to generate up-sampled images U 1 , U 2 and U 3 , wherein each of the up-sampled images U 1 , U 2 and U 3 is of a same up-sampled resolution.
- each of the up-sampled images U 1 , U 2 and U 3 also has the resolution of 200 ⁇ 200.
- the concatenating module 106 is coupled to the up-sampling module 104 , and is configured to concatenate the up-sampled images U 1 , U 2 and U 3 into an output image.
- the output image generated by the concatenating module 106 is the image F in ′.
- the comparison module 108 is configured to calculate a cost value CV based on a cost function according to a difference between a processed image processed by at least one operation layer subsequent to the image enhancement device 10 and a predetermined image P corresponding to the under-processed input image F in .
- the processed image is the processed output image F out .
- the processed image can be the image processed by either the operation layer L 1 , L 2 , L 3 , . . . or Ln.
- the comparison module 108 may calculate the cost value CV based on the cost function according to a difference between the output image F out and the predetermined image P.
- the output image is such as, but not limited to a feature map
- the predetermined image P is such as, but not limited to a Ground Truth.
- the predetermined image P is different when different images processed by one of the operation layer L 1 , L 2 , L 3 , . . . or Ln is used as the processed image.
- each of the correction modules 102 A, 102 B and 102 C is further configured to receive the cost value CV and adjust the correction parameter accordingly so as to decrease the cost value CV.
- the under-processed input image F in can be a blurred version of the predetermined image P and can be processed by the down-sampling module 100 , the correction modules 102 A, 102 B and 102 C, the up-sampling module 104 and the concatenating module 106 as described above.
- the comparison module 108 compares the predetermined image P and the output image, i.e. the image F in ′, to generate at least one cost value CV such that each of the correction modules 102 A, 102 B and 102 C adjusts the correction parameter accordingly.
- the correction modules 102 A, 102 B and 102 C determine whether the cost value CV is smaller than a predetermined threshold value such that when the cost value CV is smaller than the predetermined threshold value, the image enhancement device 1 is operated in the operation mode and each of the correction modules 102 A, 102 B and 102 C ceases to adjust the correction parameter.
- the under-processed input image F in can be processed by the modules mentioned above again, in which each of the correction modules 102 A, 102 B and 102 C uses the adjusted correction parameter to perform correction.
- the comparison module 108 compares the predetermined image P and the newly generated output image to generate the cost value CV.
- each of the correction modules 102 A, 102 B and 102 C keeps adjusting the correction parameter and the procedure described above is repeated until the cost value CV is smaller than the predetermined threshold value.
- each of the correction modules 102 A, 102 B and 102 C ceases to adjust the correction parameter.
- each of the correction modules 102 A, 102 B and 102 C simply uses the correction parameter to perform correction such that the concatenating module 106 generates the image F in ′ based on the correction performed according to the correction parameter.
- the image enhancement device 10 in the present invention is able to perform image enhancement based on the learning result to perform such as, but not limited blur, color and luminous correction.
- the convolutional network apparatus 1 in the present invention is able to use the enhanced image to perform convolution to obtain a better image segmentation result.
- the operation layers of the convolutional network apparatus 1 and the modules in the image enhancement device 10 can be implemented by either hardware components or software modules.
- a validation accuracy is calculated to determine whether the training mode is finished.
- the validation accuracy is calculated based on such as, but not limited to a training set and a validation set of input images. When the validation accuracy is higher than a predetermined value, the training mode is finished and the image enhancement device 10 is further operated in the operation mode. Further, in other embodiments, a multiple of indexes, such as, but not limited to the cost value CV, the validation accuracy, a training set error, a validation error or a combination thereof can be used to determine whether the training mode is finished.
- FIG. 3 is an image enhancement method 300 in an embodiment of the present invention.
- the image enhancement method 300 can be used in the image enhancement device 10 illustrated in FIG. 2 , or be implemented by using other hardware components such as a database, a common processor, a computer, a server, other unique hardware devices that have a specific logic circuit or an equipment having a specific function, e.g. a unique hardware integrated by a computer program and a processor or a chip.
- the image enhancement method 300 is implemented by using a computer program having computer readable instructions to control the modules in the image enhancement device 10 .
- the instructions can be stored in a memory, in which the memory can be a non-transitory computer readable medium such as a ROM (read-only memory), a flash memory, a floppy disc, a hard disc, an optical disc, a flash disc, a tape, an database accessible from a network, or any storage medium with the same functionality that can be contemplated by persons of ordinary skill in the art to which this disclosure pertains.
- the detail of the image enhancement method 300 illustrated in FIG. 3 is described in accompany with FIG. 2 .
- the image enhancement method 300 includes the steps outlined below (The steps are not recited in the sequence in which the steps are performed. That is, unless the sequence of the steps is expressly indicated, the sequence of the steps is interchangeable, and all or part of the steps may be simultaneously, partially simultaneously, or sequentially performed).
- an input image e.g. the under-processed input image F in
- the down-sampling module 100 to generate the down-sampled images D 1 , D 2 and D 3 .
- step 302 correction is performed on the down-sampled images D 1 , D 2 and D 3 each according to at least one correction model by the correction modules 102 A, 102 B and 102 C based on at least one correction parameter to generate the corrected images C 1 , C 2 and C 3 .
- step 303 the corrected images C 1 , C 2 and C 3 are up-sampled by the up-sampling module 104 to generate the up-sampled images U 1 , U 2 and U 3 , wherein each of the up-sampled images U 1 , U 2 and U 3 is of a same resolution.
- step 304 the up-sampled images U 1 , U 2 and U 3 are concatenated into an output image, e.g. the image F in ′, by the concatenating module 106 .
- step 305 the output image, e.g. the image F out , is compared with a predetermined image P corresponding to the under-processed input image F in by the comparison module 108 to generate at least one cost value CV.
- step 306 the correction modules 102 A, 102 B and 102 C determine whether the cost value CV is smaller than a predetermined threshold value.
- step 307 when the cost value CV is not smaller than the predetermined threshold value, the image enhancement device 1 is operated in the training mode such that each of the correction modules 102 A, 102 B and 102 C adjusts the correction parameter according to the cost value CV.
- step 308 when the cost value CV is smaller than the predetermined threshold value, the image enhancement device 1 is operated in the operation mode such that each of the correction modules 102 A, 102 B and 102 C ceases to adjust the correction parameter.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
Description
- This application claims priority to U.S. provisional Application Ser. No. 62/410,407 filed Oct. 20, 2016, which is herein incorporated by reference.
- The present disclosure relates to an image-processing technology. More particularly, the present disclosure relates to an image enhancement device and image enhancement method for a convolutional network apparatus.
- Image segmentation is a challenging problem in computer graphics, particularly with respect to digitally separating a desired object (foreground) from a background of the digital image. Such separation is useful during digital editing of still images and video. In some approaches, deep learning model-based segmentation method is used to perform segmentation. However, the segmentation result of such a method is often affected by image scale, lighting and location changes.
- Accordingly, what is needed is an image enhancement device and image enhancement method for a convolutional network apparatus to address the issues mentioned above.
- An aspect of the present disclosure is to provide an image enhancement device that includes a down-sampling module, correction modules and an up-sampling module is provided. The down-sampling module is configured to down-sample an input image to generate down-sampled images having different down-sampled resolutions. Each of the correction modules is coupled to the down-sampling module, and is configured to perform correction on one of the down-sampled images according to at least one correction model based on at least one correction parameter to generate one of corrected images. The up-sampling module is coupled to the correction modules, and is configured to up-sample the corrected images to generate up-sampled images, wherein each of the up-sampled images is of a same up-sampled resolution. The concatenating module is coupled to the up-sampling module, and concatenates the up-sampled images into an output image.
- Another aspect of the present disclosure is to provide a convolutional network apparatus that includes operation layers and an image enhancement device. The operation layers are operated in series and are configured to perform convolution on an under-processed input image to generate a processed output image. The image enhancement device includes a down-sampling module, correction modules and an up-sampling module is provided. The down-sampling module is configured to down-sample an input image to generate down-sampled images having different down-sampled resolutions, wherein the input image is either received from a first one of the operation layers or is either the under-processed input image. Each of the correction modules is coupled to the down-sampling module, and is configured to perform correction on one of the down-sampled images according to at least one correction model based on at least one correction parameter to generate one of corrected images. The up-sampling module is coupled to the correction modules, and is configured to up-sample the corrected images to generate up-sampled images, wherein each of the up-sampled images is of a same up-sampled resolution. The concatenating module is coupled to the up-sampling module, and concatenates the up-sampled images into an output image.
- Yet another aspect of the present disclosure is to provide an image enhancement method used in an image enhancement device that includes the steps outlined below. An input image is down-sampled to generate a plurality of down-sampled images. Correction is performed on the down-sampled images each according to at least one correction model based on at least one correction parameter to generate a plurality of corrected images. The corrected images are up-sampled to generate a plurality of up-sampled images, wherein each of the up-sampled images is of a same resolution. The up-sampled images are concatenated into an output image.
- These and other features, aspects, and advantages of the present invention will become better understood with reference to the following description and appended claims.
- It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the invention as claimed.
- The invention can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:
-
FIG. 1 is a diagram illustrating a convolutional network apparatus in an embodiment of the present disclosure; -
FIG. 2 is a block diagram of the image enhancement device in an embodiment of the present invention; and -
FIG. 3 is an image enhancement method in an embodiment of the present invention. - Reference will now be made in detail to the present embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
- It will be understood that, in the description herein and throughout the claims that follow, when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present. Moreover, “electrically connect” or “connect” can further refer to the interoperation or interaction between two or more elements.
- It will be understood that, in the description herein and throughout the claims that follow, although the terms “first,” “second,” etc. may be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the embodiments.
- It will be understood that, in the description herein and throughout the claims that follow, the terms “comprise” or “comprising,” “include” or “including,” “have” or “having,” “contain” or “containing” and the like used herein are to be understood to be open-ended, i.e., to mean including but not limited to.
- It will be understood that, in the description herein and throughout the claims that follow, the phrase “and/or” includes any and all combinations of one or more of the associated listed items.
- It will be understood that, in the description herein and throughout the claims that follow, words indicating direction used in the description of the following embodiments, such as “above,” “below,” “left,” “right,” “front” and “back,” are directions as they relate to the accompanying drawings. Therefore, such words indicating direction are used for illustration and do not limit the present disclosure.
- It will be understood that, in the description herein and throughout the claims that follow, unless otherwise defined, all terms (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
- Any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. § 112(f). In particular, the use of “step of” in the claims herein is not intended to invoke the provisions of 35 U.S.C. § 112(f).
- Reference is made to
FIG. 1 .FIG. 1 is a diagram illustrating aconvolutional network apparatus 1 in an embodiment of the present disclosure. - The
convolutional network apparatus 1 includes operation layers L1, L2, L3, . . . , Ln and animage enhancement device 10. The operation layers L1, L2, L3, . . . , Ln may include such as, but not limited to a convolution layer, a pooling layer, a dropout layer, a batch normalization layer, a dilation convolution layer, an active function layer or a combination thereof. - In an embodiment, the
convolutional network apparatus 1 is a fully convolutional neural network for performing image segmentation. The operation layers L1, L2, L3, . . . , Ln included therein are operated in series along the direction A and are configured to perform convolution on an under-processed input image Fin to generate a processed output image Fout. The processed output image Fout may include such as, but not limited to at least one segmented feature (not illustrated). - More specifically, as illustrated in
FIG. 1 , each of the operation layers L1, L2, L3, . . . , Ln is illustrated as a cubic and is operated to perform convolution on an input image processed by a previous operation layer to generate an output image to a subsequent operation layer, except that the first operation layer L1 performs convolution on the under-processed input image Fin and the last operation layer Ln generates the processed output image Fout. - For example, the operation layer L2 performs convolution on an image (not illustrated) processed by the operation layer L1 and generates a processed image (not illustrated) to the operation layer L3.
- The
image enhancement device 10 is disposed either before the operation layer L1 or between any two of the operation layers, e.g. the operation layers L2 and L3. - In the present embodiment, the
image enhancement device 10 is disposed before the operation layer L1 theimage enhancement device 10 is configured to perform image enhancement on the under-processed input image Fin such that the operation layer L1 substantially receives the enhanced result, e.g. an enhanced image Fin′ illustrated inFIG. 1 . - As a result, the operation layers disposed after the
image enhancement device 10 can perform convolution by using an enhanced image. Theconvolutional network apparatus 1 can accomplish a better segmentation result. - It is appreciated that the embodiment of the
image enhancement device 10 disposed before the operation layer L1 is merely an example. In another embodiment, theimage enhancement device 10 can be disposed between such as, but not limited to the operation layers L2 and L3 to perform image enhancement on the image finished processing by the operation layer L2 such that the operation layer L3 substantially receives the enhanced result. Moreover, in an embodiment, theconvolutional network apparatus 1 may include more than one image enhancement devices each disposed between different pair of operation layers. - The detail of the operation of the
image enhancement device 10 is described in the following paragraphs. - Reference is now made to
FIG. 2 .FIG. 2 is a block diagram of theimage enhancement device 10 in an embodiment of the present invention. Theimage enhancement device 10 includes a down-sampling module 100,correction modules sampling module 104, aconcatenating module 106 and acomparison module 108. - In some embodiments, the down-
sampling module 100, thecorrection modules sampling module 104, the concatenatingmodule 106 and thecomparison module 108 can be realized by, for example, one or more processors, such as central processors and/or microprocessors, but are not limited in this regard. - In some embodiments, the down-
sampling module 100, thecorrection modules sampling module 104, the concatenatingmodule 106 and thecomparison module 108 include one or more memory devices, each of which includes, or a plurality of which collectively include a computer readable storage medium. The computer readable storage medium may include a read-only memory (ROM), a flash memory, a floppy disk, a hard disk, an optical disc, a flash disk, a flash drive, a tape, a database accessible from a network, and/or any storage medium with the same functionality that can be contemplated by persons of ordinary skill in the art to which this disclosure pertains. - In some embodiments, the down-
sampling module 100, thecorrection modules sampling module 104, the concatenatingmodule 106 and thecomparison module 108 may run or execute various software programs and/or sets of instructions stored in memory (not illustrated) to perform various functions for theimage enhancement device 10 and to process data. - The down-
sampling module 100 is configured to down-sample an input image, such as but not limited to the under-processed input image Fin, to generate down-sampled images D1, D2 and D3 having different down-sampled resolutions. - For example, when the under-processed input image Fin has a resolution of 200×200, the resolutions of the down-sampled images D1, D2 and D3 can be such as, but not limited to 200×200 (down-sampled by 1), 100×100 (down-sampled by ½) and 50×50 (down-sampled by ¼) respectively.
- Each of the
correction modules sampling module 100, and is configured to perform correction on one of the down-sampled images D1, D2 and D3 according to at least one correction model based on at least one correction parameter to generate one of corrected images C1, C2 and C3. In different embodiments, the correction model can be any appropriate mathematics model used to perform image correction. - In the present embodiment, each of the
correction modules - The blur correction model BL is configured to perform blur correction on an image, e.g. the under-processed input image Fin, according to at least one correction parameter related to the blur correction operation. The blur region in the image processed by the blur correction model BL would become clear when the at least one correction parameter is well-selected.
- The color correction model CO is configured to perform color correction on an image, e.g. the under-processed input image Fin, according to at least one correction parameter related to the color correction operation. The region having false colors in the image processed by the color correction model CO would be corrected when the at least one correction parameter is well-selected.
- The luminous correction model LU is configured to perform luminous correction on an image, e.g. the under-processed input image Fin, according to at least one correction parameter related to the luminous correction operation. The region having low brightness in the image processed by the luminous correction model LU would become brighter when the at least one correction parameter is well-selected.
- The up-
sampling module 104 is coupled to thecorrection modules - The
concatenating module 106 is coupled to the up-sampling module 104, and is configured to concatenate the up-sampled images U1, U2 and U3 into an output image. In an embodiment, the output image generated by the concatenatingmodule 106 is the image Fin′. - The
comparison module 108 is configured to calculate a cost value CV based on a cost function according to a difference between a processed image processed by at least one operation layer subsequent to theimage enhancement device 10 and a predetermined image P corresponding to the under-processed input image Fin. - In an embodiment, the processed image is the processed output image Fout. In other embodiments, the processed image can be the image processed by either the operation layer L1, L2, L3, . . . or Ln.
- In an embodiment, the
comparison module 108 may calculate the cost value CV based on the cost function according to a difference between the output image Fout and the predetermined image P. In practice, the output image is such as, but not limited to a feature map, and the predetermined image P is such as, but not limited to a Ground Truth. - In an embodiment, the predetermined image P is different when different images processed by one of the operation layer L1, L2, L3, . . . or Ln is used as the processed image.
- In an embodiment, under a training mode, each of the
correction modules - For example, the under-processed input image Fin can be a blurred version of the predetermined image P and can be processed by the down-
sampling module 100, thecorrection modules sampling module 104 and theconcatenating module 106 as described above. Thecomparison module 108 compares the predetermined image P and the output image, i.e. the image Fin′, to generate at least one cost value CV such that each of thecorrection modules - In an embodiment, under the training mode, the
correction modules image enhancement device 1 is operated in the operation mode and each of thecorrection modules - More specifically, in an embodiment, the under-processed input image Fin can be processed by the modules mentioned above again, in which each of the
correction modules comparison module 108 compares the predetermined image P and the newly generated output image to generate the cost value CV. - When the cost value CV is still not smaller than a predetermined threshold value, each of the
correction modules - In an embodiment, under an operation mode, each of the
correction modules correction modules concatenating module 106 generates the image Fin′ based on the correction performed according to the correction parameter. - As a result, the
image enhancement device 10 in the present invention is able to perform image enhancement based on the learning result to perform such as, but not limited blur, color and luminous correction. Moreover, theconvolutional network apparatus 1 in the present invention is able to use the enhanced image to perform convolution to obtain a better image segmentation result. - It is appreciated that in different embodiments, the operation layers of the
convolutional network apparatus 1 and the modules in theimage enhancement device 10 can be implemented by either hardware components or software modules. - It is also appreciated that in another embodiment, a validation accuracy is calculated to determine whether the training mode is finished. In an embodiment, the validation accuracy is calculated based on such as, but not limited to a training set and a validation set of input images. When the validation accuracy is higher than a predetermined value, the training mode is finished and the
image enhancement device 10 is further operated in the operation mode. Further, in other embodiments, a multiple of indexes, such as, but not limited to the cost value CV, the validation accuracy, a training set error, a validation error or a combination thereof can be used to determine whether the training mode is finished. -
FIG. 3 is animage enhancement method 300 in an embodiment of the present invention. Theimage enhancement method 300 can be used in theimage enhancement device 10 illustrated inFIG. 2 , or be implemented by using other hardware components such as a database, a common processor, a computer, a server, other unique hardware devices that have a specific logic circuit or an equipment having a specific function, e.g. a unique hardware integrated by a computer program and a processor or a chip. - More specifically, the
image enhancement method 300 is implemented by using a computer program having computer readable instructions to control the modules in theimage enhancement device 10. The instructions can be stored in a memory, in which the memory can be a non-transitory computer readable medium such as a ROM (read-only memory), a flash memory, a floppy disc, a hard disc, an optical disc, a flash disc, a tape, an database accessible from a network, or any storage medium with the same functionality that can be contemplated by persons of ordinary skill in the art to which this disclosure pertains. - The detail of the
image enhancement method 300 illustrated inFIG. 3 is described in accompany withFIG. 2 . Theimage enhancement method 300 includes the steps outlined below (The steps are not recited in the sequence in which the steps are performed. That is, unless the sequence of the steps is expressly indicated, the sequence of the steps is interchangeable, and all or part of the steps may be simultaneously, partially simultaneously, or sequentially performed). - In
step 301, an input image, e.g. the under-processed input image Fin, is down-sampled by the down-sampling module 100 to generate the down-sampled images D1, D2 and D3. - In
step 302, correction is performed on the down-sampled images D1, D2 and D3 each according to at least one correction model by thecorrection modules - In
step 303, the corrected images C1, C2 and C3 are up-sampled by the up-sampling module 104 to generate the up-sampled images U1, U2 and U3, wherein each of the up-sampled images U1, U2 and U3 is of a same resolution. - In
step 304, the up-sampled images U1, U2 and U3 are concatenated into an output image, e.g. the image Fin′, by the concatenatingmodule 106. - In
step 305, the output image, e.g. the image Fout, is compared with a predetermined image P corresponding to the under-processed input image Fin by thecomparison module 108 to generate at least one cost value CV. - In
step 306, thecorrection modules - In
step 307, when the cost value CV is not smaller than the predetermined threshold value, theimage enhancement device 1 is operated in the training mode such that each of thecorrection modules - In
step 308, when the cost value CV is smaller than the predetermined threshold value, theimage enhancement device 1 is operated in the operation mode such that each of thecorrection modules - Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.
- It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.
Claims (19)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/716,490 US10402943B2 (en) | 2016-10-20 | 2017-09-26 | Image enhancement device and method for convolutional network apparatus |
TW106133764A TWI632526B (en) | 2016-10-20 | 2017-09-29 | Image enhancement device and method for convolutional network apparatus |
CN201710917175.2A CN107967514B (en) | 2016-10-20 | 2017-09-30 | Convolution network device and image enhancement device and method thereof |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662410407P | 2016-10-20 | 2016-10-20 | |
US15/716,490 US10402943B2 (en) | 2016-10-20 | 2017-09-26 | Image enhancement device and method for convolutional network apparatus |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180114294A1 true US20180114294A1 (en) | 2018-04-26 |
US10402943B2 US10402943B2 (en) | 2019-09-03 |
Family
ID=61971084
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/716,490 Active 2038-02-27 US10402943B2 (en) | 2016-10-20 | 2017-09-26 | Image enhancement device and method for convolutional network apparatus |
Country Status (3)
Country | Link |
---|---|
US (1) | US10402943B2 (en) |
CN (1) | CN107967514B (en) |
TW (1) | TWI632526B (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112581373A (en) * | 2020-12-14 | 2021-03-30 | 北京理工大学 | Image color correction method based on deep learning |
US20210256657A1 (en) * | 2018-11-08 | 2021-08-19 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US20210272246A1 (en) * | 2018-11-26 | 2021-09-02 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US11164085B2 (en) * | 2019-04-25 | 2021-11-02 | Booz Allen Hamilton Inc. | System and method for training a neural network system |
US20220215505A1 (en) * | 2019-04-09 | 2022-07-07 | Michael Brown | System and method of processing of a captured image to facilitate post-processing modification |
US20230020060A1 (en) * | 2020-12-28 | 2023-01-19 | Fujifilm Corporation | Imaging support apparatus, imaging apparatus, imaging support method, and program |
CN115984705A (en) * | 2023-02-16 | 2023-04-18 | 耕宇牧星(北京)空间科技有限公司 | Convolution network-based large-amplitude remote sensing network image bridge segmentation method |
US11887215B2 (en) | 2018-12-21 | 2024-01-30 | Samsung Electronics Co., Ltd. | Image processing apparatus and method for style transformation |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109472194B (en) * | 2018-09-26 | 2022-02-11 | 重庆邮电大学 | Motor imagery electroencephalogram signal feature identification method based on CBLSTM algorithm model |
TWI694414B (en) * | 2018-12-18 | 2020-05-21 | 國立成功大學 | Analyzing method, computer program product, and electric device using convolutional neural network |
CN110796649B (en) * | 2019-10-29 | 2022-08-30 | 北京市商汤科技开发有限公司 | Target detection method and device, electronic equipment and storage medium |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030222998A1 (en) * | 2000-12-20 | 2003-12-04 | Satoru Yamauchi | Digital still camera system and method |
US8036494B2 (en) * | 2004-04-15 | 2011-10-11 | Hewlett-Packard Development Company, L.P. | Enhancing image resolution |
US8411910B2 (en) | 2008-04-17 | 2013-04-02 | Biometricore, Inc. | Computationally efficient feature extraction and matching iris recognition |
TWI406187B (en) | 2010-08-19 | 2013-08-21 | Univ Nat Cheng Kung | Fast and high quality image/video interpolation method and apparatus |
US9014470B2 (en) * | 2011-08-31 | 2015-04-21 | Adobe Systems Incorporated | Non-rigid dense correspondence |
CN103716630B (en) | 2012-09-29 | 2017-02-22 | 华为技术有限公司 | Upsampling filter generation method and device |
US9336582B1 (en) * | 2015-04-17 | 2016-05-10 | Google Inc. | Convolutional color correction |
US10097805B2 (en) * | 2015-10-13 | 2018-10-09 | Apple Inc. | Multi-image color refinement with application to disparity estimation |
CN106127684B (en) * | 2016-06-22 | 2019-03-15 | 中国科学院自动化研究所 | Image super-resolution Enhancement Method based on forward-backward recutrnce convolutional neural networks |
US9965863B2 (en) * | 2016-08-26 | 2018-05-08 | Elekta, Inc. | System and methods for image segmentation using convolutional neural network |
US10699412B2 (en) * | 2017-03-23 | 2020-06-30 | Petuum Inc. | Structure correcting adversarial network for chest X-rays organ segmentation |
CN107133933B (en) * | 2017-05-10 | 2020-04-28 | 广州海兆印丰信息科技有限公司 | Mammary X-ray image enhancement method based on convolutional neural network |
US9984325B1 (en) * | 2017-10-04 | 2018-05-29 | StradVision, Inc. | Learning method and learning device for improving performance of CNN by using feature upsampling networks, and testing method and testing device using the same |
CN109598722B (en) * | 2018-12-10 | 2020-12-08 | 杭州帝视科技有限公司 | Image analysis method based on recurrent neural network |
-
2017
- 2017-09-26 US US15/716,490 patent/US10402943B2/en active Active
- 2017-09-29 TW TW106133764A patent/TWI632526B/en active
- 2017-09-30 CN CN201710917175.2A patent/CN107967514B/en active Active
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210256657A1 (en) * | 2018-11-08 | 2021-08-19 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US11887280B2 (en) * | 2018-11-08 | 2024-01-30 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US20210272246A1 (en) * | 2018-11-26 | 2021-09-02 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US11741578B2 (en) * | 2018-11-26 | 2023-08-29 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method, system, and computer-readable medium for improving quality of low-light images |
US11887215B2 (en) | 2018-12-21 | 2024-01-30 | Samsung Electronics Co., Ltd. | Image processing apparatus and method for style transformation |
US20220215505A1 (en) * | 2019-04-09 | 2022-07-07 | Michael Brown | System and method of processing of a captured image to facilitate post-processing modification |
US11164085B2 (en) * | 2019-04-25 | 2021-11-02 | Booz Allen Hamilton Inc. | System and method for training a neural network system |
CN112581373A (en) * | 2020-12-14 | 2021-03-30 | 北京理工大学 | Image color correction method based on deep learning |
US20230020060A1 (en) * | 2020-12-28 | 2023-01-19 | Fujifilm Corporation | Imaging support apparatus, imaging apparatus, imaging support method, and program |
CN115984705A (en) * | 2023-02-16 | 2023-04-18 | 耕宇牧星(北京)空间科技有限公司 | Convolution network-based large-amplitude remote sensing network image bridge segmentation method |
Also Published As
Publication number | Publication date |
---|---|
TW201816716A (en) | 2018-05-01 |
TWI632526B (en) | 2018-08-11 |
CN107967514B (en) | 2021-04-27 |
US10402943B2 (en) | 2019-09-03 |
CN107967514A (en) | 2018-04-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10402943B2 (en) | Image enhancement device and method for convolutional network apparatus | |
CN113033537B (en) | Method, apparatus, device, medium and program product for training a model | |
US10810721B2 (en) | Digital image defect identification and correction | |
US20170032222A1 (en) | Cross-trained convolutional neural networks using multimodal images | |
CN112598045A (en) | Method for training neural network, image recognition method and image recognition device | |
CN110008961B (en) | Text real-time identification method, text real-time identification device, computer equipment and storage medium | |
CN110852385B (en) | Image processing method, device, equipment and storage medium | |
CN112489037B (en) | Defect detection and related model training method, electronic equipment and storage device | |
CN111275660A (en) | Defect detection method and device for flat panel display | |
CN113689373B (en) | Image processing method, device, equipment and computer readable storage medium | |
CN110717864B (en) | Image enhancement method, device, terminal equipment and computer readable medium | |
CN113256662B (en) | Pathological section image segmentation method and device, computer equipment and storage medium | |
CN106407991B (en) | Image attribute recognition method and system and related network training method and system | |
CN114998172A (en) | Image processing method and related system | |
CN112990046B (en) | Differential information acquisition method, related device and computer program product | |
CN110544256B (en) | Deep learning image segmentation method and device based on sparse features | |
US10832076B2 (en) | Method and image processing entity for applying a convolutional neural network to an image | |
CN114549348A (en) | Panoramic image denoising method and device, server and readable storage medium | |
CN112288748B (en) | Semantic segmentation network training and image semantic segmentation method and device | |
CN113139463A (en) | Method, apparatus, device, medium and program product for training a model | |
CN116071625B (en) | Training method of deep learning model, target detection method and device | |
CN112087556A (en) | Dark light imaging method and device, readable storage medium and terminal equipment | |
CN117809092B (en) | Medical image processing method and device, electronic equipment and storage medium | |
CN117911908B (en) | Enhancement processing method and system for aerial image of unmanned aerial vehicle | |
CN117456562B (en) | Attitude estimation method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: HTC CORPORATION, TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YANG, HUNG-YI;LIN, CHENG-HSIEN;CHO, PO-CHUAN;SIGNING DATES FROM 20170915 TO 20170918;REEL/FRAME:043735/0026 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |