CN106162112B - Image color processing method, device and terminal device - Google Patents

Image color processing method, device and terminal device Download PDF

Info

Publication number
CN106162112B
CN106162112B CN201610616222.5A CN201610616222A CN106162112B CN 106162112 B CN106162112 B CN 106162112B CN 201610616222 A CN201610616222 A CN 201610616222A CN 106162112 B CN106162112 B CN 106162112B
Authority
CN
China
Prior art keywords
frame image
color
image
pixel
optical filter
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201610616222.5A
Other languages
Chinese (zh)
Other versions
CN106162112A (en
Inventor
李龙佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to CN201610616222.5A priority Critical patent/CN106162112B/en
Publication of CN106162112A publication Critical patent/CN106162112A/en
Application granted granted Critical
Publication of CN106162112B publication Critical patent/CN106162112B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/64Circuits for processing colour signals

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

The application proposes a kind of image color processing method, device and terminal device, and this method is used in the terminal device with shoot function, and MEMS control imaging sensor movement, imaging sensor includes filter unit, and filter unit includes:The first color filter disc, the second color filter disc and the 3rd color filter disc laterally disposed;Including:The first two field picture is shot to preview screen in initial position;Triggering MEMS moves imaging sensor from initial position to preset direction one pixel distance to first position, and the second two field picture is shot in first position;Triggering MEMS moves imaging sensor from first position to the preset direction one pixel distance to the second place, and the 3rd two field picture is shot in the second place;4th two field picture is synthesized according to color component corresponding to each location of pixels in the first two field picture, the second two field picture and the 3rd two field picture.Thus, the degree of accuracy of image color reduction is improved.

Description

Image color processing method and device and terminal equipment
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image color processing method and apparatus, and a terminal device.
Background
Various terminal devices with shooting functions are widely applied to daily life, and the terminal devices need to perform image color reduction processing on pictures to be acquired during shooting so as to acquire shot images with good quality as far as possible.
Since true color of an image means that in each pixel value making up a color image there are R, G, B primary color components, each primary color component directly determining the primary color intensity of the display device to produce color. However, the data format from the image sensor in the prior art is a bayer data format, and each pixel in this format has only one of three color channels, so that each pixel has only one real color component, and other missing color components need to be estimated and obtained by other processing methods, and the estimated color component and the real color component are subjected to synthesis processing.
Therefore, when the current terminal device shoots a picture, the obtained image color and the real picture color have larger difference, and the color reduction degree of the image is not good.
Disclosure of Invention
The present application is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, a first objective of the present application is to provide an image color processing method, which enables more color components in an image pixel to be obtained, and improves the restoration effect of the true color of the image and the image quality.
A second object of the present application is to provide an image color processing apparatus.
A third object of the present application is to provide a terminal device.
In order to achieve the above object, an embodiment of a first aspect of the present application provides an image color processing method, where the method is applied in a terminal device, and an imaging module of the terminal device includes: the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, wherein the optical filter comprises a plurality of optical filtering units, each optical filtering unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the method comprises the following steps:
shooting a first frame image on a preview picture at an initial position;
triggering the micro-electro-mechanical system to move the image sensor to a first position from an initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
triggering the micro-electro-mechanical system to move the image sensor to a second position from the first position to a preset direction by one pixel distance, and shooting a third frame of image at the second position;
and performing synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture.
The image color processing method of the embodiment of the application accurately controls the image sensor to move by utilizing the micro electro mechanical system, the image sensor is provided with the optical filter and the photosensitive pixel array, the optical filter comprises a plurality of optical filter units, the optical filter units form a matrix structure with a row and three columns, the optical filter units comprise a first color filter, a second color filter and a third color filter, a first frame image is firstly shot at an initial position, the micro electro mechanical system with the accurate displacement control function controls the image sensor to move for a pixel distance along a preset direction and then a second frame image is obtained, the micro electro mechanical system controls the image sensor to continuously move for a pixel distance along the preset direction, so that the terminal equipment obtains the first frame image, the second frame image and the third frame image, and the first frame image is obtained according to the color component of each pixel position of each frame image, The second frame image and the third frame image are synthesized to form a fourth frame image, namely a final image, so that more color components in the image pixels can be obtained, and the real color restoration effect and the image quality of the image are improved.
In order to achieve the above object, a second aspect of the present application provides an image color processing apparatus, which is applied in a terminal device having a shooting function,
imaging module among the terminal equipment includes: a micro-electro-mechanical system and an image sensor, wherein,
the MEMS controls the image sensor to move, the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the device comprises:
the first processing module is used for shooting a first frame image on the preview picture at an initial position;
the second processing module is used for triggering the micro-electro-mechanical system to move the image sensor to a first position from the initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
the third processing module is used for triggering the micro electro mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame of image at the second position;
and the synthesis module is used for carrying out synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image so as to generate a fourth frame image of the preview picture.
The image color processing device is applied to a terminal device, an imaging module in the terminal device comprises a micro electro mechanical system and an image sensor, the micro electro mechanical system is connected with the image sensor and controls the image sensor to move, an optical filter and a photosensitive pixel array are arranged on the image sensor, the optical filter comprises a plurality of optical filter units, the optical filter units comprise a first color filter, a second color filter and a third color filter which are arranged in a matrix structure of a row and a column, a first frame of image is shot at an initial position, the micro electro mechanical system with an accurate displacement control function controls the image sensor to move for one pixel distance along a preset direction and then obtain a second frame of image, and then the image sensor is controlled by the micro electro mechanical system to move for one pixel distance along the preset direction, so that the terminal device obtains the first frame of image, The second frame image and the third frame image are synthesized to form a fourth frame image, namely a final image, according to the color component of each pixel position of each frame image, so that more color components in the image pixels can be obtained, and the real color restoration effect and the image quality of the image are improved.
To achieve the above object, a third aspect of the present application provides a terminal device, including: the casing with set up imaging module in the casing, wherein, imaging module includes: a micro-electro-mechanical system, an image sensor, a lens, a memory, and a processor,
the micro-electro-mechanical system controls the movement of the image sensor,
the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, wherein the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the memory is used for storing executable program codes;
the processor performs by reading executable program code stored in the memory:
shooting a first frame image on a preview picture at an initial position;
triggering the micro-electro-mechanical system to move the image sensor to a first position from the initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
triggering the micro-electro-mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame of image at the second position;
and performing synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture.
In the terminal device of the embodiment of the application, a micro electro mechanical system in the terminal device controls the image sensor to move, an optical filter and a photosensitive pixel array are arranged on the image sensor, the optical filter comprises a plurality of optical filter units, the plurality of optical filter units form a matrix structure of a row and three columns, the optical filter units comprise a first color filter, a second color filter and a third color filter which are transversely arranged, a first frame image is firstly shot at an initial position, the micro electro mechanical system with an accurate displacement control function controls the image sensor to move for a pixel distance along a preset direction, then a second frame image is obtained, then the micro electro mechanical system continuously moves for a pixel distance along the preset direction, so that the terminal device obtains the first frame image, the second frame image and the third frame image, and synthesizes the first frame image, the second frame image and the third frame image according to the color component of each pixel position of each frame image to form a fourth frame image, namely the final image, therefore, more color components in the image pixel can be obtained, and the restoration effect of the real color of the image and the image quality are improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic structural diagram of an optical filter in a terminal device applied in an image color processing method according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a MEMS and an image sensor according to an embodiment of the present disclosure;
FIG. 3 is a flow chart of an image color processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of color components obtained by the optical filter in the first frame image;
FIG. 5 is a schematic diagram of color components obtained by the optical filter in the second frame image;
FIG. 6 is a schematic diagram of color components obtained by the optical filter in the third frame of image;
FIG. 7 is a flow chart of an image color processing method according to another embodiment of the present application;
FIG. 8 is a schematic structural diagram of an image color processing apparatus according to an embodiment of the present application;
FIG. 9 is a schematic diagram of an image color processing apparatus according to another embodiment of the present application;
fig. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
An image color processing method, an image color processing device, and a terminal device according to embodiments of the present application are described below with reference to the drawings.
Specifically, the image color processing method provided by the present invention is applied to a terminal device with a shooting function, and it should be noted that the types of the terminal device are many, and include: cell-phones, panel computer, wearable equipment etc..
The terminal equipment with the shooting function comprises an imaging module. The image sensor in the imaging module comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array. The light sensing principle is that a light sensing unit of a light sensing pixel array generates an electric signal by receiving a light signal filtered from an optical filter, and obtains color output through exposure.
The structure of the optical filter determines the distribution of the color components filtered corresponding to the pixel position of each captured image, and different optical filter structures can be selected according to actual application requirements for image capturing. The filter structure applied in the image color processing method provided in this embodiment is shown in fig. 1, and specifically as follows:
fig. 1 is a schematic structural diagram of an optical filter in a terminal device applied in an image color processing method according to an embodiment of the present application; referring to fig. 1, the filter 10 includes a plurality of filter units 11, each filter unit 11 is a matrix unit with a row and three columns, and the matrix unit includes: a first color filter 111, a second color filter 112, and a third color filter 113 disposed in the lateral direction.
It should be noted that the positions of the first color filter, the second color filter, and the third color filter can be arranged according to the application requirement. For example, the arrangement of the color filter in each filter unit in the filter shown in fig. 1 is: the first color filter 111 is a red (R) filter, the second color filter 112 is a green (G) filter, and the third color filter 113 is a blue (B) filter, which are arranged in a 1 × 3 matrix in a horizontal direction in an arrangement of 1 red, 1 green, and 1 blue.
It should be noted that, in the present embodiment, the structural disposition of each filtering unit of the optical filter of the present embodiment may be the same or different, and in the present embodiment, it is preferable that the structural disposition of each filtering unit is the same.
Since each color filter can only obtain one color component, the other two color components need to be estimated through a series of algorithms, for example, the first color filter 111 in fig. 1 is an image pixel position corresponding to a red (R) filter, and only the corresponding red component can be obtained, and the blue component and the green component corresponding to the pixel position need to be estimated; alternatively, for example, the second color filter 112 in fig. 1 is an image pixel position corresponding to a green (G) filter, only the corresponding green component can be obtained, and the red component and the blue component corresponding to the pixel position need to be estimated. Therefore, based on the image captured by the filter adopted in the embodiment, since each pixel position in the whole image needs to be estimated to restore two color components, the color restoration degree of the whole image is not high, and the display effect of the whole image is affected.
In order to solve the above problems, the image color processing method provided by the present invention introduces a micro-electromechanical system, determines a preset moving distance according to the distribution structure of the color filter in the filter unit, and triggers the micro-electromechanical system to control the image sensor to move to a corresponding position according to the preset moving distance to shoot a reference image, thereby obtaining more real color components at the same pixel position from the reference image for synthesis, and improving the color restoration degree of the image. Among them, Micro-Electro-Mechanical systems (MEMS), also called Micro-Electro-Mechanical systems, microsystems, micromachines, etc., are developed based on the Micro-electronic technology (semiconductor manufacturing technology), and incorporate the technologies of lithography, etching, thin film, LIGA, silicon micromachining, non-silicon micromachining, and precision machining to make high-tech electromechanical devices. The operation range is in the micrometer range, the precise displacement control can be realized, the precision is very high, and the pixel level can be achieved (namely, the distance of the MEMS driving the image sensor to move each time can be equivalent to the size of the image sensor pixel).
Fig. 2 is a schematic structural diagram of a micro-electromechanical system (MEMS) and an image sensor according to an embodiment of the present invention, and the following describes a principle that the micro-electromechanical system (MEMS) drives the image sensor to move with reference to fig. 2, specifically, the micro-electromechanical system (MEMS) includes a fixed electrode 21, a movable electrode 22, and a deformable connecting member 23. The movable electrode 22 is engaged with the fixed electrode 21. The connector 23 fixedly connects the fixed electrode 21 and the movable electrode 22. The fixed electrode 21 and the movable electrode 22 are used to generate an electrostatic force by a driving voltage. The link 23 is configured to deform in a direction in which the movable electrode 22 moves under an electrostatic force to allow the movable electrode 22 to move and thus move the image sensor 30.
It should be noted that, according to different application requirements, the corresponding mems is configured to control the image sensor to move in different directions, for example: the micro electro mechanical systems can be respectively arranged in the horizontal direction and the vertical direction of the image sensor, so that the micro electro mechanical systems can drive the image sensor to horizontally move leftwards or rightwards, horizontally move upwards or downwards and the like. The micro-electromechanical system controls the step length of the image sensor in each movement, and the like, and can be calibrated by the system according to a large amount of experimental data, and can be set by a user according to requirements.
FIG. 3 is a flow chart of an image color processing method according to an embodiment of the present application; as shown in fig. 3, the method may specifically include the following steps:
s101, a first frame image is captured on the preview screen at an initial position.
S102, triggering the micro-electro-mechanical system to move the image sensor to a first position in a preset direction by one pixel distance, and shooting a second frame image at the first position.
S103, triggering the micro electro mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame image at the second position.
Specifically, during shooting, a user focuses the terminal device on a shooting object, and shoots a first frame image corresponding to a preview picture after focusing is finished.
Fig. 4 is a schematic diagram of color components obtained by the optical filter in the first frame image, for example:
the pixel position a corresponding to the red filter (R)111 acquires a red component, the pixel position B corresponding to the green filter (G)112 acquires a green component, and the pixel position c corresponding to the blue filter (B)113 acquires a blue component.
Since the color distribution structure in each filter unit in the image sensor related by the invention is a matrix unit with one row and three columns, the matrix unit comprises: a first color filter 111 and a second color filter 112, a third color filter 113 disposed in the lateral direction. Therefore, for the same pixel position, if three color components are to be acquired through different color filters, on the basis of a first color component acquired in a first frame image, the micro electro mechanical system can be triggered to control the image sensor to move a pixel distance from an initial position to a preset direction to a first position, a second frame image is shot at the first position, a second color component corresponding to the pixel position is acquired in the second frame image, then the micro electro mechanical system is triggered again to control the image sensor to continue to move the pixel distance from the first position to the preset direction to a second position, a third frame image is shot at the second position, and a third color component corresponding to the pixel position is acquired in the third frame image.
It should be noted that the preset direction may be set according to actual application requirements. For example: if the preset direction is rightward, triggering the micro electro mechanical system to move the image sensor from the initial position to a first position by one pixel distance in a rightward way, and shooting a second frame image at the first position; and if the preset direction is leftward, triggering the micro electro mechanical system to move the image sensor to the left from the initial position by one pixel distance to a first position, and shooting a second frame image at the first position. Wherein, the left and right directions are column directions of the filter units arranged in a matrix form.
For a more clear description of the above implementation, the following description is given in conjunction with fig. 4 to 6.
FIG. 5 is a schematic diagram of color components obtained by the optical filter in the second frame image;
FIG. 6 is a schematic diagram of color components obtained by the optical filter in the third frame of image;
as shown in fig. 4-6, taking a reference pixel position c (a certain point corresponding to the image of the external scene) selected in the figure, taking the pixel position c as an example, when the initial position is, i.e. the mems does not drive the image sensor to move, the color component obtained by the pixel position through the optical filter is a blue (B) color component; the image sensor is driven by the micro-electro-mechanical system to move a pixel distance in a preset direction (for example, to move to the right), and then is in a first position state as shown in fig. 5, and a color component obtained by the pixel position through the optical filter is a green (G) color component; then, based on the first position state shown in fig. 5, the mems drives the image sensor to move a pixel distance in the preset direction (continuously moving to the right), and then the image sensor is in the second position state shown in fig. 6, and the color component obtained by the pixel position through the optical filter is a red (R) color component. In this way, the three primary color components can be obtained at the pixel position respectively.
Of course, it can be understood that the image sensor moved in this manner may not necessarily obtain three primary color components at each pixel position, for example, taking the pixel position b in fig. 4 to 6 as an example, before the image sensor is moved, as shown in fig. 4, the color component obtained by the pixel position b through the optical filter is a green (G) color component, while after the mems drives the image sensor to move a pixel distance to the first position in the preset direction, as shown in fig. 5, the color component obtained by the pixel position through the optical filter is a red (R) color component, and after the mems drives the image sensor to move the pixel distance to the second position in the preset direction again, as shown in fig. 6, the color component obtained by the pixel position through the optical filter is not available. Thus, the pixel location can only obtain two color components (green and red), while the blue color component is still missing.
In addition, even after the image sensor moves twice in the above manner, only one color component can be obtained at the pixel position at some edges corresponding to the optical filter, for example, taking the pixel position at a in fig. 4 to 6 as an example, before the image sensor moves, as shown in fig. 4, the color component obtained by the pixel position through the optical filter is a red (R) color component, and after the mems drives the image sensor to move a pixel distance to a first position in the preset direction, as shown in fig. 5, the pixel position cannot obtain the color component through the optical filter, and after the mems drives the image sensor to move a pixel distance to a second position in the preset direction again, as shown in fig. 6, the pixel position cannot obtain the color component through the optical filter. Thus, only one color component (red) can be obtained for this pixel location, while the green and blue color components are still missing.
And S104, performing synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture.
Specifically, by combining the corresponding color components at the pixel positions in the preview screen, three color components are obtained at some pixel positions, for example, the pixel position at c shown in fig. 4 to 6 can obtain three color components; while only two color components are obtained at some pixel positions, for example, the pixel position at b shown in fig. 4 to 6 can only obtain two color components; even only one color component is obtained at some pixel positions, e.g. the pixel position at a shown in fig. 4 to 6.
No matter the number of the color components obtained at each pixel position is several, the color components obtained at each pixel position are respectively and correspondingly synthesized, and for the missing color components at each pixel position, one color component may be missing at some positions, two color components may be missing at some positions, and in order to obtain the missing color components, the missing color components can be obtained by the estimation method in the prior art.
Compared with the prior art, the fourth frame image synthesized by adopting the mode does not need to estimate the color components at certain pixel positions, and only needs to estimate the residual color components at partial pixel positions, so that the number of the pixel positions needing to be estimated is greatly reduced, the workload is reduced, the inaccuracy caused by estimation can be reduced as much as possible, and the accuracy and the reality of image color restoration are improved.
The image color restoration method provided by this embodiment includes controlling movement of an image sensor by using a micro-electromechanical system, where the image sensor is provided with a photosensitive pixel array and optical filters, the optical filters include a plurality of optical filter units, the optical filter units include a first color filter, a second color filter and a third color filter, the optical filter units are arranged in a matrix structure with one row and three columns, the micro-electromechanical system (with a pixel-level control accuracy) with a precise displacement control function controls the image sensor to move a pixel distance along a preset direction to obtain a second frame image, and the micro-electromechanical system controls the image sensor to continue moving a pixel distance along the preset direction, so that a terminal device obtains the first frame image, the second frame image and the third frame image, and the first frame image, the second frame image and the third frame image are obtained according to color components at each pixel position of each frame image, The second frame image and the third frame image are synthesized to form a fourth frame image, namely a final image, so that more color components in the image pixels can be obtained, and the real color restoration effect and the image quality of the image are improved.
With the above-described embodiment step 104, since the position of the missing color component among the pixel positions is at the edge position, the images of the pixel positions located at the edge can be cut out, that is, only the color components at the pixel positions having three color components at the same time are synthesized to form the fourth frame image. Specifically, step 104 includes: acquiring a first pixel position simultaneously having a first color component, a second color component and a third color component according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image; and performing synthesis processing on all the color components of the first pixel position to generate a fourth frame image.
In addition, regarding step 104 in the above embodiment, different processing methods may be adopted to estimate the missing color components in the pixel positions according to practical application requirements, for example: in order to more clearly explain the estimation process of missing color components, the color lookup table, the interpolation method, and the like are specifically explained as follows by combining the embodiment shown in fig. 7 and by using the interpolation estimation method:
FIG. 7 is a flow chart of an image color processing method according to another embodiment of the present application; as shown in fig. 7, based on the foregoing embodiment, step 104 specifically includes:
and S1041, acquiring a first pixel position having a first color component, a second color component and a third color component simultaneously, and a second pixel position having a first color component, a second color component and a third color component simultaneously according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image.
Specifically, as can be seen from the descriptions of fig. 4 to 6 in the above embodiments, some pixel positions can obtain three color components, and some pixel positions can obtain only two color components or only one color component. According to whether the first color component, the second color component and the third color component are acquired at the same time, the pixel positions are distinguished into a first pixel position and a second pixel position, so that the two different pixel positions are processed in different modes.
S1042, acquiring color components lacking in all second pixel positions through a preset interpolation algorithm;
and S1043, performing synthesis processing according to the color components of all the second pixel positions and the color components of all the first pixel positions after the interpolation processing to generate a fourth frame image.
The missing color component corresponding to the second pixel position lacking the color component may be obtained by a preset interpolation algorithm, and specifically, the preset interpolation algorithm may be a Nearest pixel interpolation algorithm (Nearest neighbor interpolation), a bilinear interpolation algorithm, a bicubic interpolation algorithm, a fractal algorithm, and the like. Therefore, three color components can be obtained at each pixel position, the first color component, the second color component and the third color component of the first pixel position with the three color components are directly synthesized, and the color components at the pixel positions of the rest color components obtained through an interpolation algorithm are synthesized, so that a fourth frame image with high color reduction degree, namely a final image, can be obtained.
In addition, in the present embodiment, since the number of color components that can be obtained at each pixel position may be one or two or three, for a pixel position where three color components cannot be obtained, if the second pixel position has one color component, the other two color components that are missing from all the second pixel positions are obtained by a preset interpolation algorithm; or, if the second pixel position has two color components, acquiring another color component missing from all the second pixel positions by a preset interpolation algorithm.
The image color restoration method provided by this embodiment includes controlling movement of an image sensor by using a micro electro mechanical system, where the image sensor is provided with a photosensitive pixel array and optical filters, the optical filters include a plurality of optical filter units, the optical filter units include a first color filter, a second color filter and two third color filters, and the optical filter units are arranged in a matrix structure with one row and three columns, and first the micro electro mechanical system (with pixel-level control accuracy) with a precise displacement control function controls the image sensor to move a pixel distance along a preset direction to obtain a second frame image, and then continuously moves a pixel distance along the preset direction to obtain a third frame image, so that a terminal device obtains the first frame image, the second frame image and the third frame image, and the first frame image, the second frame image and the third frame image are obtained according to color components at each pixel position of each frame image, The second frame image and the third frame image are synthesized to form a fourth frame image, namely a final image, so that a plurality of color components of each pixel position can be obtained as far as possible through the three frame images, partial missing color components are estimated through an interpolation method, the estimation efficiency of the interpolation method is high, the estimation accuracy is close to the real color components, and therefore the color restoration accuracy is improved.
In order to implement the above embodiments, the present application also provides an image color processing apparatus.
Fig. 8 is a schematic structural diagram of an image color processing apparatus according to an embodiment of the present application.
The image color processing device is applied to a terminal device with a shooting function, as shown in fig. 2, an imaging module in the terminal device comprises: a micro-electro-mechanical system and an image sensor, wherein,
the micro electro mechanical system controls the image sensor to move, and the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array.
As shown in fig. 1, the optical filter 10 includes a plurality of filter units 11, each filter unit 11 is a matrix unit with a row and three columns, and the matrix unit includes: a first color filter 111, a second color filter 112, and a third color filter 113 disposed in the lateral direction;
as shown in fig. 8, the image color processing apparatus includes:
a first processing module 41, configured to capture a first frame image of the preview screen at an initial position;
a second processing module 42, configured to trigger the mems to move the image sensor from the initial position to a first position in a preset direction by one pixel distance, and capture a second frame image at the first position;
the third processing module 43 is configured to trigger the mems to move the image sensor from the first position to the preset direction by a pixel distance to a second position, and capture a third frame of image at the second position;
and a synthesizing module 44, configured to perform synthesis processing according to the color component corresponding to each pixel position in the preview image acquired through the optical filter in the first frame image, the second frame image, and the third frame image, so as to generate a fourth frame image of the preview image.
In the embodiment, if the preset direction is rightward, triggering the micro electro mechanical system to move the image sensor from the initial position to a first position by a pixel distance in a rightward way, and shooting a second frame image at the first position; and if the preset direction is leftward, triggering the micro electro mechanical system to move the image sensor to the left from the initial position by one pixel distance to a first position, and shooting a second frame image at the first position.
It should be noted that the foregoing explanation of the embodiment of the image color processing method is also applicable to the image color processing apparatus of the embodiment, and is not repeated herein.
The image color processing device of the embodiment of the application is applied to a terminal device, an imaging module in the terminal device comprises a micro electro mechanical system and an image sensor, the micro electro mechanical system is connected with the image sensor, the micro electro mechanical system controls the image sensor to move, an optical filter and a photosensitive pixel array are arranged on the image sensor, the optical filter comprises a plurality of optical filter units, the optical filter units form a matrix structure with a row and a column, the optical filter units comprise a first color filter, a second color filter and a third color filter, a first frame image is firstly shot at an initial position, the micro electro mechanical system with an accurate displacement control function controls the image sensor to move for a pixel distance along a preset direction and then a second frame image is obtained, and then the micro electro mechanical system controls the image sensor to continuously move for a pixel distance along the preset direction, the terminal device obtains the first frame image, the second frame image and the third frame image before moving, and synthesizes the first frame image, the second frame image and the third frame image according to the color component of each pixel position of each frame image to form a fourth frame image, namely a final image.
Since the position of missing color components in each pixel position is at the edge position, the images of the pixel positions at the edge can be cut out, that is, only the color components at the pixel positions having three color components at the same time are synthesized to form the fourth frame image. Specifically, the synthesis module 44 is specifically configured to: acquiring a first pixel position simultaneously having a first color component, a second color component and a third color component according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image; and performing synthesis processing on all the color components of the first pixel position to generate a fourth frame image.
Fig. 9 is a schematic structural diagram of an image color processing apparatus according to another embodiment of the present application.
As shown in fig. 9, based on the embodiment shown in fig. 8, the synthesis module 44 may further specifically include: an acquisition unit 441, a calculation unit 442, and a generation unit 443.
The obtaining unit 441 is configured to: and acquiring a first pixel position simultaneously having a first color component, a second color component and a third color component and a second pixel position not simultaneously having the first color component, the second color component and the third color component according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image.
The calculation unit 442 is configured to: and acquiring the missing color components of all the second pixel positions by a preset interpolation algorithm.
The generating unit 443 is configured to: and performing synthesis processing according to the color components of all the second pixel positions and the color components of all the first pixel positions after the interpolation processing to generate a fourth frame image of the preview picture.
In this embodiment, further, the calculating unit 442 is configured to: if the second pixel position has one color component, acquiring other two color components lacking in all the second pixel positions through a preset interpolation algorithm; or, if the second pixel position has two color components, acquiring another color component missing from all the second pixel positions by a preset interpolation algorithm.
It should be noted that the foregoing explanation of the embodiment of the image color processing method is also applicable to the image color processing apparatus of the embodiment, and is not repeated herein.
In order to implement the above embodiments, the present application further provides a terminal device.
Fig. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application. The terminal apparatus 1000 in the present embodiment may be a mobile phone or the like having a photographing function.
As shown in fig. 10, the terminal device includes: casing and the imaging module assembly 1000 of setting in the casing, wherein, imaging module assembly 1000 includes: a micro-electro-mechanical system 20, an image sensor 30, a lens 1001, a memory 1002, and a processor 1003,
the mems 20 controls the image sensor movement,
the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, wherein the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the memory 1002 is used for storing executable program codes;
the processor 1003 executes by reading executable program code stored in the memory 1002:
shooting a first frame image on a preview picture at an initial position;
triggering the micro-electro-mechanical system 20 to move the image sensor 30 to a first position from the initial position to a preset direction by one pixel distance, and capturing a second frame image at the first position;
triggering the micro-electro-mechanical system 20 to move the image sensor 30 to a second position from the first position to a preset direction by one pixel distance, and taking a third frame image at the second position;
and performing synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture.
It should be noted that the foregoing explanation of the embodiment of the image color processing method is also applicable to the image color processing apparatus of the embodiment, and is not repeated herein.
In the terminal device of the embodiment of the application, a mems in the terminal device controls movement of an image sensor, the image sensor is provided with a photosensitive pixel array and optical filters, the optical filters include a plurality of optical filter units, the optical filter units include a first color filter, a second color filter and a third color filter, the optical filter units are arranged in a matrix structure of one row and three columns, a first frame image is first captured at an initial position, the mems with a precise displacement control function (with a pixel-level control precision) controls the image sensor to move a pixel distance along a preset direction, and then the image sensor continues to move a pixel distance along the preset direction, so that the terminal device obtains the first frame image, the second frame image and the third frame image before moving, and the first frame image, the second frame image and the third frame image are obtained according to a color component of each pixel position of each frame image, The second frame image and the third frame image are synthesized to form a fourth frame image, namely a final image, so that more color components in the image pixels can be obtained, and the real color restoration effect and the image quality of the image are improved.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

Claims (9)

1. An image color processing method is applied to a terminal device with a shooting function, wherein an imaging module in the terminal device comprises: a micro-electro-mechanical system and an image sensor, wherein,
the MEMS controls the image sensor to move, the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the method comprises the following steps:
shooting a first frame image on a preview picture at an initial position;
triggering the micro-electro-mechanical system to move the image sensor to a first position from the initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
triggering the micro-electro-mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame of image at the second position;
synthesizing and processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture;
the synthesizing, according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image, and the third frame image, a fourth frame image is generated, including:
according to color components corresponding to pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image, acquiring first pixel positions simultaneously having a first color component, a second color component and a third color component and second pixel positions not simultaneously having the first color component, the second color component and the third color component;
acquiring color components lacking in all second pixel positions through a preset interpolation algorithm;
and performing synthesis processing according to the color components of all the second pixel positions and the color components of all the first pixel positions after the interpolation processing to generate a fourth frame image of the preview picture.
2. The method of claim 1,
if the preset direction is rightward, triggering the micro electro mechanical system to move the image sensor from the initial position to a first position by one pixel distance in a rightward way, and shooting a second frame image at the first position;
and if the preset direction is leftward, triggering the micro electro mechanical system to move the image sensor to the left from the initial position by one pixel distance to a first position, and shooting a second frame image at the first position.
3. The method according to claim 1, wherein the generating a fourth frame image by performing a synthesis process based on color components corresponding to respective pixel positions in a preview screen acquired through the optical filter in the first frame image, the second frame image, and the third frame image includes:
acquiring a first pixel position simultaneously having a first color component, a second color component and a third color component according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image;
and performing synthesis processing on all the color components of the first pixel position to generate a fourth frame image.
4. The method according to claim 1, wherein the obtaining the missing color components of all the second pixel positions by a preset interpolation algorithm comprises:
if the second pixel position has one color component, acquiring other two color components lacking in all the second pixel positions through a preset interpolation algorithm; or,
and if the second pixel position has two color components, acquiring another color component which is lacked in all the second pixel positions through a preset interpolation algorithm.
5. An image color processing device, characterized in that, the device is applied in a terminal device with a shooting function, an imaging module in the terminal device comprises: a micro-electro-mechanical system and an image sensor, wherein,
the MEMS controls the image sensor to move, the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter and a third color filter disposed in a transverse direction;
the device comprises:
the first processing module is used for shooting a first frame image on the preview picture at an initial position;
the second processing module is used for triggering the micro-electro-mechanical system to move the image sensor to a first position from the initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
the third processing module is used for triggering the micro electro mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame of image at the second position;
the synthesis module is used for carrying out synthesis processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image so as to generate a fourth frame image of the preview picture;
the synthesis module comprises:
an obtaining unit, configured to obtain, according to color components corresponding to pixel positions in the preview picture obtained through the optical filter in the first frame image, the second frame image, and the third frame image, a first pixel position having a first color component, a second color component, and a third color component at the same time, and a second pixel position having a first color component, a second color component, and a third color component at the same time;
the computing unit is used for acquiring color components lacking in all second pixel positions through a preset interpolation algorithm;
and the generating unit is used for performing synthesis processing on the color components of all the second pixel positions and the color components of all the first pixel positions after interpolation processing to generate a fourth frame image of the preview picture.
6. The image color processing apparatus according to claim 5,
if the preset direction is rightward, triggering the micro electro mechanical system to move the image sensor from the initial position to a first position by one pixel distance in a rightward way, and shooting a second frame image at the first position;
and if the preset direction is leftward, triggering the micro electro mechanical system to move the image sensor to the left from the initial position by one pixel distance to a first position, and shooting a second frame image at the first position.
7. The image color processing apparatus of claim 5, wherein the composition module is configured to: acquiring a first pixel position simultaneously having a first color component, a second color component and a third color component according to the color component corresponding to each pixel position in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image;
and performing synthesis processing on all the color components of the first pixel position to generate a fourth frame image.
8. The image color processing apparatus according to claim 5, wherein said calculation unit is configured to:
if the second pixel position has one color component, acquiring other two color components lacking in all the second pixel positions through a preset interpolation algorithm; or,
and if the second pixel position has two color components, acquiring another color component which is lacked in all the second pixel positions through a preset interpolation algorithm.
9. A terminal device, comprising: the casing with set up imaging module in the casing, wherein, imaging module includes: a micro-electro-mechanical system, an image sensor, a lens, a memory, and a processor,
the micro-electro-mechanical system controls the movement of the image sensor,
the image sensor comprises a photosensitive pixel array and an optical filter arranged on the photosensitive pixel array, wherein the optical filter comprises a plurality of optical filter units, each optical filter unit is a matrix unit with a row and three columns, and the matrix unit comprises: a first color filter, a second color filter, and a third color filter disposed laterally;
the memory is used for storing executable program codes;
the processor performs by reading executable program code stored in the memory:
shooting a first frame image on a preview picture at an initial position;
triggering the micro-electro-mechanical system to move the image sensor to a first position from the initial position to a preset direction by one pixel distance, and shooting a second frame image at the first position;
triggering the micro-electro-mechanical system to move the image sensor to a second position from the first position to the preset direction by one pixel distance, and shooting a third frame of image at the second position;
synthesizing and processing according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image to generate a fourth frame image of the preview picture;
the synthesizing, according to the color components corresponding to the pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image, and the third frame image, a fourth frame image is generated, including:
according to color components corresponding to pixel positions in the preview picture acquired through the optical filter in the first frame image, the second frame image and the third frame image, acquiring first pixel positions simultaneously having a first color component, a second color component and a third color component and second pixel positions not simultaneously having the first color component, the second color component and the third color component;
acquiring color components lacking in all second pixel positions through a preset interpolation algorithm;
and performing synthesis processing according to the color components of all the second pixel positions and the color components of all the first pixel positions after the interpolation processing to generate a fourth frame image of the preview picture.
CN201610616222.5A 2016-07-29 2016-07-29 Image color processing method, device and terminal device Expired - Fee Related CN106162112B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610616222.5A CN106162112B (en) 2016-07-29 2016-07-29 Image color processing method, device and terminal device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610616222.5A CN106162112B (en) 2016-07-29 2016-07-29 Image color processing method, device and terminal device

Publications (2)

Publication Number Publication Date
CN106162112A CN106162112A (en) 2016-11-23
CN106162112B true CN106162112B (en) 2017-11-10

Family

ID=57327715

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610616222.5A Expired - Fee Related CN106162112B (en) 2016-07-29 2016-07-29 Image color processing method, device and terminal device

Country Status (1)

Country Link
CN (1) CN106162112B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106507080B (en) 2016-11-29 2018-07-17 广东欧珀移动通信有限公司 Control method, control device and electronic device
CN107147886A (en) * 2017-05-31 2017-09-08 广东欧珀移动通信有限公司 Imaging sensor, imaging device and imaging method
CN114125234A (en) * 2021-11-26 2022-03-01 信利光电股份有限公司 Sensor displacement-based multi-frame shooting method and device
CN115278002B (en) * 2022-06-24 2024-11-01 维沃移动通信有限公司 Image sensor, camera module, shooting method, shooting device and electronic equipment

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102162978A (en) * 2010-02-24 2011-08-24 银河港(北京)技术有限公司 Method and device for upgrading common charge coupled device (CCD) camera device into multi-waveband CCD camera device
CN102316259B (en) * 2011-09-23 2013-08-21 深圳市君曼科技发展有限公司 Lens module for monitoring camera
CN202907105U (en) * 2012-10-15 2013-04-24 光宝电子(广州)有限公司 Imaging system
CN105635381A (en) * 2014-10-28 2016-06-01 天津欧本创意科技有限公司 Convenient-to-edit type external scanning device for handset
CN105338229A (en) * 2015-11-18 2016-02-17 努比亚技术有限公司 Camera module

Also Published As

Publication number Publication date
CN106162112A (en) 2016-11-23

Similar Documents

Publication Publication Date Title
CN106210677B (en) Image color processing method, device and terminal device
CN106210678A (en) Image color processing method, device and terminal unit
CN106162112B (en) Image color processing method, device and terminal device
JP5226600B2 (en) Image deformation apparatus and operation control method thereof
US10129479B2 (en) Photographing apparatus and photographing method with optical low-pass filter effect
CN107404619B (en) image zooming processing method and device and terminal equipment
CN107483839B (en) Multi-frame image synthesis method and device
KR20100039430A (en) Image processor, image processing method, digital camera, and imaging apparatus
US11445118B2 (en) Image pickup apparatus, an image processing method and a non-transitory computer-readable medium
CN106162111B (en) Image color processing method and terminal device
CN107076959B (en) Controlling a focus lens assembly
JP5906427B2 (en) Imaging device, image processing device
JP2008054062A (en) Image pickup device
CN106303221A (en) control method, device and mobile terminal
CN106162110A (en) Image color processing method, device and terminal unit
JP2001221945A (en) Automatic focusing device
JP2020008785A (en) Imaging apparatus
JP2008053787A (en) Multiple-lens electronic camera and parallax correcting method of multi-lens electronic camera
CN108322654B (en) Lens zooming method and device and mobile terminal
JP6597080B2 (en) Imaging device
JP2014236244A (en) Image processing system, image processing program, and digital camera
JP2008064847A (en) Image blur correction controller, control method, imaging apparatus and imaging method
WO2014136703A1 (en) Image-capturing device and image display method
WO2010131724A1 (en) Digital camera
JP2015103879A (en) Image processing apparatus and control method thereof

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder

Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18

Patentee after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd.

Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18

Patentee before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd.

CP01 Change in the name or title of a patent holder
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20171110

CF01 Termination of patent right due to non-payment of annual fee