JP2014102596A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a suitable cartoon-liked image by performing relatively simple processing of an optional subject image.SOLUTION: A screen tone cut-out image G10 is generated by cutting out a screen tone G0 by a mask G16 obtained by making a gray scale image G15 of a colored subject image G1 a monochrome binary image with a first threshold Th1, a monochrome binary image G11 is generated from the gray scale image with a second threshold Th2 (<Th1), a contour image G12 is generated by detecting and inverting edges of the image G15, and a cartoon-liked image G13 is generated by compositing the images G10, G11, and G12. A main portion of the cartoon-like image is extracted and pasted to a background image, and a partial transparent foreground image is pasted.

Description

  The present invention relates to an image processing apparatus that converts a still image or a moving image captured by a camera into a cartoon-like image offline or online in real time.

  Patent Document 1 below discloses an apparatus that outputs an image such as a comic.

JP 2001-222713 A

  However, it does not convert a cartoon-like paper image into an electronic image, and does not convert an image captured by a camera into a cartoon-like image. When trying to convert a camera image to comics, it is necessary to process any subject image under various conditions, so in order to convert it to an appropriate comic image, processing according to the type of subject must be performed. Therefore, the software configuration becomes complicated.

  In view of such problems, an object of the present invention is to provide an image processing apparatus capable of generating an appropriate comic image by performing a relatively simple process on an arbitrary subject image. .

According to a first aspect of the present invention, a processor, a storage unit storing a program and data, and an image output unit are provided, and the data includes a subject image captured by a camera and a screen tone, and the program In the image processing apparatus that causes the processor to convert the subject image into a cartoon tone, the program
(A) converting the subject image into a grayscale image;
(B) generating a mask in which the grayscale image is binarized with a first threshold, generating a screen tone cut-out image composed of pixels of the screen tone corresponding to the positions of black pixels of the mask,
(C) binarizing the grayscale image with a second threshold smaller than the first threshold to generate a binary image;
(D) generating a contour image obtained by inverting the edge image of the grayscale image;
(E) generating a first synthesized image obtained by product synthesis of the screen tone cut-out image, the binary image, and the contour line image;
An image including a part or all of the first composite image is supplied to the image output means as a comic image.

In the second aspect of the image processing apparatus according to the present invention, in the first aspect,
The data further includes a background image,
The program further provides the processor with
(F) Pixels including pixels on a predetermined pair of sides on the image frame in the first pixel connection region divided by the thickened contour line by thickening the contour line of the contour image The area excluding the connected area is obtained as the second connected area, and each pixel in the area excluding the second connected area from which the number of pixels is less than or equal to the threshold is set as one of the two values, and the remaining area To generate a mask with the other of the two values as
(G) a pixel corresponding to the position of the other value of the binary values among the pixels in the mask of the background image, and the binary value of the pixels in the mask of the first composite image. Generating a second composite image in which the pixel corresponding to the position of the other value is combined;
The cartoon image includes part or all of the second composite image.

In the third aspect of the image processing apparatus according to the present invention, in the second aspect,
The data further includes a foreground image with an alpha channel;
The program further provides the processor with
(H) generating a third composite image obtained by masking the second composite image with the mask of the alpha channel;
The cartoon image includes a part or all of the third composite image.

  According to the configuration of the first aspect, a relatively simple process (a) to (e) is performed on an arbitrary subject image, so that an appropriately cropped screen tone cut-out image, a binary image of the subject image, and an outline are obtained. There is an effect that a relatively appropriate cartoon-like image obtained by product synthesis with a line image can be automatically generated at a relatively high speed. This

  According to the configuration of the second aspect, only the object image having a relatively large area of the main part of the subject image can be synthesized with the pre-stored cartoon-like background image, and at the periphery of the subject image. Even if there is a contour line that is connected to the main part of the subject image, it is possible to synthesize the outlines from the subject, or remove details from the subject before taking the image, regardless of the type of subject. There is an effect that a more appropriate cartoon-like image can be generated without manually editing and removing the image of the object. As a result, it is possible to convert a still image or a moving image captured by the camera into an appropriate cartoon-like image offline or online in real time.

  According to the configuration of the third aspect, it is possible to further synthesize a cartoon-like foreground image stored in advance, so that a more appropriate cartoon-like image can be generated.

  Other objects, characteristic configurations and effects of the present invention will become apparent from the following description read in connection with the appended claims and the drawings.

It is a schematic block diagram which shows the hardware constitutions of the image processing apparatus which concerns on Example 1 of this invention. It is a functional block diagram which shows the structure of the process which converts the to-be-photographed image imaged with the camera into a comic image. It is a schematic flowchart which shows the process by the main process part in FIG. It is a figure which shows the more detailed structure of the comic tone main image generation part in FIG. 2, and the kind of the input-output image. FIG. 3 is a schematic functional block diagram showing a more detailed configuration of a mask generation unit 42 in FIG. 2 and types of input / output images thereof. (A)-(D) is a figure which shows the specific example of a screen tone, a to-be-photographed image, a background image, and a foreground image, respectively. (A)-(D) are figures which show the image for process description of FIG. (A)-(D) are figures which show the image for process description of FIG. (A)-(D) are figures which show the image for process description of FIG. (A) And (B) is a figure which shows the image for process description of FIG. 5, (C) is a figure which shows the specific example of the background and main image synthesized image in FIG. It is a figure which shows the specific example of the final comic style image produced | generated by the image processing of the present Example 1. FIG. It is a figure which shows the specific example of the other final comic style image produced | generated by the image processing of the present Example 1. FIG. It is a functional block diagram which shows the structure of the process in the case of the moving image corresponding to FIG.

  FIG. 1 is a schematic block diagram showing a hardware configuration of an image processing apparatus 10 according to an embodiment of the present invention, and shows only components necessary for this embodiment. The image processing apparatus 10 is, for example, a mobile terminal device such as a smartphone or a PDA, a digital camera, a notebook computer, or a desktop personal computer.

  In the main body 20 of the image processing apparatus 10, a processor 21 is coupled to a storage device 23, an input interface 24, a camera interface 25, and a display interface 26 via a bus 22. An input device 30 is coupled to the input interface 24, a camera 31 is coupled to the camera interface 25, a display device 32 is coupled to the display interface 26, and an antenna 33 is coupled to the communication unit 27.

  The input device 30 is an interactive input device and includes a touch panel, a pointing device, a keyboard, or a combination thereof. The communication unit 27 includes a network interface for coupling with the Internet via radio waves.

  A program and data are stored in the storage device 23, and this program accepts a user's selection, setting value input, or instruction from the input device 30 via the input interface 24 to the processor 21, and the camera 31 according to this input. The brightness of the input image is adjusted by adjusting the aperture value, the shutter speed, the input gain in the main body 20, etc., the subject is imaged by the camera 31, and the image data of the still image or moving image is stored in the storage device 23. The image data is processed and processed, and the result is supplied to the display interface 26 to be displayed on the display device 32, or the result is MPEG-compressed and supplied to the communication unit 27 via network software such as a browser. As a result, the image data is transmitted to the communication partner via the antenna 33 and a server (not shown).

  FIG. 2 is a functional block diagram mainly showing processing for converting the subject image G1 captured by the camera 31 into a cartoon-like image G40 as processing of the image data.

  The data storage area 230 is a part of the data area of the storage device 23 of FIG. 1, and stores a screen tone G0, a plurality of background images G2, and a plurality of foreground images G3 in advance. The user operates the input device 30 to select the background image G2s from the plurality of background images G2, and selects the foreground image G3s from the plurality of foreground images G3.

  The main processing unit 40 is a main routine for converting the subject image G1 captured by the camera 31 into a comic image. FIG. 3 is a schematic flowchart showing processing by the main processing unit 40, which is processed in the order of steps S1 to S4. The subroutine called in each of steps S1 to S4 corresponds to the comic-like main image generation unit 41, the mask generation unit 42, the background / main image synthesis unit 43, and the foreground synthesis unit 44 in FIG.

  6A to 6D show specific examples of the screen tone G0, the subject image G1, the background image G2s, and the foreground image G3s, respectively.

  The screen tone G0 is a binary image in which a white line and a black line (line width is thicker in the white line) with an inclination of 45 degrees are alternately drawn and cut out to obtain an intermediate luminance surface (intermediate density surface) of the subject image. ).

  The subject image G1 includes a winning cup as a main part in the center, an image processing apparatus 10-yen coin in the lower right, and an image of a part of the object in the upper left corner.

  Since the image of a comic is generally simplified, it is preferable to delete an object having a relatively small area or an image of an edge. It is also preferable to make the background and foreground cartoon.

  Therefore, the subject image G1 is converted into a monochrome binary image, the subject image G1 is cut out and pasted into the intermediate luminance area, the main part in the screen is extracted, the background is replaced with the background image G2s, and the foreground image G3 is further synthesized. The subject image G1 is converted into a cartoon-like image G40.

  Returning to FIG. 2, the cartoon-tone main image generation unit 41 generates a screen tone cut-out image G10 obtained by cutting the screen tone G0 according to the subject image G1, and generates a binary image G11 obtained by binarizing the subject image G1. Then, an edge of the subject image G1 is detected to generate a contour image G12, and the screen tone cutout image G10, the binary image G11, and the contour image G12 are combined to generate a cartoon-tone main image G13. Specific examples of the screen tone cutout image G10, the binary image G11, the contour line image G12, and the cartoon-tone main image G13 are shown in FIGS. 7D, 8A, 8C, and 8D, respectively. ).

  FIG. 4 shows a more detailed configuration of the cartoon-tone main image generation unit 41 and the types of input / output images thereof.

  The resizing unit 410 enlarges or reduces the screen tone G0, the subject image G1, the background image G2s, and the foreground image G3s to be processed as preprocessing, thereby obtaining the respective sizes (number of pixels of width and height). Is matched with the size of the comic-like image G40 to be output as a result. If the size of the subject image G1 is fixed in this application and the screen tone G0, the background image G2, and the foreground image G3 are provided in accordance with these sizes, the processing of the resizing unit 410 can be omitted. Hereinafter, it is assumed that the screen tone G0, the subject image G1, the background image G2s, and the foreground image G3s have the same size.

  The gray scale conversion unit 411 converts the color subject image G1 to gray scale, and generates a gray scale image G15 as shown in FIG. This is obtained, for example, by setting the average value of the component values of R (red), G (green), and B (blue) of each pixel of the subject image G1 as the pixel value of the monochrome image. A pixel of the grayscale image G15 is represented by, for example, 8 bits of a single channel.

  The binarization unit 412 generates a screen tone clipping mask G16 by binarizing and inverting the grayscale image G15 with the first threshold Th1. FIG. 7B is an image obtained by binarizing the grayscale image G15, and FIG. 7C shows a mask G16 obtained by inverting the black and white. The binary value is the minimum value (black) or the maximum value (white). For example, in the case of 8 bits, it is 0 (black) or 255 (white), and in the case of 1 bit, it is 0 or 1.

  The first threshold value Th1 is for generating a mask for cutting out the screen tone G0, and is a variable threshold value, for example, a value obtained by adding a positive set value to an average value of local 3 × 3 pixels including the target pixel. (A threshold value for adaptive binarization), or a value obtained by multiplying the average value of all pixels by a constant larger than 1 or a predetermined experience value.

  The cutout unit 413 cuts out the screen tone G0 with the mask G16 and generates a screen tone cutout image G10 as shown in FIG. Note that the inversion process may be omitted by cutting out the screen tone G0 by regarding the black pixels in FIG. 7B as transmissive pixels. Further, the generation of FIG. 7B may be omitted by inversion at the time of binarization.

  The binarization unit 414 binarizes the grayscale image G15 with a second threshold Th2 smaller than the first threshold Th1, for example, the average value in adaptive binarization, and is shown in FIG. Such a binary image G11 is generated. The second threshold Th2 may be changed according to the operation of the input device 30 by the user. For example, the user selects one of three levels “bright”, “normal”, and “dark”, and in accordance with the result, a value obtained by multiplying the average value by a predetermined constant is set as the second threshold Th2. And

  The edge detection reversing unit 415 performs a filtering process on the grayscale image G15 while raster scanning the edge detection operator, for example, and binarizes the result, whereby a binary edge as shown in FIG. 8B is obtained. An outline image G12 as shown in FIG. 8C is generated by generating an image and inverting the black and white. Note that the edge detection inversion unit 415 may directly generate the contour image G12 by performing inversion binarization while detecting the edge.

  The product synthesizing unit 416 sets a white pixel if all of the pixels corresponding to each other in the screen tone cutout image G10, the binary image G11, and the contour image G12 are white pixels, and otherwise sets the pixels as black pixels. In other words, a cartoon-like main image G13 as shown in FIG. 8D is generated with the product of the corresponding three pixel values as the pixel.

  By such processing, it is possible to generate a cartoon-like monochrome image obtained by simplifying the subject image.

  Further, although the high-quality cartoon-like main image G13 is obtained, the processing is relatively simple, and thus high-speed processing is possible. Therefore, even if the image processing apparatus 10 is a smartphone, the processing performance is improved. Accordingly, it is possible to convert to a cartoon-like image in real time every frame or every few frames of the moving image.

  Returning to FIG. 2, the mask generation unit 42, as shown in FIG. 10B, based on the contour image G <b> 12, a background / main image composition mask for cutting out the main part from the main image and pasting it on the background image G20 is generated.

  FIG. 5 is a schematic functional block diagram showing a more detailed configuration of the mask generation unit 42 and types of input / output images thereof. The mask generation unit 42 performs processing in the order of blocks 420 to 425.

  The linear expansion unit 420 expands the contour line of the contour image G12 and generates an 8-bit binary image as shown in FIG. The inversion unit 421 inverts this image in black and white to generate an image as shown in FIG. 9B.

  The labeling unit 422 performs labeling on this image. That is, for each pixel connection region in which pixels adjacent to each other in four directions or eight directions with white pixels (the pixel value is the maximum value), a pixel connection region image G21 in which different numbers are assigned to the pixels in the region is displayed. Generate. Since the value of black pixels that are not subject to labeling (pixels on the contour line) is 0, integers of 1 or more are used for labeling. FIG. 9C is an explanatory diagram of the pixel connection region image G21, and different pixel connection regions are represented by different gray values.

  The coarse mask generation unit 423 changes a pixel connection area including pixels on a predetermined opposite side (edge) on the pixel connection area image G21 to a black pixel area (non-transmission area), and the remaining area. A mask in which is changed to a white pixel region (transmission region) is generated. For example, the pixels in the pixel connection region having the same label as the start and end pixel labels of each row of the pixel connection region image G21 are converted into black pixels, and all other regions are converted into white pixel regions. However, the pixels on the contour line excluded from the labeling by the labeling unit 422 are white pixels. This is to leave the outline when it is connected to the main part in the image (see FIG. 12 described later). By such processing, a binary image as shown in FIG. 9D is obtained.

  The labeling unit 424 performs the labeling again on the image of FIG. FIG. 10A is an explanatory diagram of a labeled image, in which different pixel connection areas are indicated by areas having different gray values. When this labeling is performed, the number of pixels in each pixel connection region is obtained as the area of the region.

  The small area white region blackening unit 425 converts pixels in a region where the area of the pixel connection region in the labeled image is equal to or less than a threshold value, for example, 10% or less of the total number of pixels, to black pixels, A labeled region other than is converted into a white pixel, and a binary background / main image synthesis mask G20 as shown in FIG. 10B is generated. As a result, a mask can be obtained in which only a relatively large object area in the main part of the screen is a white pixel (transmission area).

  Since there is processing in the small area white region blackening unit 425, the coarse mask generation unit 423 performs a part of the edge on the pixel connection region image G21, for example, the start point and end point pixels of one row every 10 rows. A configuration in which pixels in a pixel connection area having the same label as the label are converted into black pixels and all other areas are converted into white pixel areas may be adopted.

  Returning to FIG. 2, the background / main image composition unit 43 extracts only the pixels on the cartoon-tone main image G13 corresponding to the position of the white pixel of the background / main image composition mask G20 and extracts the background / main image composition mask G20. By extracting only the pixels on the background image G2s corresponding to the position of the black pixel, a background / main image composite image G30 as shown in FIG. Alternatively, the background / main image as shown in FIG. 10 (C) is overwritten on the background image G2s by overwriting the pixel on the cartoon-tone main image G13 corresponding to the position of the white pixel of the background / main image composition mask G20. A composite image G30 is generated.

  Here, each of the foreground images G3 is an image with an alpha channel and includes an alpha mask. For example, in the foreground image G3s of FIG. 6D, the alpha value of each pixel in the character region of the cartoon-like white character string “Zuqu” is 0 (the minimum value is completely opaque), and the alpha value of the pixels in the other regions is 255 (maximum value is completely transparent). The foreground image G3s may be substantially provided with an alpha channel, and may be a set of separate foreground images and masks.

  The foreground composition unit 44 keeps the pixel of the corresponding position in the foreground image G3s if the alpha value is 0 for each pixel of the background / main image composition image G30, and keeps the foreground image G3s if the alpha value is 255. A comic image G40 is generated by overwriting the luminance value of the pixel on the background / main image composite image G30. As a result, a final comic image G40 as shown in FIG. 11 is generated.

  In step S5 of FIG. 3, the cartoon-like image G40 of FIG. 2 is supplied to the display interface 26 so that the cartoon-like image G40 is displayed on the display device 32. At this time, the single channel monochrome binary image is converted into the RGB channel monochrome binary image by the software or the display interface 26.

  FIG. 12 shows a cartoon tone image G40A generated by the same processing as described above with the screen tone G0, the background image G2s and the foreground image G3s being the same as those described above, the subject image being changed.

  Although the case where the present invention is applied to a still image has been described with reference to FIG. 2, the present invention can be applied to a moving image as will be apparent from the above description, and this will be described below as a second embodiment.

  FIG. 13 is a schematic functional block diagram in the case of a moving image (moving image used in a videophone or chat) corresponding to FIG.

  In this configuration, the background image G2 and the foreground image G3 in FIG. 2 and the configuration related thereto are omitted, and the field images sequentially supplied from the camera 31 are sequentially extracted as the subject image G1 in real time while being stored in the data storage area 230. The process is performed in the same manner as described above to generate the cartoon tone main image G13 as a comic tone image. As a field image buffer area, it is sufficient to secure at least two field images. In the case of a videophone or chat with three or more participants, the resize unit 410 reduces the screen tone G0 in advance in order to deliver the images of all participants to each participant in real time via the server. The image G1 is obtained by thinning and sampling the image from the camera 31 to obtain the resized subject image G1.

  In response to an instruction from the input device 30 by the user, this comic-like image G13 is supplied to the display interface 26 of FIG. 1 to display a monitor image on the display device 32, or the comic-like image G13 is MPEG-compressed. Supply to the decompression unit 50, MPEG compression, and supply to the communication unit 27 via the Web client 51 (general-purpose or dedicated Web browser).

  Accordingly, the user can transmit and display his / her comic-like image to a specific or a plurality of communication partners via the server without showing a real face.

  Similarly, through the server, the communication unit 27, and the Web client 51, an MPEG-compressed comic-like image including a plurality of communication partners specified or synthesized by the server is received, and this is compressed by the MPEG compression / decompression unit 50. The comic image is decompressed and reproduced to be supplied to the display interface 26 in FIG. 1, thereby causing the display device 32 to display the received image.

  The MPEG compression / decompression unit 50 includes a graphic chip or graphic board (not shown in FIG. 1), or a part of the multi-core processor 21 and software.

  The preferred embodiments of the present invention have been described above. However, the present invention includes various modifications, and other combinations of the components described in the above-described embodiments, and the functions of the components. Other configurations that use other configurations, and other configurations that would be conceived by those skilled in the art from these configurations or functions are also included in the present invention. For example, the present invention includes the following modifications.

  In the second embodiment, the foreground image G3s in FIG. 2 may be combined with the background / main image composite image G30 in the same manner as in the first embodiment, and a handle name or the like may be displayed on the foreground image G3s. Further, the background image G2s may be combined with the background / main image combination mask G20 and the background image G2s may be changed in accordance with an instruction from the input device 30 in order to add a change as in the first embodiment.

  Further, the present invention may be configured to convert a moving image into a cartoon tone by converting each frame image of a captured moving image into a cartoon tone offline.

  Further, an existing color image may be used as the background image G2s or the foreground image G3s, and when applied to the present invention, this may be converted into a monochrome image and used. The background image G2s or the foreground image G3s may be used as it is or after being edited. The alpha value of the foreground image is not limited to binary, and may be multivalued. The background / main image composition mask G20 is not limited to a binary mask, and a predetermined peripheral portion in the image may be a multi-value mask having a lower transparency as it is farther from the central portion.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 Main-body part 21 Processor 22 Bus 23 Storage device 230 Data storage area 24 Input interface 25 Camera interface 26 Display interface 27 Communication part 30 Input device 31 Camera 32 Display apparatus 33 Antenna 40 Main processing part 41 Manga tone main image generation Unit 42 mask generation unit 43 background / main image synthesis unit 44 foreground synthesis unit 410 resize unit 411 gray scale unit 412, 414 binarization unit 413 clipping unit 415 edge detection inversion unit 416 product synthesis unit 420 linear expansion unit 421 inversion unit 422, 424 Labeling unit 423 Coarse mask generation unit 425 Small area white area blackening unit 50 MPEG compression / decompression unit 51 Web client G0 Screen tone G1 Subject image G2, G2s Background image G3, G 3s foreground image G10 Screen tone clipping image
G11 Binary image G12 Outline image G13 Cartoon-tone main image G15 Grayscale image G16 Mask G20 Background / main image composite mask G21 Pixel connected region image G30 Background / main image composite image G40 Cartoon-tone image

Claims (5)

A processor, storage means for storing the program and data, and image output means, the data including a subject image captured by a camera and a screen tone, and the program sends the subject to the processor In an image processing apparatus for converting an image into a cartoon tone, the program
(A) converting the subject image into a grayscale image;
(B) generating a mask in which the grayscale image is binarized with a first threshold, generating a screen tone cut-out image composed of pixels of the screen tone corresponding to positions of white pixels of the mask,
(C) binarizing the grayscale image with a second threshold smaller than the first threshold to generate a binary image;
(D) generating a contour image obtained by inverting the edge image of the grayscale image;
(E) generating a first synthesized image obtained by product synthesis of the screen tone cut-out image, the binary image, and the contour line image;
An image processing apparatus, wherein an image including a part or all of the first composite image is supplied to the image output means as a comic image. The data further includes a background image,
The program further provides the processor with
(F) Pixels including pixels on a predetermined pair of sides on the image frame in the first pixel connection region divided by the thickened contour line by thickening the contour line of the contour image The area excluding the connected area is obtained as the second connected area, and each pixel in the area excluding the second connected area from which the number of pixels is less than or equal to the threshold is set as one of the two values, and the remaining area To generate a mask with the other of the two values as
(G) a pixel corresponding to the position of the other value of the binary values among the pixels in the mask of the background image, and the binary value of the pixels in the mask of the first composite image. Generating a second composite image in which the pixel corresponding to the position of the other value is combined;
The image processing apparatus according to claim 1, wherein the cartoon image includes a part or all of the second composite image. The data further includes a foreground image with an alpha channel;
The program further provides the processor with
(H) generating a third composite image obtained by masking the second composite image with the mask of the alpha channel;
The image processing apparatus according to claim 2, wherein the cartoon image includes a part or all of a third composite image. An image processing apparatus according to any one of claims 1 to 3,
A camera that captures the subject image;
The mobile terminal device is characterized in that the subject image is a still image or a frame image of a moving image, and the output means is an image display means. An image processing apparatus according to claim 1;
A camera that captures the subject image;
The image processing device converts each of the time-series images constituting the moving image captured by the camera into the comic image as the subject image, sequentially supplies the comic image to the output unit, and The portable terminal device, wherein the output means is a display means or a communication means.