JP2011002875A - Plotting support device, plotting support method, and plotting support program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device that supports drawing enabling, even a person who does not have sufficient level of drawing techniques or sufficient aptitude for painting, to readily perform high-level graphic drawing.SOLUTION: The drawing support device is configured to input the rough shape of a graphic plotted on a display picture by a user, and to extract an object similar to the shape of the graphic plotted by a user from the database of an image or an object, and to arrange the extracted object at a position corresponding to the graphic plotted by the user, and to combine it with a background image as a figure to complete and output a single image without imparting one a sense of incongruity.

Description

  The present invention relates to a drawing support technique using a computer that can easily and highly realistically draw even a person who does not have an advanced drawing technique and does not have a “picture heart”.

  The desire to “draw well” is universal. However, this is not easy. In particular, drawing highly realistic pictures requires high skills and can only be achieved by a limited number of people.

  Currently, the act of drawing on a PC (Personal Computer) is naturally performed. Against this background, there is widespread use of drawing support software that runs on PCs. As a device having a simple function, there is “paint” preinstalled in Windows (registered trademark) which is an OS (Operating System), so that drawing can be performed using a pointing device such as a mouse. . As a more advanced one, “Illustrator (registered trademark)” by Adobe is famous.

  Such software has a wide variety of drawing support functions. For example, a simple figure such as “Circle” or “Ellipse” can be drawn easily, or a specific area can be “painted”. There are even advanced layers that are difficult to handle, such as “layers” that overlay drawings, “deformations” that bend figures, or “image filters” that freely control the tone and contrast of images.

  Further, in the following Non-Patent Document 1, a sensitivity word is associated with a drawing feature (a shape, a color, and a composition) prepared in advance, and the drawing information desired by the user is simply expressed and input as a sensitivity word. The technique and the user interface for presenting the associated drawing are disclosed.

  Speaking of digital images, not only pictures but also photographs are recognized as being representative. There is also a platform for publishing and sharing such paintings and photographs. For example, in an image sharing site on the Web called “Flickr (registered trademark)” (http://www.flickr.com), it is said that nearly 2 million images are being uploaded every day and grows. It forms a “super huge image database”. Many of the images published in “Flickr” are photographs. It is no exaggeration to say that the “super huge image database” contains images of all objects in the world.

  In addition, an image sharing site represented by “Flickr” is also introducing a system called “Creative Commons”. "Creative Commons" is a standard for declaring that image publishers partially release their copyrights. A typical one is one that permits modification, and an image that is permitted to be used can be used as a material, for example, by cutting out a part of it and diverting it. In this way, the foundation for realizing a new creation, such as creating a new image by selecting an appropriate one from a digital image such as a picture taken by another person or a picture drawn, is already in place. It's getting on.

Junko Ichino, Shunichi Tano, “User Interface to Support Design Drawing”, IEICE Transactions D-II, Vol. J82-D-II, No. 10, pp. 1693-1709, 1999.

  Representing the aforementioned “Illustrator”, in order to make full use of the advanced functions provided in many drawing support softwares, the user still needs advanced technology and “picture heart”. In other words, it is difficult for users without advanced technology and “picture mind” to effectively use these functions, and as a result, nobody could easily draw “highly realistic pictures”. .

  In addition, in the technique described in Non-Patent Document 1, it is possible to create a certain design drawing only by inputting a sensitivity word. However, the sensitivity word is associated with the shape, color, and composition of the figure constituting the design in advance. I had to leave. In addition, the final design is a simple pattern prepared in advance, and it was not possible to create an arbitrary drawing desired by the user. From the above two points, the barrier for use was very high.

  The present invention has been made in view of the above-mentioned problems, and the object of the present invention is to easily realize a highly realistic image even for a person who does not have advanced drawing technology and does not have sufficient “picture centroid”. It is to provide a means capable of performing realistic drawing.

  In order to solve the above-mentioned problem, the present invention searches for similar existing image content based on the rough shape of a graphic drawn by the user, and applies it to the graphic portion drawn by the user to compose a single picture. Is the main feature. Existing elemental techniques can be used for searching for similar images. As described above, the present invention includes a processing unit that applies a similar image to a graphic portion drawn by the user and composes it into a single picture, so that the user can easily draw a realistic picture simply by drawing a rough shape. Will be able to.

  More particularly, the present invention relates to a drawing support apparatus that supports drawing of a figure by a computer, and uses the object database storing image objects and the outline of the figure drawn by the user as a query to the object database. An object query unit that performs a calculation, a similarity calculation unit that calculates a similarity between the query and one or more objects in the object database, and a graphic drawn by the user on the calculated object with a high similarity The figure is created by associating it with the corresponding position of the outline or the specified position of the figure, and the machining part to be processed and the outline of the figure drawn by the user are input from the input / output device, and the calculated similarity Input / output control unit that outputs a high-quality object or a figure synthesized by the processing unit to an input / output device Characterized in that it comprises a.

  In the above invention, the similarity calculation unit calculates a similarity based on at least one of a shape similarity and a color information similarity between the query and one or more objects in the object database. It is characterized by calculating.

  Further, the invention is characterized in that the image processing apparatus further comprises an object extraction unit that divides and extracts an image in an image database storing an image as an object extraction target for each object and stores the image in the object database.

  Further, in the above invention, the object extraction unit is further based on at least one of the area of the object extracted from the image or the degree of saliency calculated by a predetermined method for estimating the ease of drawing the eye in the image. , Identifying an unnecessary object and deleting the unnecessary object.

  In the above invention, the object extraction unit further determines an object name based on text information around the image data, and the input / output control unit calculates the similarity calculated by the similarity calculation unit, and The object to be output is selected based on the degree of coincidence between the keyword input by the user and the names of one or more objects in the object database.

  In the drawing support program according to the present invention, the processing of the drawing support apparatus is described as a computer-readable program.

  According to the present invention, a user simply draws an outline of an object to be drawn as a figure, and extracts objects similar to the outline from images such as pictures and photographs stored in an actual image database. However, it is possible to realize a drawing support device that can easily draw highly realistic pictures by pasting and processing at the position of the corresponding figure. As a result, even a person who does not have an “art” can easily perform highly realistic drawing.

It is a figure explaining the outline | summary of the drawing assistance by this invention. It is a figure which shows an example of the basic composition of a drawing assistance apparatus. It is a figure which shows an example of the outline extraction of an object. It is a figure which shows an example of a drawing assistance process. It is a figure which shows an example of a drawing screen. It is a figure which shows an example of a background extraction object image. It is a figure which shows an example of a background extraction result.

  FIG. 1 is a diagram for explaining the outline of the drawing support according to the present invention. Prior to describing the embodiment of the present invention, an example of drawing using the present invention will be briefly described with reference to FIG.

  First, as shown in FIG. 1A, the user writes a draft of a picture that the user wants to draw on the display screen. As shown in Fig. 1 (B), the drawing support device draws the image written by the user using the sketch image search system (Query-by-Sketch) from the image database such as Flickr or the object database storing the image object. Extract and search for objects close to the shape. In this search, color information used by the user, a keyword specified by the user, or the like may be used in combination.

  Next, as shown in FIG. 1C, the drawing support apparatus arranges and displays the searched object at a position corresponding to the portion where the user has drawn the draft. The user can repeat the above operation or change the position of the object as necessary.

  After that, as shown in FIG. 1D, the drawing support apparatus finishes the image into a single image with no sense of incongruity by the user's background designation or Poisson blending, and outputs the result.

  Hereinafter, an embodiment of the present invention that realizes the above drawing will be described in detail. FIG. 2 shows an example of the basic configuration of the drawing support apparatus according to the embodiment of the present invention.

  In this system, a server 10 and a PC (personal computer) 20 which is a terminal used by a user are connected via a communication network 21.

  The server 10 includes a CPU, a memory, an external storage device, other hardware, a software program for drawing support, and the like. , A similarity calculation unit 15, an input / output control unit 16, and a processing unit 17.

  Note that the drawing support apparatus according to the present invention is not limited to the configuration shown in FIG. 2, and for example, the image database 11 or the object database 12 is connected via the communication network 21. Alternatively, the server 10 may not be provided, and the PC 20 may have a processing function of each unit in the server 10 illustrated in FIG. Further, some of the processing functions in the server 10 (for example, the object inquiry unit 14 and the processing unit 17) may be installed in the PC 20.

  It is assumed that image data is stored in the image database 11. This image data may be, for example, an image published on an image sharing site or a database in which image data collected from such a site is collected. When the image data is collected as in the latter case, it is preferable to collect the image data regularly, for example, a fixed number of times a day. In addition, if the collection is made only for images that are permitted to be modified by Creative Commons, the technology can be used without infringing the copyright.

  Next, an example of basic processing executed by the server 10 will be described. The basic processing of this apparatus is to extract an object, which is an image element (drawing object), from the image database 11 and store it in the object database 12, and implement drawing support when the user actually draws. There are two types of drawing support processing.

[Object extraction processing]
First, an example of the former object extraction process will be described in detail. This process is a process that must be performed once before using the embodiment of the present invention. In principle, it may be performed once. However, in order to keep the object stored in the object database 12 always new, it is preferable to perform it regularly, for example, a certain number of times a day.

  In this process, the object extraction unit 13 extracts an object from the image data stored in the image database 11 and stores it in the object database 12.

  Details of processing performed by the object extraction unit 13 will be described. First, in order to cut out the outline of an object stored in an image in the image database 11, a region division method is applied. For example, it is preferable to apply the method based on the graph cut described in [Reference Document 1], because an image region can be divided with high accuracy.

[Reference 1] J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 8, pp. 888-905, 2000.
An example of outline extraction of an object will be described using an example of a landscape photograph 301 shown in FIG. An object is extracted from a landscape photograph 301 in which a house is photographed as shown in FIG. There are various objects in the photograph 301 such as a house 302, a mountain 303, a sun 304, and a cloud 305. If the method of [Reference 1] is applied to such a photograph 301, the similarity of a specific region and the dissimilarity with other regions can be taken into account. As described above, only the outline portion of the house 302 can be separated and extracted. Similarly, other objects can be extracted. As in this example, the object extraction unit 13 stores each extracted object in the object database 12.

  At this time, if necessary, the color information of each object may be stored together. The object color information may be stored as a vector value such as a color histogram of RGB, Lab, or the like. Alternatively, the object may be divided into several partial areas and a color histogram for each area may be stored.

  In addition, a name label may be assigned to the object. To get this, we apply the image annotation method. As the image annotation method, for example, the method described in [Reference Document 2] or [Reference Document 3] below may be applied.

[Reference 2] J. Jeon, V. Lavrenko, and R. Manmatha, Automatic Image Annotation and Retrieval using Cross-Media Relevance Models, Proceedings on ACM SIGIR Conference, pp.119-126, 2003.
[Reference 3] Automatic Image Annotation by Ensemble of Visual Descriptors, Proceedings on IEEE Conference on Computer Vision and Pattern Recognition, pp.17-22, 2007.
Through the above procedure, for example, a name for an object such as “house” can be given to the house 302 in FIG.

[Drawing support processing]
Next, the drawing support process will be described in detail. FIG. 4 shows an example of a drawing support process performed by the drawing support apparatus according to this embodiment.

  In step S401, the input / output control unit 16 presents a drawing screen to the PC 20. The drawing screen presented in step S401 is a screen 501 as shown in FIG. 5, for example, and the display area of this screen is a drawing area 502, an object information presentation area 503, an inquiry input / output area 504, and a completed drawing presentation area 505. Consists of. Note that these display areas may be provided within one screen, or may be provided separately on a plurality of screens.

  In step S <b> 402, the user who uses the PC 20 draws an outline of an object that he / she wants to draw in the drawing area 502 of the screen 501. The input device used for drawing may be a pointing device such as a mouse. In the example of FIG. 5, a house outline 506 and a mountain outline 507 are drawn. The outline of the object drawn by the user in the drawing area 502 is input to the input / output control unit 16 via the communication network 21.

  In step S403, the object inquiry unit 14 extracts the outline of the object drawn by the user from the input information and uses it as a query to make an inquiry to the object database 12. The query is a query sentence described in the search language of the object database 12 or key information for the search.

  In step S404, the similarity calculation unit 15 calculates the similarity between the query in step S403 and the outline of the object stored in the object database 12, and outputs an object having a high value.

  Here, an example of processing details of steps S403 to S404 will be described. The processing performed in steps S403 to S404 is processing for searching for an object suitable for this outline using a user's drawing (sketch) as a query. As a technique for realizing such processing, for example, a known method such as [Reference Document 4] or [Reference Document 5] can be applied.

[Reference 4] ADBimbo and P.Pala, Visual Image Retrieval by Elastic Matching of User Sketches, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.19, No.2, pp.121-132, 1997.
[Reference 5] S.Berretti, ADBimbo, and P.Pala, Retrieval by Shape Similarity with Perceptual Distance and Effective Indexing, IEEE Transactions on Multimedia, Vol.2, No.4, pp.225-239, 2000.
When searching for an object based only on approximate similarity, the search accuracy may be low. In order to improve the accuracy in such a case, the following processing may be performed.

  For example, the similarity may be calculated including the color information of the image. The user shall draw with color when drawing. This color information is compared with the color information of the object to be searched, and the closer the information is, the higher the similarity is. As the similarity, for example, a normalized mean square value can be used. If the object color information is extracted in a certain format in advance during the object extraction process described above, the color information of the user's drawing can also be set in the same format to speed up the search.

  In the object extraction process described above, when an object name is given, a search may be performed using a keyword query. For example, when searching for a house object, it is possible to search for a more accurate object by presenting a figure with a high degree of similarity in the shape of the figure drawn by the user within the range of objects with a name that hits the keyword “house”. The result can be obtained. The keyword “house” is input from the inquiry input / output area 504.

  Further, here, a background that does not apply to the above characteristic objects (house, mountain, etc.) may be extracted and inserted. An example of this process will be described.

  As an example of this, there is a method of using an image containing a background video in the image database 11. For example, when the user desires a background of “sky”, the user inputs an inquiry input / output area 504 as a search query and makes an inquiry to the image database 11. In the image database 11, an image having a tag and peripheral text matching “empty” is searched, and the result is output to the query input / output area 504. The user selects an appropriate one from the displayed results, and the selected image is drawn as a background in the completed drawing presentation area 505.

  At this time, the image selected as the background may partially include unnecessary objects not intended by the user. Therefore, processing for removing such unnecessary objects is performed. For example, in the example of FIG. 6, the background is “sky” in the image 601, but a building 602 is stored as an unnecessary object.

  As a method for removing the unnecessary object, for example, the user may designate an unnecessary area in the drawing area 502, and the designated area may be removed.

  For example, an object can be automatically deleted. Here, an example of a method for automatically determining the object area to be deleted will be described. It is assumed that “the background occupies a larger proportion (area) of the entire image than the unnecessary object”. Based on this assumption, for example, after dividing an image into regions by the method described in [Reference 1], the area (number of pixels) of each region is obtained, and a region with a small area is regarded as an unnecessary object and deleted. .

  At this time, it is necessary to determine which region is considered to have a small area. For example, one region having the largest area may be used as a background and other regions may be deleted. Alternatively, deletion may be performed until the total area area to be deleted reaches a certain ratio (for example, 20%) of the area of the entire image.

  Further, by assuming that “the object to be deleted is a foreground object that is more prominent than the background”, a processing method based on a criterion different from the above can be realized. From this assumption, it is possible to delete noting the area but focusing on the “object that is easy to catch”. As a method for estimating the ease of drawing an eye in an image, for example, there is a method described in [Reference Document 6] below.

[Reference 6] L. Itti, C. Koch, and E. Niebur, A Model Of Saliency-based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.20, No.11, pp. 1254-1259, 1998.
In this method, the degree of saliency in the image can be calculated in units of pixels. At this time, the image is divided into regions by the method described in [Reference 1], and those having a relatively high average saliency included in each region are regarded as unnecessary objects and deleted. Through the above processing, for example, an image 701 in which only the background shown in FIG. 7 is extracted from the image of FIG. 6 can be obtained.

  In step S405, the processing unit 17 resizes the output object so as to fit the size of the outline drawn by the user.

  In step S406, the input / output control unit 16 places the resized object at the drawing position of the user and presents it on the screen of the PC 20.

  The processes in steps S401 to S406 are repeated until a predetermined stop condition is satisfied. The stop condition may be, for example, “a user operating the PC 20 issues a stop command through the drawing screen”, or when a certain time has elapsed since the drawing was started, or the number of object inquiries is set to a certain number. It may be the case when it reaches.

  When the stop condition is satisfied, in step S407, the processing unit 17 processes the image drawn on the drawing screen, and finishes and outputs the image without any sense of incongruity.

  For this processing, a known method such as the Poisson image composition method described in [Reference 7] can be used.

[Reference 7] P. Perez, M. Gangnet, and A. Blake, Poisson Image Editing, ACM Transactions on Graphics, Vol. 22, No. 3, pp.313-318, 2003.
Further, after the smoothing is completed, a filter that processes the photograph like a painting may be applied. Such a filter has already been realized in Adobe Photoshop (registered trademark) or the like.

  Finally, the input / output control unit 16 outputs the completed image to the completed drawing presentation area 505.

  The above is the description of the drawing support apparatus in an example of the embodiment of the present invention. It goes without saying that the processing process executed by the drawing support apparatus can be described as a computer-readable program.

  The drawing support apparatus according to the exemplary embodiment of the present invention has been described in detail above. As described above, in an example of an embodiment of the present invention, an object that the user wants to draw is simply drawn as a figure, and an object similar to the outline is displayed in a picture, a photograph, or the like stored in the image database. Anyone can easily draw a highly realistic picture by extracting from the image, pasting it at the position of the corresponding figure, and processing it.

  This provides a solution to the fact that the prior art required advanced technology and “picture-centre” for the user, and that only simple pattern drawing prepared in advance could be supported. .

  As mentioned above, although an example of the embodiment of the present invention has been described, the present invention is not limited to the example of the described embodiment, and various modifications can be made within the technical scope described in the claims. It is.

DESCRIPTION OF SYMBOLS 10 Server 11 Image database 12 Object database 13 Object extraction part 14 Object inquiry part 15 Similarity calculation part 16 Input / output control part 17 Processing part 20 PC
21 Communication network

Claims (10)

A drawing support device for supporting drawing of a figure by a computer,
An object database containing image objects;
An object inquiry unit for making an inquiry to the object database, using the outline of the figure drawn by the user as a query;
A similarity calculator for calculating the similarity between the query and one or more objects in the object database;
A processing unit that synthesizes and processes the calculated high similarity object as a figure in association with a corresponding position or a designated position of the outline of the figure drawn by the user;
An input / output control unit that inputs an outline of the figure drawn by the user from an input / output device, and outputs the calculated high similarity object or a figure synthesized by the processing unit to the input / output device; A drawing support apparatus characterized by the above. The drawing support apparatus according to claim 1,
The similarity calculation unit calculates a similarity based on at least one of a shape similarity and a color information similarity between the query and one or more objects in the object database. Drawing support device. The drawing support apparatus according to claim 1 or 2,
A drawing support apparatus comprising: an object extracting unit that divides and extracts an image in an image database in which an object extraction target image is stored for each object and stores the image in the object database. The drawing support apparatus according to claim 3,
The object extraction unit further identifies an unnecessary object based on at least one of the area of the object extracted from the image and the degree of saliency calculated by a predetermined method for estimating the ease of drawing the eye in the image. , A drawing support device characterized by deleting the unnecessary object. The drawing support apparatus according to claim 3 or 4, wherein
The object extraction unit further determines an object name based on text information around the image data;
The input / output control unit outputs the similarity based on the similarity calculated by the similarity calculation unit and the degree of coincidence between the keyword input by the user and the names of one or more objects in the object database. A drawing support apparatus characterized by selecting an object. A drawing support method executed by a drawing support device for supporting drawing of a figure by a computer,
A process of inputting an outline of a figure drawn by a user from an input / output device;
An object inquiry process for making an inquiry to an object database in which an object of an image is stored, using the outline of the figure drawn by the user as a query;
A similarity calculation process for calculating a similarity between the query and one or more objects in the object database;
Outputting the calculated high similarity object to an input / output device;
When a predetermined stop condition is satisfied, the output one or more objects are combined as a figure in correspondence with a corresponding position or a designated position of the outline of the figure drawn by the user, and processed Machining process,
A drawing support method comprising: a step of outputting the figure processed by the processing step to an input / output device. The drawing support method according to claim 6, comprising:
In the similarity calculation step, similarity is calculated based on at least one of shape similarity and color information similarity between the query and one or more objects in the object database. Drawing support method to do. A drawing support method according to claim 6 or 7, wherein
A drawing support method, comprising: an object extraction step of dividing an image in an image database storing an object extraction object image for each object and storing the divided image in the object database. The drawing support method according to claim 8, comprising:
In the object extraction process, unnecessary objects are further identified based on at least one of the area of the object extracted from the image and the degree of saliency calculated by a predetermined method for estimating the ease of drawing the eye in the image. A drawing support method characterized by deleting the unnecessary object. The object database, the object inquiry unit, the similarity calculation unit, the processing unit, and the input / output control unit provided in the drawing support apparatus according to any one of claims 1 to 5, or further, As an object extractor,
A drawing support program to make a computer function.

Images