JP2014085723A - Information providing device and information providing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a virtual object to be applied to be easily viewed when providing information by applying the virtual object to an image in a real space and displaying the image.SOLUTION: An information providing device includes: image acquisition means for acquiring an image in a real space; virtual object storage means for storing a virtual object to be displayed in association with an object in the image; component conversion means for dividing a subject included in the image acquired by the image acquisition means into a plurality of objects; specification means for specifying a specific object as an object to be displayed in association with the virtual object among the plurality of objects; non-emphasis means for performing image processing of making inconspicuous at least any of the objects other than the specific object among the plurality of objects; virtual object insertion means for acquiring a virtual object corresponding to the specific object from the virtual object storage means, and for inserting the virtual object in association with the specific object; and display means for displaying the image into which the virtual object has been inserted.

Description

  The present invention relates to an information providing technique for adding a virtual object to an image in real space for display.

  In recent years, a product that takes in a scenery outside a vehicle as a video by combining a car navigation and a vehicle-mounted camera and expands this video to give a virtual object has begun to appear (Non-Patent Document 1). This product provides an intuitive route guide for the driver by adding an arrow pointing to the guide route and a logo indicating the type of surrounding building to the image acquired from the in-vehicle camera. However, when the number of virtual objects to be displayed increases, there is a problem that the display screen becomes cluttered and difficult to understand.

  On the other hand, in Patent Document 1, when a virtual object is added to a 3D image captured from a camera of a mobile phone and displayed, the virtual object is displayed when the number of displayed objects increases. By shifting the display position and grouping, the problem of difficulty in seeing is avoided.

  Moreover, in patent document 2, the number of the virtual objects given to the vehicle-mounted camera and displayed is adjusted according to the speed of the vehicle. As a result, an easy-to-understand display can be provided.

JP 2010-238098 A JP 2011-123807 A

"Carrozzeria Cyber Navi", [Online], [Search June 13, 2012], Internet <URL: http://pioneer.jp/carrozzeria/cybernavi/>

  In any of the methods disclosed in Patent Documents 1 and 2, if an object having a shape or color similar to the virtual object to be displayed exists in the 3D image, the screen is still cluttered and difficult to understand. End up. Particularly in urban areas, there are various prominently colored objects, which hinders the display of virtual objects.

  An object of the present invention is to provide a technique for making a virtual object to be provided easy to see when providing and displaying information by adding a virtual object to an image in real space.

An information providing apparatus according to the present invention includes:
Image acquisition means for acquiring an image of real space;
Virtual object storage means for storing a virtual object to be displayed in association with an object in the image;
Componentizing means for dividing a subject included in an image acquired by the image acquiring means into a plurality of objects; a specifying means for specifying a specific object that is an object to be displayed in association with the virtual object among the plurality of objects;
Non-emphasizing means for performing image processing on the image that makes at least one of objects other than the specific object out of the plurality of objects inconspicuous;
A virtual object insertion means for acquiring a virtual object corresponding to the specific object from the virtual object storage means, and inserting the virtual object into the image in association with the specific object;
Display means for displaying an image in which the virtual object is inserted;
Is provided.

  In this way, the subject included in the real space image is divided into several objects, and the object other than the object displayed in association with the virtual object is processed to make it inconspicuous, thereby making the virtual object relatively conspicuous. be able to.

  Moreover, in the present invention, it is preferable that the componentizing unit sets a result of area division performed based on at least one of an edge, a color, and a character as one object. When an image is taken from an in-vehicle camera, a road, a building, a signboard, a person, a vehicle, or the like is assumed as an object included in the image. In order to recognize each object as one object, it is preferable to employ the above-described region division processing. In particular, since characters are often described on a signboard, the region can be identified with high accuracy by using character recognition processing.

  In the present invention, the specifying unit acquires the content of the object by character recognition or image recognition, and specifies the object when a virtual object corresponding to the object is stored in the virtual object storage unit. It is preferable to determine that the object is an object.

  Which object the virtual object is displayed in association with is stored in the virtual object storage means. With the above processing, the virtual object can be displayed when an object associated with the virtual object appears in the image.

  In the present invention, the image processing apparatus further includes condition acquisition means for acquiring a condition related to the object to be displayed in association with the virtual object, and the specifying means is the specific object when the object matches the condition It is preferable to judge.

  When an object associated with a stored virtual object is captured, displaying all the virtual objects makes it difficult to see the display. Therefore, it is preferable to display only the virtual object desired by the user. As described above, by acquiring information about the virtual object to be displayed by the condition acquisition unit and displaying only the virtual object that matches the condition, it is possible to avoid the display becoming difficult to see.

  Further, in the present invention, the non-emphasizing means makes the object inconspicuous by performing any of the processes of excluding an object other than the specific object, painting, decoloring, and wireframe. It is preferable.

  Any process can be used to make it inconspicuous, but the reason why the display is cluttered is considered to be because objects of various colors exist in the image. Therefore, it is preferable to make the color inconspicuous by the above processing.

In the present invention,
The image acquisition means is an in-vehicle camera that acquires an image outside the vehicle,
The specific object and the object subjected to the non-emphasizing process are signboards.
It is also preferable that

When a real-space image is taken by an in-vehicle camera, a road, a building, a signboard, or the like becomes a subject. Here, by displaying the virtual object together with the signboard with the signboard as the specific object, additional information can be provided to the signboard. In addition, since various noticeable colors are used for the signboard, it becomes difficult to see when the virtual object is displayed in a superimposed manner. Therefore, signboards other than those that are displayed in association with the virtual object are subjected to de-emphasis processing so as not to stand out, thereby ensuring visibility even when the virtual object is displayed.

  In addition, this invention can also be grasped | ascertained as an information provision system provided with at least one part of the said means. It can also be understood as an information providing method for executing at least a part of the above processing of the present invention. The present invention can also be understood as a computer program for causing a computer to execute this method. Each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

  According to the present invention, when a virtual object is attached to a real space image and displayed to provide information, it is possible to easily display the attached virtual object.

It is a figure which shows the functional block of the information provision system which concerns on this embodiment. It is a flowchart which shows the flow of the process which the information provision system concerning this embodiment performs. It is a figure which shows the example of the image of the real space image | photographed with the camera. It is a figure which shows the example of the image processed so that signboards other than the signboard which provides a virtual object among the images of FIG. 3 may not stand out. It is a figure which shows the example of the image which inserted the virtual object with respect to the image of FIG.

<System configuration>
The information providing system according to the present embodiment is a system implemented as a part of a car navigation system mounted on a vehicle, and provides information by superimposing a virtual object on a real space image obtained from an in-vehicle camera. It is a system that performs. A virtual object (virtual object) refers to a video object generated and displayed by a computer. The virtual object may have any content, but it is desirable to provide some useful information to the vehicle occupant. Specifically, supplementary explanations and advertisements for actual scenery, or highlighting of actual scenery can be given as examples of virtual objects.

  In the following description, an information providing system that displays advertisement information on a signboard in a video acquired from a vehicle-mounted camera in a superimposed manner on an actual video is taken up as an example. That is, in the following description, the virtual object is advertisement information.

  FIG. 1 is a diagram showing functional blocks of an information providing system 10 according to the present embodiment. The information providing system 10 includes a camera 100, an image processing unit 200, a display condition input unit 300, and an image display unit 400.

  The camera 100 mainly acquires an external scenery in front of the vehicle. The resolution and frame rate of the image captured by the camera 100 may be arbitrary. Also, any existing format of the captured image can be adopted.

The image processing unit 200 is realized by executing a program by a computer including an arithmetic device (processor) and a memory. The image processing unit 200 includes, as its functional units, a resolution conversion unit 201, a component conversion unit 202, a video component database 203, a virtual object insertion component identification unit 204, a component non-emphasis unit 205, a virtual object database 206, and a virtual object insertion unit. 207.

  The resolution conversion unit 201 performs processing for reducing the resolution of an image captured by the camera 100. As a result, the subsequent processing load is reduced, so that real-time processing is possible even with a processor having relatively low performance.

  The component converting unit 202 performs an encoding process such as wire frame conversion or monochrome conversion on the reduced resolution image, and further performs a process of dividing the subject into a plurality of components in the image. Here, the parts may be any objects, but for example, roads (roadways and sidewalks), buildings, vehicles, people, signboards, sky, traffic signs, and the like can be used as parts. Various methods can be used for image processing for componentization. As an example, it is possible to employ a region division process performed based on low-level image features such as boundaries, edges, and color-specific histograms as introduced in the following document (Section 1.2).

Benjamin Z. Yao, et. Al., "I2T: Image Parsing to Text Description", Proceedings of IEEE, Vol. 98, no.8, pp 1485-1508, August, 2010. (Special issue on Internet
Vision)

  In addition, since signboards in the real world often include characters in a rectangle with a single color background, the area of the signboard can be specified by combining character recognition processing with area division processing using edges, colors, etc. Can be done with high accuracy.

  What kind of object is adopted as a part can be determined according to the requirements of the application. Specifically, it is desirable to identify an object that is desired to be displayed with a virtual object attached thereto or an object that is desired to be inconspicuous on the screen as a component. Other objects may not necessarily be identified as parts.

  The component conversion unit 202 refers to the componentized scenery registered in the video component database 203 and identifies what type of component each component is. Specifically, it identifies which type of component, such as a road, a building, a vehicle, a person, and a signboard, each component converted into a component by the component converting unit 202 corresponds to. In addition, more detailed identification is performed to identify which type of vehicle and which type of signboard (which may be classified for each advertiser or each advertiser's genre). It is also preferable. In addition, for an object on which characters such as a signboard are written, it is possible to determine which advertiser's signboard the signboard is by using character recognition processing. Alternatively, the image data of the signboard and the advertiser are stored in the video parts database 203 in association with each other, and it is possible to determine which advertiser's signboard the signboard is by collating the image with the billboard made into a component. it can.

  The virtual object insertion component specifying unit 204 (hereinafter, referred to as a specifying unit) is a functional unit that determines which of the objects converted into components by the component converting unit 202 is to be assigned a virtual object. The specifying unit 204 determines whether there is a part associated with the virtual object registered in the virtual object database 206 in the image, and when there is an object (part) associated with the virtual object, On the other hand, it is determined that a virtual object is assigned. If all the virtual objects are displayed, the display may be cluttered, and those that are not of interest to the user are displayed. Therefore, the conditions acquired from the display condition input unit 300 are It is preferable to display only matching virtual objects. As this condition, for example, it is possible to adopt a condition in which only information relating to a specific genre (for example, a restaurant) is displayed.

  The component de-emphasizing unit 205 performs processing (non-emphasizing processing) for making components unnecessary for the driver out of the components in the image inconspicuous in the image. As a process for making it inconspicuous, methods such as deleting (excluding) a part, painting with an inconspicuous color, forming a wire frame, and decoloring can be considered. The parts to be made inconspicuous by the part non-emphasizing unit 205 are specific types of parts (for example, vehicles and signs are inconspicuous), parts having a specific color (for example, parts having a specific hue, brightness, etc. It is conceivable to make the high-intensity parts inconspicuous). Note that the parts to be subjected to the de-emphasis processing are other than the parts specified by the specifying unit 204.

  The virtual object database 206 stores virtual objects to be displayed in association with components in the image. More specifically, information about which type of component is associated with the virtual object to be displayed and what kind of virtual object to display is stored. The contents of the virtual object may be image information or character information.

  The virtual object insertion unit 207 inserts a virtual object registered in the virtual object database 206 into the image data obtained from the in-vehicle camera. The virtual object insertion unit 207 acquires a virtual object related to the object specified by the specifying unit 204 from the virtual object database 206 and inserts it into the image data.

  The image that has been subjected to the insertion of the virtual object and the part non-emphasis processing by the image processing unit 200 is output to the image display unit 400 and displayed.

  The image display unit 400 is an image output device such as a liquid crystal display. The display condition input unit 300 is a functional unit that allows a user to input conditions related to information (virtual object) provided by the information providing system via an input device such as a touch screen or a button.

(Operation example)
A specific example of processing performed by the information providing system according to the present embodiment will be described with reference to FIGS. FIG. 2 is a flowchart showing a flow of processing performed by the information providing system. 3 to 5 are diagrams for explaining the contents of the image processing based on a specific example.

  First, the camera 100 captures an image outside the vehicle, and passes the captured image to the image processing unit 200. (S11). An example of an image photographed by the camera 100 is shown in FIG. In the example shown in FIG. 3, a roadway, a sidewalk, a building, and a signboard are shown as subjects in the image. Note that the shaded object is a signboard, but this is only shaded for convenience of explanation, and no processing is performed on the captured image at this time. Actually, in addition to these, vehicles, persons, roadside trees, traffic signs, and the like may be included in the image, but these objects are omitted for the sake of simplicity.

  The image processing unit 200 performs image processing on an image captured by the camera 100. First, the resolution conversion unit 201 performs a process of reducing the resolution of the input image (S12). As a result, the subsequent processing can be performed at high speed. Note that although the image is blurred as a whole due to the reduction in resolution, in the description using FIGS.

Next, the component converting unit 202 divides an image in which the resolution has been reduced using low-level image features such as boundaries, edges, and color-specific histograms to divide an object in the image into a plurality of components. The parts collating unit 204 identifies the parts by comparing with the scenery stored in the video part database 203 (S13). In the example of FIG. 3, the roadway, the sidewalk, the building, the sky, and the signboard can be identified as one part. Vehicles, persons, roadside trees, traffic signs, etc. not shown in FIG. 3 can also be identified as parts.

  In FIG. 3, eight signboards 301 to 308 are identified. There are various types of signboards. In the present system, the following signboards are targeted for identification. The signboard 301 is a signboard that is pasted on the wall or window of a building. The signboard 302 is a banner signboard. The signboard 303 is a rooftop signboard installed on the rooftop of the building. The signboards 304 to 306 are sleeve signboards that are attached to the outer wall of the building and project from the building. The signboards 307 and 308 are stand signboards installed on the ground. In addition to those shown in FIG. 3, signboards such as field signboards, cutout signboards, window films, banners, and goodwill can be identified.

  The signboard can be extracted based on the edge and color as described above. Since the signboard has a rectangular shape and often has a single background color, the signboard can be identified based on such characteristics. In addition, since characters are often described on a signboard, it is also preferable to identify an object in which characters are described in a rectangular shape having a single background color as a signboard using character recognition processing. As another method, a comparison may be made with an object (image data) registered in advance as a signboard to identify whether the object in the image is a signboard. Further, the advertiser of the signboard is specified based on the characters written on the signboard and collation with the registered image.

  Next, the specifying unit 204 determines to which object the virtual object is to be assigned among the plurality of parts that are componentized (S15). The present embodiment is an information providing system that adds additional information (advertisement) as a virtual object to a signboard. Therefore, the signboard related to the virtual object stored in the virtual object database 206 is specified as the object to which the virtual object is assigned. That is, when it is registered in the virtual object database 206 that an advertisement of a certain content is given to a certain advertiser's signboard and displayed, the advertiser's part is identified as an object to which the virtual object is attached. Is done. In addition, when the condition regarding the object to which the virtual object is added is specified from the display condition input unit 300, a signboard that satisfies this condition is further specified as the object to which the virtual object is added. An example of the condition is a condition that a signboard of a specific genre is targeted. Here, the description will be continued assuming that the signboard 301 in FIG. 3 is specified as a component to which a virtual object is assigned.

  The component de-emphasizing unit 205 performs a de-emphasizing process so that components other than the component specified by the specifying unit 204 (hereinafter referred to as a specific component) are not conspicuous in the image (S16). In the present embodiment, a signboard other than the specific part is a target of the de-emphasis process. This is because a signboard usually has a conspicuous color, and when a virtual object is displayed, if the virtual object is also displayed in a conspicuous color, the display becomes difficult to see. By making the signboards other than the specific part inconspicuous, the specific part and the virtual object are relatively emphasized, and the difficulty in viewing is reduced. It should be noted that parts other than the signboard may be subjected to the de-emphasis processing. In this case, all of a specific type of component (for example, a vehicle or a building) may be targeted for non-emphasis processing, or a component having a conspicuous color (for example, a color with high brightness) among objects other than the specific component may be excluded. It may be a target of enhancement processing.

  For the de-emphasis processing performed by the component de-emphasis unit 205, it is possible to employ a method such as deletion of a component, painting with an inconspicuous color, wire frame, or decolorization.

In the example illustrated in FIG. 4, the signboard 301 is identified as a part to which a virtual object is added in step S <b> 15, and the other signboards 302 to 308 are decolored so as not to stand out. Note that decolorization is image processing that reduces at least one of lightness and saturation.
This includes making it grayscale.

  The virtual object insertion unit 207 acquires a virtual object associated with the signboard identified in step S15 from the virtual object database 206 and superimposes it on the image data (S17). In this embodiment, the virtual object is advertisement information related to a signboard. Then, the image to which the virtual object is added is displayed on the image display unit 400. FIG. 5 is an example of an image to which a virtual object is assigned. Here, an image to which advertisement information (virtual objects) 311 to 313 is assigned is displayed in association with the signboard 301 (S18). Here, three virtual objects are assigned to specific parts and displayed, but the number of virtual objects to be assigned is arbitrary.

(Operation and effect of this embodiment)
According to the information providing system according to the present embodiment, when a virtual object is added to a real space image and displayed, an object having a color similar to the virtual object is present and the display is difficult to see. Can do. In real space, especially in urban areas, a large number of signboards are installed, and each of them uses a conspicuous color. When displaying a virtual object, the same color as that of a signboard is often used because it is desired to make the virtual object stand out. Then, if virtual objects are simply superimposed and displayed, the display becomes cluttered and difficult to see. On the other hand, in the present embodiment, since non-emphasis processing such as decolorization is performed on a signboard other than a signboard displaying a virtual object or other objects with a conspicuous color, it is possible to prevent display from becoming difficult to see. .

  In addition, the virtual object and the target object (signboard) to which the virtual object is assigned become conspicuous in the screen, so that the information presented by the user can be understood more quickly.

(Modification)
In the above example, the virtual object has been described as character information of an advertisement, but the content of the virtual object may be arbitrary. For example, image information may be presented instead of text information. Specifically, the signer's company logo may be displayed as a virtual object. In addition, a display for emphasizing the signboard, for example, an arrow pointing to the signboard or a display surrounding the signboard may be used as the virtual object.

  In the above example, the target object to which the virtual object is assigned is described as the signboard. However, the target object to which the virtual object is assigned may be an arbitrary object. For example, a virtual object may be assigned to a building, a road, a traffic sign, a vehicle, and the like. Also in this case, it is possible to prevent the display from becoming difficult to see by giving the virtual object by performing the process of making the object similar in color to the virtual object inconspicuous.

DESCRIPTION OF SYMBOLS 100 Camera 200 Image processing part 201 Resolution conversion part 202 Componentization part 203 Image | video component database 204 Virtual object insertion part specification part (specification part)
205 parts de-emphasis unit 206 virtual object database 207 virtual object insertion unit 300 display condition input unit 400 image display unit

Claims (7)

Image acquisition means for acquiring an image of real space;
Virtual object storage means for storing a virtual object to be displayed in association with an object in the image;
Componentizing means for dividing a subject included in an image acquired by the image acquiring means into a plurality of objects; a specifying means for specifying a specific object that is an object to be displayed in association with the virtual object among the plurality of objects;
Non-emphasizing means for performing image processing on the image that makes at least one of objects other than the specific object out of the plurality of objects inconspicuous;
A virtual object insertion means for acquiring a virtual object corresponding to the specific object from the virtual object storage means, and inserting the virtual object into the image in association with the specific object;
Display means for displaying an image in which the virtual object is inserted;
An information providing apparatus comprising: The componentization means uses a result of area division performed based on at least one of an edge, a color, and a character as one object.
The information providing apparatus according to claim 1. The specifying unit acquires the content of the object by character recognition or image recognition, and determines that the object is a specific object when a virtual object corresponding to the object is stored in the virtual object storage unit. ,
The information providing apparatus according to claim 1 or 2. Further comprising condition acquisition means for acquiring a condition relating to an object to be displayed in association with the virtual object;
The specifying means determines that the object is a specific object when the object meets the condition;
The information providing apparatus according to claim 3. The non-emphasizing means removes an object other than the specific object, paints, decolors, or wireframes to make the object inconspicuous,
The information provision apparatus in any one of Claims 1-4. The image acquisition means is an in-vehicle camera that acquires an image outside the vehicle,
The specific object and the object subjected to the non-emphasizing process are signboards.
The information provision apparatus in any one of Claims 1-5. An information providing method performed by the information providing apparatus,
An image acquisition step of acquiring an image of real space;
A componentization step of dividing a subject included in the image acquired in the image acquisition step into a plurality of objects; a specification step of specifying a specific object that is an object to be displayed in association with a virtual object among the plurality of objects;
A non-enhancement step of performing image processing on the image to make at least one of the objects other than the specific object out of the plurality of objects inconspicuous;
A virtual object insertion step of acquiring a virtual object corresponding to the specific object and inserting it into the image in association with the specific object;
A display step of displaying an image in which the virtual object is inserted;
Information providing method including

Images