JP2014096057A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To adjust a position of information to be additionally displayed by augmented reality so as not to hide information which is considered to be important to a viewer.SOLUTION: An image processing apparatus includes: acquiring means for acquiring a captured image; specifying means for specifying an object included in the image acquired by the acquiring means; determination means for determining whether the object specified by the specifying means is to be displayed preferentially or not; and generation means for generating information for displaying the image by combining the image with information on the object specified by the specifying means. The generation means displays the information on the object specified by the specifying means in a position which does not overlap the object determined by the determination means to be displayed preferentially.

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that generates image information for display by adding information to acquired image information.

  2. Description of the Related Art Conventionally, a technique called augmented reality has been provided in which an image captured by a camera or an image acquired from another device is displayed with information added thereto.

  For example, Non-Patent Document 1 ("Sekai Camera Web", [October 30, 2012 search], Internet <URL: http://sekaicamera.com>) and Non-Patent Document 2 (Sekai Camera for Android HOWTO- “Sekai Camera Support Center”, [October 30, 2012 search], Internet <URL: http://support.sekaicamera.com/en/how-to/sekaicamera-android>) Is an example. According to “Sekai Camera”, a user can display an air tag on a finder image of the camera and record the air tag together with the finder image.

  Furthermore, in such a technique, in Patent Document 1 (Japanese Patent Laid-Open No. 2012-60240), when the first user and the second user are included in the input image captured by the imaging device, imaging for each user is performed. Image processing for determining which of the first display information associated with the first user and the second display information associated with the second user is to be superimposed on the front based on a parameter corresponding to the distance from the device An apparatus is disclosed.

JP 2012-60240 A

“Sekai Camera Web”, [October 30, 2012 search], Internet <URL: http://sekaicamera.com> "Sekai Camera for Android HOWTO-Sekai Camera Support Center", [October 30, 2012 search], Internet <URL: http://support.sekaicamera.com/en/how-to/sekaicamera-android>

  However, in Patent Document 1, only the context of two or more pieces of information (air tag) to be additionally displayed is examined, and the position in the image where the air tag is displayed is examined. It wasn't done. In augmented reality, when a captured image is displayed with an air tag added, the person's face in the image may be hidden depending on the information. It is assumed that the user feels uncomfortable. In addition, when an air tag is displayed, an important object in an image is hidden, so that a person who sees an augmented reality display may not obtain important information. For example, if a police station building is included in the captured image but the police station building is hidden by an air tag in the augmented reality display, the photographer will continue to take the reason. In some cases, it may be treated as a suspicious person against the police station.

  The present invention has been devised in view of such circumstances, and its purpose is to adjust the position of information additionally displayed by augmented reality so as not to hide information considered to be important to the viewer. It is to be.

  According to a certain aspect, the acquisition unit that acquires the captured image, the identification unit that identifies the target included in the image acquired by the acquisition unit, and the target identified by the identification unit are given priority. Judgment means for judging whether or not the image should be displayed, and generation means for generating information for displaying the image by combining the image and information on the object identified by the identification means. An image processing apparatus is provided. The generating means displays information related to the object specified by the specifying means at a position that does not overlap with the object determined by the determining means to be displayed with priority.

  Preferably, the acquisition unit further acquires a position where the image was shot and a shooting direction when the image was shot, and the specifying unit is based on the position and the shooting direction acquired by the acquisition unit. Is identified.

  Preferably, the specifying unit specifies an object included in the image by performing image recognition on the image acquired by the acquiring unit.

  Preferably, the determination means stores information for specifying an object displayed with priority, and the specifying means is based on whether the object specified by the specifying means corresponds to the stored object. It is determined whether the object specified by is to be displayed with priority.

  Preferably, the generation unit pays attention to the image when information relating to the object specified by the specifying unit cannot be displayed at a position that does not overlap with the object determined by the determination unit to be displayed with priority. Information indicating that the target object should be included is displayed.

  Preferably, the generation unit displays information related to the object specified by the specifying unit at a position that does not overlap a specific part of the object determined by the determining unit to be displayed with priority.

  According to another aspect, an image processing method is provided. In the image processing method, a step of acquiring a captured image, a step of specifying a target object included in the acquired image, and whether the target object should be displayed with priority for the specified target object A step of determining whether or not, and a step of generating information for displaying the image by combining the image and information related to the object specified by the specifying means. In the step of generating information for displaying the image, information on the identified object is displayed at a position that does not overlap with the object determined to be displayed with priority.

  According to yet another aspect, a program executed by a computer of an image processing apparatus is provided. The program includes a step of acquiring a captured image, a step of specifying a target object included in the acquired image, and the target object to be displayed with priority on the specified target object on the image processing apparatus. And a step of generating information for displaying the image by combining the image and information related to the object specified by the specifying means. In the step of generating information for displaying the image, information on the identified object is displayed at a position that does not overlap with the object determined to be displayed with priority.

  According to the present invention, when an object to be displayed with priority is included in the captured image, the object included in the image is not overlapped with the object to be displayed with priority. Information about the object is displayed.

  Thereby, the position of the information additionally displayed by augmented reality can be adjusted so as not to hide the information considered important for the viewer.

It is a figure which shows an example of the external appearance of an image processing apparatus. It is a figure which shows the specific example of the hardware constitutions of an image processing apparatus. It is a figure for demonstrating the outline | summary of the process performed in an image processing apparatus. It is a figure which shows the specific example of the functional block of an image processing apparatus. It is a flowchart of an example of the process performed in order to produce | generate the image which displays an augmented reality. It is a figure for demonstrating the example of a display containing the augmented reality displayed as a transparent image. It is a figure which shows an example of the display screen of the augmented reality in a comparative example.

  Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings. In the following description, elements having the same function and action are denoted by the same reference numerals throughout the same drawings, and redundant description will not be repeated.

[Appearance of image processing device]
FIG. 1 is a diagram illustrating an example of the appearance of the image processing apparatus. As shown in FIG. 1, the image processing apparatus 1 of the present embodiment includes a shutter button 25 </ b> A and a display device 30. The image processing apparatus can be realized by various information processing apparatuses such as a digital camera, a tablet terminal, a mobile phone, and a high-function mobile phone (smart phone). Note that the image processing apparatus 1 is only required to process at least a captured image, and the imaging apparatus is not an essential element of the image processing apparatus 1. The image processing apparatus 1 may process an image captured by an imaging apparatus such as a camera included in the image processing apparatus, or may process an image acquired from another device.

[Hardware configuration of image processing apparatus]
FIG. 2 is a diagram illustrating a specific example of the hardware configuration of the image processing apparatus 1.

  Referring to FIG. 2, the image processing apparatus 1 includes a CPU (Central Processing Unit) 20 that is an arithmetic device for controlling the whole, a ROM 21 for storing a program executed by the CPU 20, and a program executed by the CPU 20. RAM 22 for functioning as a work area when executing the operation, an electronic compass 23 for detecting the orientation of the image processing apparatus 1 including a magnetic sensor, a GPS (Global Positioning System) signal, or a position from a base station A GPS controller 24 for receiving a signal and acquiring position information of the portable terminal, an operation unit 25 for receiving operation input, a camera 26, a communication device 27 for communicating with an external device, an image A media controller 28 for writing and reading information to and from the recording medium 100 detachable from the processing apparatus 1; Display device 30. The display device 30 is realized by, for example, a liquid crystal display or a plasma display, and may be realized as a touch panel integrally with the operation unit 25. The electronic compass 23 and the GPS controller 24 input the acquired signal indicating the orientation and position information of the image processing apparatus 1 to the CPU 20.

[Outline of processing in image processing apparatus]
FIG. 3 is a diagram for explaining an outline of processing executed in the image processing apparatus 1. FIG. 3A shows an example of an image photographed by the camera 26. FIG. 3B shows an example of information relating to the object extracted from the image 31 of FIG. FIG. 3C shows an example of an augmented reality display screen generated by combining the information shown in FIG. 3B with the image 31 of FIG.

  From the image 31 of FIG. 3A, the image processing apparatus 1 extracts the subject 501 and the subject 502 by image recognition such as pattern recognition. Then, the image processing apparatus 1 tries to acquire information on each of the subjects 501 and 502 via, for example, a network.

  An example of an information acquisition mode will be described. Based on the orientation and position information of the image processing apparatus 1 when the image 31 is photographed, the image processing apparatus 1 is photographed at the position identified by the position information in the direction identified by the orientation. In this case, the area to be imaged is specified. Then, the image processing apparatus 1 identifies an object having an appearance that approximates the appearance of the subjects 501 and 502 with an accuracy equal to or higher than a predetermined probability in the region. And the information regarding the said object is acquired. In addition, the acquisition of such information can employ a known technique in the Sekai camera or the like. Here, the detailed description of the known technology will not be repeated.

  When the information about the subject in the image 31 is acquired, the image processing apparatus 1 generates an image of the information as shown in FIG. 3B, for example. In this specification, an example in which the image processing apparatus 1 acquires information about the subject 501 is described. In FIG. 3B, a balloon 903 is an image that displays information about the subject 501.

  The image processing apparatus 1 generates an image displaying augmented reality by combining the image 31 and the balloon 903. It should be noted that the subject 502 extracted by image recognition is located at a position corresponding to the balloon 903 in the image 31. As will be described later, the subject 502 is registered as a subject to be displayed with priority. Then, the image processing apparatus 1 combines the balloon 903 with the image 31 so as not to overlap the area of the subject 502, as shown as an image 33 in FIG. Generate. In FIG. 3C, a region 503A indicated by a broken line indicates a default display position of the balloon 903 as shown in FIG. A balloon 503 indicates an actual display position of the balloon 903. The image processing apparatus 1 appropriately changes the position for displaying the balloon from the default position in the augmented reality display according to the position of the subject in the image 31. In this specification, the balloon 903 is an example of an “augmented reality image”, and the image 33 is an example of an “augmented reality image”.

  Depending on the subject, the degree of discomfort caused by being hidden by the blowing may vary depending on the location of the subject. For example, when the subject is a human being, the discomfort is great when the face is hidden by the blowout, but the discomfort does not occur so much even if other parts (for example, legs) are hidden by the blowout. Conceivable. The image processing apparatus 1 can specify the type of the subject (building, human, non-human animal, etc.), can specify the part of the subject, and further discomfort about being hidden by the blowout according to the part of the subject. When the degree at which it is predicted that the occurrence of the above is set is set, the balloon may be displayed at a position where the degree is low.

[Functional configuration of image processing apparatus]
FIG. 4 is a functional block diagram of the image processing apparatus 1. The image processing apparatus 1 functions as an image acquisition unit 51, a position information acquisition unit 52, an orientation information acquisition unit 53, a gaze impossibility target determination unit 54, an augmented reality generation unit 55, and an output image generation unit 50. Including.

  The image acquisition unit 51 acquires an image to be processed. The image acquisition unit 51 is realized by, for example, the camera 26, but may be realized by the communication device 27 that receives an image from another device.

  The position information acquisition unit 52 acquires position information of the imaging device when the processing target image is captured. The position information acquisition unit 52 is realized by the electronic compass 23, for example. However, the position information acquisition unit 52 may be realized by the CPU 20 that executes the process of extracting the position information as described above from information received together with an image received from another device. possible. The azimuth information acquisition unit 53 acquires the shooting direction of the imaging device when the processing target image is shot. The azimuth information acquisition unit 53 is realized by, for example, the GPS controller 24, but is realized by the CPU 20 that executes processing for extracting the shooting direction as described above from information received together with an image received from another device. There is also a possibility.

  The gaze impossibility target determination unit 54 extracts a subject from the processing target image, and determines whether or not each subject is a subject (specific subject) that cannot be watched. This determination corresponds to an example of determining whether or not each subject is a subject to be displayed with priority. The augmented reality generation unit 55 extracts a subject from the processing target image, acquires information about each subject, and generates an image for displaying the information. The output image generation unit 50 generates an image for displaying augmented reality by combining the image to be processed and the image generated by the augmented reality generation unit 55, and outputs the generated image to the display device 30. As a result, the display device 30 displays an image as shown in FIG.

  The output image generation unit 50 includes a display position determination unit 57. The display position determination unit 57 determines the position at which the image generated by the augmented reality generation unit 55 is displayed in the image displaying the augmented reality based on the determination result by the gaze impossibility target determination unit 54. More specifically, as described with reference to FIGS. 3A to 3C, when it is determined that the subject included in the image to be processed (image 31) is a specific subject. The display position determination unit 57 displays the image generated by the augmented reality generation unit 55 in the image that displays the augmented reality at a position that does not overlap the subject.

  The gaze impossibility target determination unit 54, the augmented reality generation unit 55, and the output image generation unit 50 are realized by, for example, the CPU 20 that executes a predetermined program. Note that at least one of these may be realized by a dedicated hardware element.

[Process overview]
FIG. 5 is a flowchart of an example of processing executed by the CPU 10 of the image processing apparatus 1 to generate an image displaying augmented reality. The contents of the process will be described with reference to FIG.

  First, in step S10, the CPU 10 acquires an image to be processed, position information and orientation information at the time of shooting the image, and advances the process to step S20. For example, the CPU 10 acquires an image to be processed from the camera 26, acquires position information from the electronic compass 23, and acquires orientation information from the GPS controller 24.

  In step S20, the CPU 10 extracts a subject from the processing target image acquired in step S10, acquires information about the extracted subject, and based on the acquired information, augmented reality such as the balloon 903 in FIG. Is generated, and the process proceeds to step S30.

  In step S <b> 30, the CPU 10 determines whether or not the subject extracted from the processing target image acquired in step S <b> 10 includes a gaze impossible target (specific subject). If the CPU 10 determines that the specific subject is included, the process proceeds to step S40. If the CPU 10 determines that the specific subject is not included, the process proceeds to step S80.

  In step S40, the CPU 10 determines whether or not the default display position of the augmented reality information generated in step S20 on the augmented reality display screen overlaps the subject determined as the specific subject in step S30. As an example of the default display position of augmented reality, for example, a position (a position 1 cm away on the right side) having a predetermined relationship with the subject (subject 501) to which the augmented reality (speech 903) corresponds is given. Can do. If the CPU 10 determines that the default display position overlaps the specific subject, the process proceeds to step S50. If the CPU 10 determines that the default display position does not overlap, the process proceeds to step S80.

  In step S50, the CPU 10 determines whether or not the augmented reality position determined to overlap with the specific subject in step S40 can be changed. If the CPU 10 determines that the change is possible, the process proceeds to step S60. If the CPU 10 determines that the change cannot be made, the process proceeds to step S70. The determination in step S50 is, for example, a determination of whether or not there is an area for displaying the augmented reality generated in step S20 other than the area occupied by the subject that is the “specific subject” in the processing target image. is there. If it is left, the CPU 10 determines that the augmented reality position can be changed, and if not, determines that the augmented reality position cannot be changed.

  In step S60, the CPU 10 changes the augmented reality display position generated in step S20 to a region other than the region occupied by the specific subject in the processing target image, and proceeds to step S80. As a result, as shown in FIG. 3C, the display position of the balloon is changed from the area 503 </ b> A to the balloon 503 on the screen displaying the augmented reality.

  In step S70, the CPU 10 changes the augmented reality to a design that is as transparent as possible, and advances the process to step S80.

  In step S80, CPU10 produces | generates the image output to the display apparatus 30 as a screen which displays an augmented reality, and complete | finishes a process.

[Summary of processing in image processing apparatus]
As described above, according to the processing described with reference to FIG. 5, when the image processing apparatus 1 determines that the specific subject is included in the processing target image, the augmented reality of the other subject included in the processing target image. Is displayed at an augmented reality position so that it does not overlap a specific subject. FIG. 7 is a diagram illustrating an example of an augmented reality display screen in the comparative example.

  7A shows an image 91 that is an image to be processed, FIG. 7B shows a balloon 903 that is an augmented reality of the subject 901 in the image 91, and FIG. 7C shows an image 91 and a balloon An example of an augmented reality display screen generated by combining with 903 is shown. In this example, a subject 901 and a subject 902 are extracted from the image 91. In FIG. 7C, a balloon 903 that is an augmented reality of the subject 901 is displayed superimposed on the subject 902. As a result, a person who has viewed the image 93 that is the augmented reality display screen may not be able to confirm the subject 902 in the image 93.

  On the other hand, in the image 33 in FIG. 3C, the balloon 503 is arranged so as not to overlap the subject 502. Thereby, the person who has visually recognized the image 33 can confirm the subject 502 together with the balloon that is the augmented reality of the subject 501.

  As shown in step S70, augmented reality is displayed as transparently as possible as an example of a case where augmented reality cannot be displayed in a region other than a specific subject. FIG. 6 is a diagram for explaining a display example including augmented reality displayed as a transparent image.

  6A shows an image 34 to be processed, FIG. 6B shows a balloon 503 that is an augmented reality of the subject 501 in the image 34, and FIG. 6C shows an image 34 and a balloon 503. An example of a display screen of augmented reality generated by being combined is shown. In this example, a subject 501, a subject 502, and a subject 510 are extracted from the image 34.

  As understood from FIGS. 6A and 6B, the image 34 does not include a space that can display the balloon 503 in an area other than the subjects 501, 502, and 510. In this case, the image processing apparatus 1 displays the balloon 503 as a transmission image. Thereby, in the image 36 in FIG. 6C, the outer shape of the subject 510 located behind the balloon 503 can be visually recognized.

  In step S70, the augmented reality is made a transparent image, and when the transparent image is combined with the processing target image in step S80, the image processing apparatus 1 changes the arrangement of the transparent image to the processing target image. It may be determined according to the priority order between two or more included subjects. That is, when determining the arrangement of the balloons 503 in the image 36 in FIG. 6C, the CPU 10 sets priorities between subjects other than the subject 501 corresponding to the balloons 503, that is, between the subjects 502 and 510. Identify. Then, the image processing apparatus 1 determines the arrangement of the balloons at positions where the balloons do not overlap with priority in descending order of priority. In the example of FIG. 6, the subject 502 has a higher priority than the subject 510. Thereby, the image processing apparatus 1 arranges the balloon 503 subjected to the transmission processing while avoiding the position of the region 503A overlapping the subject 502 in the image 36 of FIG.

  The priority order is set based on, for example, the type of object specified as the subject. More specifically, for example, when the object whose subject is specified is “person”, the highest priority is given, followed by “animal” and “building”. Information specifying the priority order is stored in, for example, the ROM 21, the RAM 22, or the recording medium 100.

  If it is determined that the augmented reality display space cannot be secured, the image processing apparatus 1 displays the image to be processed without including the augmented reality display instead of combining the images after the transparent process in step S70. May be. However, in this case, it is preferable that the image processing apparatus 1 notifies the user that the augmented reality has not been performed by display or the like.

  Further, the image processing apparatus 1 may display the augmented reality image superimposed on the subject 902 as shown in FIG. 7C instead of combining the images after the transmission processing in step S70. . However, in this case, it is preferable that the image processing apparatus 1 informs the subject 902 that it is difficult to visually recognize the augmented reality image (the balloon 903).

  In addition, when an object specified as a subject is an important certain type of object that causes danger when it is overlooked, such as a traffic light or a road sign, it is preferable to display the object particularly preferentially. . That is, when such an object is specified in the processing target image, it is particularly preferable to adjust the position of the augmented reality image so as not to overlap the object.

  Note that the change or adjustment of “augmented reality image position” in the present embodiment is not limited to a simple position change as described with reference to FIGS. It also includes changing the position of the outer edge of the image by reducing the actual image.

  It should be thought that embodiment disclosed this time and its modification are illustrations in all the points, and are not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims. It is intended that the techniques disclosed in the embodiments and the modifications thereof can be implemented alone or in combination as much as possible.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 23 Electronic compass, 24 Controller, 25 Operation part, 25A Shutter button, 26 Camera, 27 Communication apparatus, 30 Display apparatus, 50 Output image generation part, 51 Image acquisition part, 52 Position information acquisition part, 53 Direction Information acquisition unit, 54 gaze impossibility target determination unit, 55 augmented reality generation unit, 57 display position determination unit.

Claims (5)

An acquisition means for acquiring a captured image;
Specifying means for specifying an object included in the image acquired by the acquiring means;
Determination means for determining whether the object should be displayed with priority for the object specified by the specifying means;
Generating means for generating information for displaying the image by combining the image and information on the object specified by the specifying means;
The image processing apparatus, wherein the generation unit displays information on the object specified by the specifying unit at a position that does not overlap with the object determined by the determination unit to be displayed with priority. The acquisition means further acquires a position where the image is captured and a photographing direction when the image is photographed,
The image processing apparatus according to claim 1, wherein the specifying unit specifies an object based on the position and the shooting direction acquired by the acquiring unit.   The image processing apparatus according to claim 1, wherein the specifying unit specifies an object included in the image by performing image recognition on the image acquired by the acquiring unit. The determination means includes
Stores information that identifies objects that are displayed with priority,
Based on whether the object specified by the specifying means corresponds to the stored object, it is determined whether the object specified by the specifying means should be displayed with priority. The image processing device according to any one of claims 1 to 3.   The generating means is cautionary about the image when information relating to the object specified by the specifying means cannot be displayed at a position that does not overlap with the object determined by the determining means to be preferentially displayed. The image processing apparatus according to claim 1, wherein information indicating that an object to be included is included is displayed.

Images