JP2014215977A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for making it possible to easily edit extension information associated with certain feature information and associate it with different feature information.SOLUTION: First feature information and second feature information are extracted, extension information associated with the first feature information is acquired, new extension information is generated from the extension information, and the new extension information is associated with the second feature information.

Description

  The present invention relates to a display technique for displaying a virtual image superimposed on an image.

  In recent years, augmented reality (AR) technology has been developed that expands reality by superimposing and displaying virtual augmented information on an image obtained by an imaging apparatus.

  For example, there is a technique for superimposing and displaying extended information (AR content) such as a moving image associated with a specific physical object in a captured image in accordance with the position of the target.

  In Patent Document 1, region information indicating a region of a specific image is detected from a photographed image, and a computer graphics image is synthesized with a region other than the region of the specific image in the photographed image using the detected region information. As a result, when a CG object is superimposed on the background of a live-action landscape and an image of a live-action subject is synthesized in front of the CG object, an image taken at the same time without separately shooting the background of the live-action subject and the subject is used. Can be synthesized with CG.

  Furthermore, there is a technique for easily associating AR content with a physical object.

  For example, in Patent Document 2, in a digital work space expanded by a projector and a camera, a digital object operated by a pointer is dragged and dropped onto a physical object provided with a marker. A technique for linking a digital object to a marker is disclosed.

  In Patent Document 3, the AR information associated with the basic marker is decomposed into semantic units, elements are extracted from the image of the physical object, and each meaning of the AR information is associated with each element of the image. The technique which uses each element of an image as a substitute marker is disclosed.

Patent 04136420 Patent 0433964 JP2012-48720

  However, in the technique described in Patent Document 2, the original digital object is not extended information related to feature information such as a marker. Displaying a digital object requires a digital work space extended by a projector and a camera, so that a large-scale preparation is required to acquire a digital object (extended information) to be operated and edited, and it can be easily used. difficult.

  Further, in the technique described in Patent Document 3, since the basic marker information is directly associated with the physical object element in a semantic unit, the extension information (AR content) associated with the basic marker can only be copied as it is. Can not.

  The present invention has been made in view of the above-described problem, and is an image for enabling extended information (AR content) associated with certain feature information to be easily associated with other feature information as intended. An object is to provide a processing apparatus. Moreover, it aims at providing the method and program.

  The image processing apparatus of the present invention has the following configuration. That is, extraction means for extracting first feature information and second feature information, acquisition means for acquiring extension information associated with the first feature information, and new extension information from the extension information. Generating means for generating and associating means for associating the new extended information with the second feature information;

  According to the present invention, it is possible to easily associate extended information associated with certain feature information with other feature information.

It is a functional block diagram of the image processing apparatus in an embodiment. 1 is a conceptual diagram of an overview of an image processing apparatus in an embodiment. 3 is a flowchart illustrating main processing of the image processing apparatus according to the first embodiment. It is a figure which shows the AR content acquisition process flow in Embodiment 1. FIG. It is a figure which shows the marker in Embodiment 1. FIG. It is a figure which shows AR content information in Embodiment 1 notionally. It is a figure which shows notionally the recording content corresponding to AR content and a marker in Embodiment 1. FIG. 6 is a diagram illustrating a display processing flow of the image processing apparatus according to the first embodiment. FIG. 3 is a display example of the image processing apparatus according to the first embodiment. It is a figure which illustrates the selection method of AR content in Embodiment 1. FIG. 10 is a diagram illustrating a UI for editing AR content in the second embodiment. FIG. 10 is a diagram illustrating a UI for editing AR content in the second embodiment. It is a figure which shows notionally the recording content corresponding to AR content and a marker in Embodiment 3. It is a figure which shows AR content information in Embodiment 3 notionally. FIG. 10 is a diagram illustrating a display processing flow of an image processing apparatus according to a third embodiment. It is a hardware block diagram of an image processing apparatus.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments having the configurations described in the claims.

<Embodiment 1>
First, a functional configuration example according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the present embodiment includes an image processing device 10 that performs processing such as acquisition, editing, and linking of AR content, and an AR content management device 20 that manages AR content.

  Further, the image processing apparatus 10 includes a photographing unit 11, a control unit 12, a communication unit 13, a display unit 14, and an instruction unit 15.

  The AR content management apparatus 20 includes a control unit 21, a communication unit 22, and a recording unit 23.

  As a premise in the present embodiment, the image processing apparatus 10 has a shape as shown in FIG. That is, a photographing unit is mounted on one surface, and a display / instruction unit is mounted on the other surface. This display / instruction unit can be realized by using, for example, a touch panel.

  A hardware configuration example of the image processing apparatus in FIG. 1 will be described with reference to the block diagram in FIG. In the figure, reference numeral 3 denotes a central processing unit (CPU) for controlling the entire apparatus. A read only memory (ROM) 4 stores programs and parameters that do not need to be changed. A random access memory (RAM) 5 temporarily records programs and data supplied from an external device or the like. Reference numeral 12 denotes an external recording medium capable of reading and writing data represented by a memory card. Reference numeral 14 functions as a large-capacity recording device represented by a hard disk or SSD, and is generally larger than the recording capacity of the external recording medium 12. Reference numerals 11 and 13 denote I / Os for connecting the external recording medium 12 and the recording device 14 to a computer device. Reference numeral 9 denotes an I / O with the imaging apparatus 10 that acquires an image. Reference numeral 7 denotes an I / O with a touch panel that receives a user operation. Reference numeral 2 denotes video I / O with the display monitor 1 for displaying data held by the image processing apparatus and supplied data. Reference numeral 15 denotes a communication I / F for connecting to a network line such as the Internet. A system bus 6 connects the units 1 to 15 so that they can communicate with each other.

  The processes in the control units 12 and 21 of each device will be described below using the flowchart of FIG. Hereinafter, the flowchart is realized by the CPU executing the control program.

  First, in the image acquisition process 101 of the image processing apparatus 10, a command is issued to the imaging unit 11 to capture a physical object and acquire an image.

  Next, in the feature extraction process 102 of the image processing apparatus 10, a plurality of pieces of feature information as candidates for superimposing the AR content are extracted from the physical object image.

  Note that the target feature information on which the AR content is superimposed on the photographed image includes a marker type and a non-marker type. The marker type is a technique in which a specific pattern drawn in a predetermined format is printed on a document or the like and is superposed by reading it. For example, it is a QR code (registered trademark). The non-marker type does not require a specific pattern such as a marker, and is a method of superimposing using image feature values (SIFT feature values, SURF feature values) instead. In the present embodiment, description will be made using a marker-type AR content superimposition method, but a non-marker type can also be applied to the present embodiment.

  In the present embodiment, for the sake of explanation, it is assumed that the feature information on which AR content can be superimposed is a marker surrounded by a black spot square as shown in FIG.

  For example, when an image in which a paper document and a sticky note are captured as physical objects is acquired as shown in FIG. 4A, the feature extraction processing 102 acquires two markers as shown in FIG. 4B. It becomes.

  Next, in the AR content information acquisition process 103 of the image processing apparatus 10, AR content information is acquired from the AR content management apparatus. In this process, a plurality of markers (feature information) obtained in the feature extraction process 102 are transmitted to the AR content management apparatus via the communication unit 13, and AR content information corresponding to each marker is received. The AR content information here refers to an AR content correspondence flag indicating whether each marker has a corresponding relationship with the AR content (extended information) as shown in the conceptual diagram of FIG. It includes information such as 3D model information of AR content and a display position with respect to the marker. Note that the AR content display position may be received in a coordinate system based on the marker as shown in FIG.

  Next, the AR content information generation process 201 on the AR content management apparatus 20 side will be described. Here, as a precondition, the AR content management apparatus 20 records the correspondence between the AR content and the marker in the recording unit 23 in advance. This correspondence is as shown in the conceptual diagram of FIG.

  In the AR content information generation processing 201, it is determined whether or not there is a corresponding AR content (extended information) for each marker (feature information) received via the communication unit 22, and AR content information is generated. As described above, this AR content information includes an AR content correspondence flag indicating whether or not there is a corresponding AR content as shown in the table of FIG. 6, and 3D model information of the AR content when there is a corresponding AR content, Including AR content display coordinates. Note that the AR content 3D model information may be created using CAD or the like, as in the case of normal 3D content creation. The AR content display coordinates may be specified using 3D content display coordinates in a coordinate system based on the AR marker using an authoring tool or the like. Whether the received marker corresponds to the AR content may be determined by pattern matching between the marker on the recording unit 23 and the received marker.

  Next, the AR content information transmission process 202 will be described. In this process, the AR content information obtained by the AR content information generation process 201 is transmitted to the image processing apparatus 10 as the reception source via the communication unit 22.

  Next, in the display process 104 of the image processing apparatus 10, an image to be displayed on the display unit 14 in FIG. 1 is generated and displayed using the received AR content information. This process will be described with reference to the flowchart of FIG. 8 and the example of FIG. Hereinafter, the flowchart is realized by the CPU executing the control program.

  First, in step S1041 in FIG. 8, a variable n indicating the number of the marker being processed is initialized with 0, and the process proceeds to step S1042.

  Next, step S1041 refers to the AR content correspondence flag in the AR content information corresponding to the marker Mn being processed. If the flag is TRUE, that is, if the corresponding AR content exists, the process proceeds to step S1043. If the flag is FALSE, that is, the corresponding AR content does not exist, the process proceeds to step S1044.

  In step S1043, the AR content is superimposed and drawn on the marker Mn being processed using the AR content information. Since the technique for superimposing and drawing the AR content on the marker is well known, description thereof is omitted. When the AR content is superimposed and drawn, the process proceeds to step S1045.

  In step S1044, a message indicating that there is no corresponding AR content is displayed for the marker Mn being processed, and the process proceeds to step S1045.

  In step S1045, it is determined whether the number Nmax of markers in the image acquired in the image acquisition process is greater than a variable n indicating the number of the marker being processed. If larger, the process proceeds to step S1046. If not, the process ends.

  In step S1046, 1 is added to the variable n indicating the number of the marker being processed, and the flow advances to step S1042.

  Through the above processing, the image processing apparatus 10 can display the AR content associated with the marker on the display unit 14 as shown in FIG. For a marker not associated with AR content, a message indicating that AR content is not associated is displayed.

  Depending on the AR content, the display position of the message, and the shooting angle of the marker, any of the AR content, the marker, and the message may overlap on the screen. In this case, if the AR content and the message are made translucent, the message is moved, the size is changed, or only the marker is displayed in a separate frame, the subsequent selection process can be easily performed.

  Next, the AR content selection process 105 of the image processing apparatus 10 prompts the user to select a part of the AR content superimposed on the marker. This selection method is preferably changed according to the type of AR content to be displayed.

  For example, FIG. 10A shows a case where a number of images are associated with the marker. In this case, one or more desired images may be selected via the instruction unit 15. If the image processing apparatus 10 has a shape as shown in FIG. 2, a desired image portion in a large number of images displayed on the AR content display unit is specified by touching with a finger as shown in FIG. Can be realized.

  FIG. 10B shows a case where an electronic document is associated with the marker as AR content. In this case, a desired part in the electronic document may be selected via the instruction unit 15. If the image processing apparatus 10 has a shape as shown in FIG. 2, a desired part of the electronic document displayed on the AR content display unit is swiped with a finger as shown in FIG. 10B to designate a rectangle. Can be realized.

  FIG. 10C shows a case where a 3D object is associated with a marker. In this case, a desired part in the 3D object may be selected via the instruction unit 15. If the image processing apparatus 10 has a shape as shown in FIG. 2, a desired part portion of the 3D object displayed on the AR content display unit is specified by touching with a finger as shown in FIG. 10C. Can be realized.

  Next, the selected AR content link instruction processing 106 of the image processing apparatus 10 prompts the user to perform an instruction for associating AR content with a marker for which no corresponding AR content exists.

  The AR content to be associated here is the partial AR content selected in the AR content selection process 105 as the previous process. For example, in the example of FIG. 9, the association instruction can be realized by dragging the partial AR content and dropping it in the drop area where the message was displayed. In addition, this process can be realized by an operation such as touching a marker for which no corresponding AR content exists.

  Through the above processing, the user designates a desired partial AR content and a link destination marker that associates the partial AR content. This process finally transmits the partial AR content information and the link destination marker information to the AR content management apparatus 20 via the communication unit 13, and the process ends.

  Finally, the AR content management apparatus 20 associates and stores the partial AR content information received from the image processing apparatus 10 via the communication unit 22 and the link destination marker information.

  By performing the above processing, it is possible to associate a new AR content obtained by editing the existing AR content with the feature information (marker) of the physical object that is not associated with the AR content.

<Embodiment 2>
In the first embodiment, the selection operation is described as an example of the editing operation of the AR content information in the AR content selection processing 105, but other editing operations may be performed. In this embodiment, other editing operations will be described.

  As an example, a method for changing the display mode of the AR content for the link destination marker will be described. FIG. 11 shows an example of a UI screen for performing the editing operation.

  First, UI1 in FIG. 11 is a live-action image display unit and AR content display position designation unit. The user touches and slides the AR content portion of the screen to determine the display position for the link destination marker.

  Next, UI2 in FIG. 11 is an AR content display angle designation unit. The user moves the slide bar in this part up and down to determine the display angle of the link destination marker. This display angle can be specified for each of the x-axis direction, the y-axis direction, and the z-axis direction. Note that, as shown in the UI1 screen of FIG. 11, the xyz coordinate system may be one based on the link destination marker.

  Finally, UI3 in FIG. 11 is an AR content display size designation unit. By rotating the bar on the displayed ring, the display size of the AR content with respect to the link destination marker can be designated. In addition to this, more detailed designation such as changing the shape or color of the AR content may be performed.

  As an example, a method for selecting and laying out a plurality of partial AR contents for a link destination marker will be described. FIG. 12 shows an example of a UI screen for performing an operation for associating a plurality of AR contents. The UI 4 in FIG. 12 lists the history of AR content selected so far. The user can instruct to link the AR content to the link destination marker by dragging and dropping the desired AR content with the finger. At this time, since the marker may be reflected on a plurality of screens, it is desirable to provide an effect on the screen to indicate which link destination marker being processed corresponds to as shown in FIG. In addition, regarding the layout of each AR content, a UI for performing the editing operation described above may be used.

  By performing the editing operation as described above, the user can instruct a more detailed AR content form.

<Embodiment 3>
In the first embodiment, a method of selecting a part of AR content and associating it with a link destination marker has been described. However, this method does not take into account the user authority for association. If the method of the first embodiment is allowed for all users, there is a problem in terms of security in the case of AR content with confidentiality. In the present embodiment, in view of such circumstances, a technique for restricting processing according to the authority of the user will be described below. The processes other than those described below may be the same as those in the first embodiment.

  First, as a precondition, it is assumed that the AR content management apparatus 20 records information as shown in FIG. That is, in addition to the correspondence between the marker and the AR content, the display position, etc., the user ID having the authority to display and copy the AR content is recorded. This recording operation itself is performed in advance by the AR content manager.

  Then, in the AR content information acquisition process 3103 in the image processing apparatus 10, in addition to the extracted marker, the user ID is also transmitted from the own device to the AR content management apparatus 20, and the AR content information as shown in the conceptual diagram of FIG. receive. This AR content information includes an AR content display and duplication authority flag in addition to the flag corresponding to the AR content of each marker, the AR content, and the display position.

  In the AR content information generation process 3201 of the AR content management apparatus 20, it is determined whether or not each marker has a corresponding AR content and whether or not it has the authority to display and copy the received device ID. AR content information as shown in the conceptual diagram is generated.

  Then, the AR content management device 20 transmits the generated AR content information to the image processing device 10 in the AR content information transmission processing 3202.

  Then, the image processing apparatus 10 displays an image on the display unit 14 using the AR content information received in the display process 3104. Details of the display processing 3104 will be described with reference to the processing flow of FIG.

  First, in step S31041 in FIG. 15, a variable n indicating the number of the marker being processed is initialized with 0, and the process proceeds to step S31042.

  In step S31041, the AR content correspondence flag in the AR content information corresponding to the marker Mn being processed is first referred to. If the flag is TRUE, that is, if the corresponding AR content exists, the process proceeds to step S31043. If the flag is FALSE, that is, the corresponding AR content does not exist, the process proceeds to step S31044.

  In step S31043, the display authority flag of the corresponding AR content is referred to the marker Mn being processed. If the display authority flag is OK, the process proceeds to step S31045; otherwise, the process proceeds to step S31046.

  In step S31044, a message indicating that there is no corresponding AR content is displayed for the marker Mn being processed, and the process proceeds to step S31047.

  In step S31045, the AR content is superimposed and drawn on the marker Mn being processed using the AR content information. When the AR content is superimposed and drawn, the process proceeds to step S31047.

  In step S31046, a message indicating that the corresponding AR content is not displayed is displayed as a message for the marker Mn being processed, and the process proceeds to step S31047.

  In step S31047, it is determined whether the number Nmax of markers in the image acquired by the image acquisition process is larger than a variable n indicating the number of the marker being processed. If larger, the process proceeds to step S31048. If not, the process ends.

  In step S31048, 1 is added to the variable n indicating the number of the marker being processed, and the flow advances to step S31042.

  With the processing up to this point, it is possible to display AR content for a user with display authority without displaying AR content for a user without display authority.

  Then, in the AR content editing processing 3105 in the image processing apparatus 10, only the AR content whose copy authority flag is OK among the displayed AR content is to be edited. In the example of FIG. 14, since the duplication authority for both of the two markers is NG, editing cannot be performed with the device being operated. The editing process itself may be the same as in the first and second embodiments.

  In the AR content link instruction processing 3106, only those whose copy authority flag is OK are set as targets for which a link instruction can be made. The instruction process itself may be the same as in the first and second embodiments.

  By performing the processing as described above, it is possible to restrict display of AR content and copying of existing AR content.

<Embodiment 4>
In the first and second embodiments, the AR content is selected and edited through a user operation. Thereby, the user can associate the AR content with the marker in a desired form. However, on the contrary, it is only necessary to copy an important part of the AR content, and for the user who does not care about details such as the form, the operations described in the first and second embodiments become a burden.

  In order to respond to such a request, this embodiment will explain a method for reducing the operation load on the user.

  First, as a precondition, an AR content manager designates an important part and form of AR content to be duplicated in advance as default and records it in the AR content management apparatus 20. The processing in the image processing apparatus 10 may be the same as that in the first embodiment up to the image acquisition processing, feature extraction processing, AR content information acquisition processing, and display processing. Next, in the AR content selection process, an AR marker as a duplication source and an AR marker as a duplication destination are designated. This may be instructed by tapping an AR marker on the screen in the UI, or may be realized by taking two markers at the same time with the camera set in the AR content duplication mode, or taking two separate shots. . Then, the AR content associated with the AR marker that is the copy destination may be associated with the marker using the part and form of the AR content specified by the AR content administrator as a default in advance.

  By performing the processing as described above, the user can easily copy a part of the AR content simply by photographing the marker.

<Other embodiments>
In the first embodiment, a method of selecting a part of AR content and associating it with a link destination marker has been described. In addition, by deleting the selected AR content portion from the link source of the AR content, it is possible to realize an effect such as cut and paste, which is referred to as a PC operation.

  In the first embodiment, in the selected AR content link instruction processing 106, an AR content association instruction is issued to a marker for which no AR content exists. However, AR content can also be associated with a marker that already has corresponding AR content by the same operation as in the first embodiment. In this case, the existing AR content is overwritten.

  In the third embodiment, the copy is restricted depending on whether or not there is an authority. However, the AR content manager can limit the random diffusion of the AR content by setting the number of times the AR content can be copied in advance. . In this case, every time a link instruction is issued from the image processing apparatus, the number of times the AR content can be copied is subtracted, and when it becomes 0, the copy is disabled. If the number of duplication requests is counted and the number of duplication times is exceeded, duplication may be prohibited.

  Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist of the present invention.

  The present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

1 Display 2 Video I / O
3 CPU
4 ROM
5 RAM
6 System bus 7 I / O
8 Touch panel 9 I / O
10 Imaging device 11 I / O
12 External recording medium 13 I / O
14 Recording device 15 Communication I / F

Claims (10)

An image processing apparatus,
Extraction means for extracting the first feature information and the second feature information;
Obtaining means for obtaining extended information associated with the first feature information;
Generating means for generating new extension information from the extension information;
An image processing apparatus comprising: association means for associating the new extension information with the second feature information.   The image processing apparatus according to claim 1, wherein the new extension information is a part of the extension information.   The image processing apparatus according to claim 1, wherein extended information is not associated with the second feature information extracted by the extraction unit.   2. The extension information is already associated with the second feature information extracted by the extraction unit, and new extension information is overwritten by the association by the association unit. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the association unit performs the association in the case of an operation of an authorized user or an operation of an authorized image processing apparatus.   The image processing apparatus according to claim 1, wherein the association unit refers to a predetermined number of times of duplication possible for the extended information, and performs association when the number of duplication times remains.   The image processing apparatus according to claim 1, wherein the generation unit generates new extended information by an editing operation preset by an administrator of extended information.   The image processing apparatus according to claim 1, wherein the feature information is a QR code, a SIFT feature amount, or a SURF feature amount. An image processing method comprising:
An extraction step of extracting the first feature information and the second feature information;
An acquisition step of acquiring extended information associated with the first feature information;
A generation step of generating new extension information from the extension information;
And an associating step of associating the new extended information with the second feature information. Computer
Extraction means for extracting the first feature information and the second feature information;
Obtaining means for obtaining extended information associated with the first feature information;
Generating means for generating new extension information from the extension information;
Association means for associating the new extended information with the second feature information;
Program to function as.

Images