WO2015072194A1

WO2015072194A1 - Display control device, display control method and program

Info

Publication number: WO2015072194A1
Application number: PCT/JP2014/071386
Authority: WO
Inventors: 俊一笠原
Original assignee: ソニー株式会社
Priority date: 2013-11-13
Filing date: 2014-08-13
Publication date: 2015-05-21
Also published as: EP3070681A1; EP3070681A4; US20160210788A1; JPWO2015072194A1; JP6337907B2; US10074216B2

Abstract

[Problem] To display more useful information by utilizing real objects contained in an image on the basis of the display position or orientation of the real objects so contained. [Solution] This display control device is provided with: a record reference unit which refers to records that associate an image, an object contained in the image, and the display position or orientation of the object in the image; a display control unit which, on the basis of the aforementioned record, shifts the display of the first image containing the object onto the display of a second image different from the first image and containing the object, while maintaining the display position or orientation of the object.

Description

Display control apparatus, display control method, and program

The present disclosure relates to a display control device, a display control method, and a program.

In recent years, a technique called augmented reality (AR) that superimposes additional information on a real image and presents it to a user has attracted attention. In the AR technique, for example, an object included in a captured image is recognized, and information related to the recognized object is displayed. Such information is also called annotation, and is visualized as various forms of virtual objects such as text, icons, or animations. An example of such an AR technique is described in Patent Document 1, for example.

JP 2013-105253 A

However, although Patent Document 1 proposes a technique for appropriately displaying a virtual object related to a real object in accordance with the position or orientation of the object, a technique for display utilizing the real object itself is proposed. Not. As described in Patent Document 1, since it is possible to detect the position and orientation of an object, it should be possible to display more useful information by utilizing the actual object itself. It is.

Therefore, in the present disclosure, a new and improved display control apparatus capable of displaying more useful information by utilizing the actual object itself based on the display position or orientation of the actual object included in the image. A display control method and program are proposed.

According to the present disclosure, a record reference unit that refers to a record that associates an image, an object included in the image, a display position or a posture of the object in the image, and a second object including the object based on the record. A display control device that includes a display control unit that transitions display of one image to display of a second image that includes the object and is different from the first image, while maintaining the display position or orientation of the object. Provided.

In addition, according to the present disclosure, referring to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image, and based on the record, There is provided a display control method including transition of display of one image to display of a second image including the object and different from the first image while maintaining the display position or orientation of the object. The

In addition, according to the present disclosure, a function that refers to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image, and a function that includes the object based on the record. A program for causing a computer to realize a function of transitioning display of one image to display of a second image that includes the object and is different from the first image while maintaining the display position or orientation of the object Is provided.

As described above, according to the present disclosure, it is possible to display more useful information by utilizing the actual object itself based on the display position or orientation of the actual object included in the image.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure showing an outline of device composition in one embodiment of this indication. It is a block diagram showing a schematic structure of a smart phone concerning one embodiment of this indication. It is a figure showing an example of an image which an imaging part generates in one embodiment of this indication. 5 is a diagram illustrating an example of a record generated in an image-object DB according to an embodiment of the present disclosure. FIG. FIG. 10 is a diagram illustrating a first display example according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating a second display example according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating a third display example according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating a fourth display example according to an embodiment of the present disclosure. FIG. 16 is a diagram illustrating a fifth display example according to an embodiment of the present disclosure. 5 is a diagram for describing an example of processing for displaying an annotation according to an embodiment of the present disclosure. FIG. FIG. 18 is a diagram illustrating a sixth display example according to an embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a hardware configuration example of an information processing apparatus according to an embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

The description will be made in the following order.
1. Device configuration2. Display example 2-1. Change in object display position or orientation 2-2. Transition between images 2-3. Annotation display 2-4. 2. Display of destination options Hardware configuration Supplement

(1. Device configuration)
FIG. 1 is a diagram illustrating an outline of a device configuration according to an embodiment of the present disclosure. Referring to FIG. 1, in this embodiment, the smartphone 100 images a real space 200 including an object 201. On the display unit 118 of the smartphone 100, a through image 300t generated by imaging is displayed. The through image 300t includes an object 301 corresponding to the actual object 201.

FIG. 2 is a block diagram illustrating a schematic configuration of a smartphone according to an embodiment of the present disclosure. Referring to FIG. 2, the smartphone 100 includes an imaging unit 102, an object recognition unit 104, an object DB 106, a record generation unit 108, a record reference unit 110, an image-object DB 112, a display control unit 114, and an image. DB 116 and display unit 118 are included. The smartphone 100 is an example of a display control device according to an embodiment of the present disclosure, and may be realized by, for example, a hardware configuration of an information processing device described later. Hereinafter, each component will be further described.

In the present embodiment, the smartphone 100 includes at least a processor such as a CPU (Central Processing), a memory or storage, a camera (imaging device), and a display (output device). For example, the object recognition unit 104, the record generation unit 108, the record reference unit 110, and the display control unit 114 can be realized by the processor operating according to a program stored in a memory or storage. The object DB 106, the image-object DB 112, and the image DB 116 can be realized by memory or storage. The imaging unit 102 can be realized by a camera. The display unit 118 can be realized by a display.

The imaging unit 102 captures an actual space and generates an image. The imaging unit 102 may generate a still image or a moving image. The image data generated by the imaging unit 102 is provided to the object recognition unit 104 and stored in the image DB 116 as necessary. The image data may be provided to the display control unit 114 and displayed on the display unit 118 as a through image or a preview image.

The object recognition unit 104 performs object recognition on the image generated by the imaging unit 102. The object recognition unit 104 may refer to the object DB 106 for object recognition. In the object DB 106, for example, model data relating to the shape or appearance of an object to be recognized is stored in advance. The model data includes data defining the shape of each object, image data such as a predetermined symbol mark or text label attached to each object, or feature set data extracted from a known image for each object. .

More specifically, for example, the object recognition unit 104 recognizes which object is included in the input image by using the image generated by the imaging unit 102 as the input image. For example, the object recognition unit 104 collates a set of feature points extracted from the input image with the shape of the object defined by the model data. Further, the object recognition unit 104 may collate image data such as a symbol mark or a text label defined by the model data with the input image. Further, the object recognizing unit 104 may collate the feature amount of the known object image defined by the model data with the feature amount extracted from the input image.

Here, when a known object is recognized in the new input image, the object recognition unit 104 uses the feature amount set representing the recognized object as data of the feature amount set extracted from the known image for the object. May be added. When a new object is recognized in the new input image, the object recognition unit 104 adds new model data to the object DB 106 based on the shape, image data, feature amount, etc. of the recognized object. Also good.

Furthermore, the object recognition unit 104 can recognize the display position and orientation of the object in the image. More specifically, for example, the object recognition unit 104 detects the display position and orientation of the object included in the input image using the image generated by the imaging unit 102 as the input image. For example, the posture of the object is a 4-by-4 matrix that represents the transformation between the model coordinate system in the model data stored in the object DB 106 and the coordinate system of the object shown in the input image. Is expressed in an integrated manner. The object recognition unit 104 can extract the angle of the object with respect to the smartphone 100 from the homogeneous transformation matrix. Further, the display position of the object in the image can be represented by, for example, the center coordinates of the object in the image.

The record generation unit 108 generates a record in the image-object DB 112 based on the result of object recognition by the object recognition unit 104. In the image-object DB 112, for example, a record that associates an image, an object included in the image, and a display position and orientation of the object in the image is generated. For example, as shown in FIG. 3, when the object recognition unit 104 recognizes two

objects

301a and 301b in the image 300 generated by the imaging unit 102, the record generation unit 108 displays the image-object DB 106 in FIG. Add a record like

FIG. 4 is a diagram illustrating an example of a record generated in the image-object DB according to an embodiment of the present disclosure. FIG. 4 illustrates a record including an image ID, an object ID, a position, and a posture as a record generated in the image-object DB 112. In the example shown, the record is unique for the combination of image ID and object ID. As described above, in the example illustrated in FIG. 3, the object recognition unit 104 recognizes two known

objects

301a and 301b in the image 300a generated by the imaging unit 102. Therefore, the record generation unit 108 adds two

records

401 a and 401 b as shown in FIG. 4 to the image-object DB 112.

For example, the record 401a is a record for the object 301a (object ID = obj_000001) included in the image 300a (image ID = img_000001), and that the image 301a includes the object 301a and the display position of the object 301a in the image 300a. (coordinates of the image (X, Y) = (0.276,0.843 )) and, to record in association with the posture of the object 301a (represented by homogeneous transformation matrices _{M 1)} in the image 300a Yes.

As described above, the record of the image-object DB 112 is unique for the combination of the image ID and the object ID. Therefore, as shown in the figure, if a plurality of

objects

301a and 301b are found in the same image 300a, a plurality of

records

401a and 401b corresponding to the objects can be generated. Furthermore, if at least one of the object 301a or the object 301b is found in another image 300b, a new record 401 is generated again.

Referring to FIG. 2 again, the record reference unit 110 refers to the record in the image-object DB 112 based on the result of object recognition by the object recognition unit 104. Referenced here is a record for another image including the object recognized by the object recognition unit 104. That is, as in the example of FIGS. 3 and 4 above, when the object 301a is recognized in the image 300a, the record reference unit 110 refers to a record for an image including the object 301a with an image other than the image 300a. . For example, the record reference unit 110 may refer to a record by issuing a query “image ID ≠ img_000001 AND object ID = obj_000001” in the image-object DB 112 including the record as shown in FIG.

Alternatively, the record reference unit 110 may refer to the record in the image-object DB 112 in response to a request from the display control unit 114. Here, for example, a record about an image including an object designated by the display control unit 114 is referred to. For example, when the record of the image-object DB 112 as shown in FIG. 4 is generated, the record reference unit 110 designates an object ID and refers to the record, thereby referring to a plurality of images including a specific object. It is possible to acquire all the records at once.

The display control unit 114 controls display of an image on the display unit 118. For example, the display control unit 114 may cause the display unit 118 to display an image generated by the imaging unit 102 (hereinafter also referred to as a captured image) as a through image or a preview image. Further, for example, the display control unit 114 may read out an image stored in the image DB 116 (hereinafter referred to as a recorded image) in accordance with an operation of the user of the smartphone 100 and display the image on the display unit 118. At this time, the display control unit 114 may process and display the captured image or the recorded image as necessary. In the present embodiment, in the captured image and the recorded image, for example, the object included in the image is recognized by the object recognition unit 104, so the display control unit 114 changes the display position of the object included in the captured image or the recorded image. (The image itself may be moved) or the posture of the object can be changed.

For example, the display control unit 114 changes the display of the first image (captured image or recorded image) including a certain object to the display of the second image (recorded image different from the first image) including the same object. You may let them. At this time, the display control unit 114 can change the display of the image while maintaining the display position or orientation of the object, for example, by changing the display position or orientation of the object in the first image or the second image. it can.

Such a display includes, for example, the second image of the transition destination (the same object as the first image stored in the image DB) based on the record referred to by the record reference unit 110. This is possible by identifying the image) and recognizing the display position and orientation of the object included in the second image. For the first image, the display control unit 114 displays the result of object recognition by the object recognition unit 104 (when the first image is a captured image) or another record (first image acquired by the record reference unit 110). Display position and orientation based on the recorded image).

In the image DB 116, for example, an image generated by the imaging unit 102 is stored in response to an operation of the user of the smartphone 100. The image DB 116 may store an image acquired from an external device (not shown) via a network. The image DB 116 may store an image content entity, or may store link information to the image content entity.

Here, in order to enable the processing of the record reference unit 110 and the display control unit 114 as described above, the image ID given to the image stored in the image DB 116 and the image ID in the image-object DB 112 are common. Or it is desirable that it is convertible. Similarly, it is desirable that the object ID given to the model data stored in the object DB 106 and the object ID in the image-object DB 112 are common or convertible.

Heretofore, the device configuration according to an embodiment of the present disclosure has been described. In the above example, a smartphone is exemplified as the display control device, but the embodiment of the present disclosure is not limited to such an example. For example, the display control device may be various devices including a display and an input device such as a desktop or notebook personal computer, a television, a tablet terminal, a media player, or a game machine. The display control apparatus does not necessarily include an imaging unit, and for example, acquires an image shared exclusively via a network, or acquires an image accumulated in a storage by reading from a removable medium. There may be.

Also, as another device configuration in the embodiment of the present disclosure, at least a part of the above configuration may be realized in a server on a network. For example, the smartphone 100 (which may be another terminal device) includes an imaging unit 102 and a display unit 118, and other components such as an object recognition unit 104, an object DB 106, a record generation unit 108, a record reference unit 110, an image The object DB 112, the display control unit 114, and the image DB 116 may be realized on a server on the network. In this case, the server is an example of a display control device. Further, in the same apparatus configuration as described above, some or all of the object DB 106, the image-object DB 112, and the image DB 116 may be realized in a storage on a network.

(2. Display example)
(2-1. Changes in object display position or orientation)
FIG. 5 is a diagram illustrating a first display example according to an embodiment of the present disclosure. In this display example, first, as shown in A, an image 310 including objects 311a to 311h is displayed. Here, when the user designates the object 311c by an operation via the touch panel or the like (for example, a double tap), as shown in B, the object 311c is highlighted and the postures of the objects 311a to 311h are mutually positioned. It changes while maintaining the relationship, and the object 311c faces the screen. Further, as shown in C, when the image 310 zooms in on the object 311c in a directly-facing state, the object 311c is enlarged and displayed.

FIG. 6 is a diagram illustrating a second display example according to an embodiment of the present disclosure. In this display example, as shown in A, an image 310 including objects 311a to 311h is displayed (similar to FIG. 5). Here, when the user designates the object 311h by an operation (for example, a double tap) via the touch panel or the like, as shown in B, the object 311h is highlighted and the postures of the objects 311a to 311h are in a positional relationship. The object 311h is in a directly facing state while changing. Further, when a predetermined operation (for example, drag / pinch-in or double tap) is further acquired on the object 311h in this state, as shown in C, the image 310 is centered on the object 311h in the directly-facing state. Is zoomed in, the object 311h is enlarged.

As shown in the above two examples, in this embodiment, an image is displayed so that the orientation of the object specified in the image is changed to make the object face the screen or the specified object is displayed at the center. The display range can be moved, or the image can be zoomed in so that the designated object is enlarged.

Such a display change is caused by a record in the image-object DB 112 referred to by the record reference unit 110 by the display control unit 114 (a record that associates an image, an object included in the image, and a display position or orientation in the object image). This is made possible by processing an image acquired from the imaging unit 102 or the image DB 116 based on the above.

More specifically, for example, the display control unit 114 deforms the image of the object portion so that each object faces the screen based on information indicating the posture of each object included in the image. Further, the display control unit 114 rearranges the deformed objects so that the positional relationship before the deformation is reproduced based on the information indicating the display position of each object in the image. Furthermore, the display control unit 114 moves the display range of the image so that the specified object is displayed at the center based on the information indicating the display position of each object in the image, or enlarges the specified object. Or zoom in on the image.

(2-2. Transition between images)
FIG. 7 is a diagram illustrating a third display example according to an embodiment of the present disclosure. In this display example, first, as shown in A, in the image 310 (first image), the object 311c is displayed facing up and enlarged. This display may be displayed through the series of displays described above with reference to FIG. Here, as shown in B, automatically or in response to a user's predetermined operation, the display of the image 310 transitions to the display of a different image 320 (second image). The image 320 includes an object 321c. The object 321c is the same object as the object 311c (that is, recorded in the image-object DB 112 with the same object ID). As shown in C described later, the image 320 includes

objects

321i and 321j in addition to the object 321c.

At this time, the display positions and orientations of the

objects

311c and 321c are maintained through the display of the image 310 shown in A and the display of the image 320 shown in B. That is, in the image 310 of A, the object 311c is arranged in the vicinity of the center, whereas in the image 320 of B, the object 321c is arranged in the vicinity of the center. Thereafter, automatically or in response to a predetermined operation by the user, as shown in C, the display of the image 320 changes to a state where the object 311c is not directly facing and is not displayed at the center. In the example shown, this is the original state of the image 320. That is, in B, the image 320 is processed and displayed so that the posture and display position of the object 311c in the image 310 before the transition are also maintained in the object 321c displayed after the transition.

As described above, in the present embodiment, the display control unit 114 changes the posture of the object to a predetermined posture (for example, directly facing) in the display of the first image, and continues after the transition to the display of the second image. It is possible to seamlessly express transitions between images including the same object by maintaining the posture of at least temporarily. Accordingly, for example, the display can be changed from the image 310 including the objects 311a to 311h to the image 320 including the

objects

321c, 321i, and 321j through the

objects

311c and 321c, and is not included in the image 310. Information regarding the

objects

321i and 321j can be newly obtained.

For example, in the example of FIG. 7 described above, by selecting the object 321j this time in the state shown in C, a transition to another image including the same object as the object 321j may be executed. In the present embodiment, in this way, the transition of the image using the object as a medium is repeated while changing the target object as necessary, thereby forming a link between the object or the information between the images based on a new context. be able to.

As another example, in the display of the image 310 (first image), the display control unit 114 changes the display position and orientation of the object 311c to the original state of the image 320 (second image) (the state shown in FIG. 7C). The display of the image 310 may be changed to the display of the image 320 after changing to the display position and orientation similar to those of the object 321c in the display. Conversely, the display control unit 114 does not change the display position and orientation of the object 311c in the display of the image 310 (first image) (without facing directly or zooming in as shown in FIG. 5), instead, In the display 320, the transition may be executed after changing the display position and orientation of the object 321c to the same display position and orientation as the object 311c in the display on the image 310.

FIG. 8 is a diagram illustrating a fourth display example according to an embodiment of the present disclosure. In this display example, first, as shown in A, an image 330 (web page) including an object 331a displayed as a graphic is displayed. In this state, for example, when the user designates the object 331a and executes a predetermined operation, the display of the image 330 transitions to a different image 340 (photograph) as shown in B. The image 340 includes an object 341a shown in the photograph. The object 341a is the same object as the object 331a (that is, recorded in the image-object DB 112 with the same object ID).

As described above, in this embodiment, the images before and after the transition may be either virtual images such as web pages or captured images (photos). As in the above display example, transition may be performed between the virtually configured image (web page) and the captured image (photo), or virtually configured images, or Transition may be performed between captured images. Although not shown in FIG. 8, also in this example, either or both of the image 330 and the image 340 are processed so that the display positions or postures of the

objects

331a and 341a are maintained before and after the transition. sell.

(2-3. Annotation display)
FIG. 9 is a diagram illustrating a fifth display example according to an embodiment of the present disclosure. In this display example, as shown in A, annotations 352 given by a plurality of users are displayed in association with the object 351 (poster) included in the image 350 in a cumulative manner. An annotation can also be treated as an object, but in this example, the object 351 and the annotation 352 are distinguished for the sake of explanation. As shown in B, the annotation 352 can be scrolled and displayed while maintaining the posture displayed superimposed on the object 351 (in contrast to the annotations 352a to 352e displayed in A). In B, annotations 352c to 352g are displayed). Further, as shown in C, the annotation 352 can be displayed facing the object 351 independently of the object 351.

In this embodiment, in this embodiment, an annotation 352 input by a plurality of users to the same object 351 included in different images 350 is recorded together with a relative positional relationship with respect to the object 351 in each image 350. It becomes possible by being. Hereinafter, an example of processing for such display will be further described with reference to FIG.

FIG. 10 is a diagram for describing an example of processing for displaying an annotation according to an embodiment of the present disclosure. In the illustrated example, the annotation 362a is input by the user A to the object 361a included in the image 360a. Here, the annotation 362a is recorded by, for example, the object ID of the object 361a, the relative position of the annotation 362a with respect to the object 361a, the relative angle of the annotation 362a with respect to the object 361a, and the content (text or image) of the annotation 362a. The

Meanwhile, another user B is referring to another image 360b including the same object 361b as the object 361a. In the image 360b, the object 361b is displayed at a different display position and orientation from the object 361a in the image 360a. For example, the object 361b can be associated with the object 361a by being recorded in the image-object DB 112 with the same object ID. Is possible. Furthermore, since the annotation 362a input by the user A is recorded together with the relative position and angle with respect to the object 361a, the annotation 362a can be displayed on the image 360b with reference to the object 361b.

More specifically, the display control unit 114 calculates the display position and orientation difference between the object 361a in the image 360a and the object 361b in the image 360b based on the image-object DB record acquired by the record reference unit 110. By calculating and adding the calculated difference to the relative position or angle of the annotation 362a with respect to the object 361a, the annotation 362a can be displayed in the image 360b with reference to the object 361b.

Here, the user B may additionally input an annotation 362b to the object 361b included in the image 360b. Similarly to the annotation 362a, the annotation 362b also has the object ID of the object 361b (same as the object 361a), the relative position of the annotation 362b with respect to the object 361b, the relative angle of the annotation 362b with respect to the object 361b, and the content of the annotation 362b ( Text or image). As a result, as shown in the drawing, even in the image 360a that the user A is referring to, the annotation 362b can be displayed with the object 361a as a reference.

Furthermore, the annotations input by each of the users A and B may be displayed in an image 360c (web page) that is referred to by another user C. In this case, both the object 361c and the

annotations

362a and 362b may be displayed facing each other in the image 360c.

The processing for displaying the object 361 and the annotation 362 as described above may be executed by a server on a network that provides services to the user A, the user B, and the user C, for example. That is, in this case, the server on the network may be an example of a display control device.

(2-4. Display of transition destination options)
FIG. 11 is a diagram illustrating a sixth display example according to an embodiment of the present disclosure. In this display example, in the image 370, an option icon 373 of an image that can be changed using the object 371 as a medium is displayed. In other words, in the illustrated example, a plurality of images including the same object as the object 371 are stored in the image DB 116, and the user can select which one to transition to using the option icon 373. In the illustrated example, four option icons 373a to 373d are displayed.

The option icon 373 is displayed, for example, when the user performs a predetermined operation (for example, long press) on the object 371 via the touch panel or the like. When the user selects one of the plurality of option icons 373, the transition from the image 370 to the corresponding image is executed. Even in the transition in this case, the display position or orientation of the object 371 may be maintained between the image 370 before the transition and the image after the transition.

Here, the selection icon 373 may display, for example, a predetermined number of images randomly selected from images that can be transitioned. Alternatively, the option icon 373 may display an image selected under a predetermined condition. In the illustrated example, the option icon 373a has the highest resolution image, the option icon 373b has the highest recommendation level, the option icon 373c has the shooting location closest to the shooting location image 370, and the option icon 373d. The images with the highest evaluations of other users are respectively displayed.

When determining an image to be displayed as the option icon 373 based on attributes such as resolution, recommendation level, shooting location, or evaluation of other users as in the above example, for example, in the image DB 116, Information indicating the attribute is stored in association with each image ID.

As described above, by displaying an image that can be transitioned using the object 371 included in the image 370 as an option icon 373, for example, it is easier for the user to reach desired information by transition of the image. Can be.

According to some display examples of the embodiment of the present disclosure as described above, a state in which a plurality of objects are present at the same place is regarded as one context, and an application sharing the state can be realized. is there.

As an example, when a user makes a transition from an image of a book placed on a desk to another image containing the same book as an object, the book image on the bookshelf in the bookstore (taken by another user) And can be an image shared on the network), it is possible to understand the context of what kind of book is selected as the book related to the book.

In addition, if you can transition to images where the books are arranged in multiple bookstores, offices, personal bookshelves, etc., you can get information on other books related to the books selected from various viewpoints. it can. In such a transition, the transition is seamlessly executed while the display position or orientation of the object (book) is maintained, so that the user can confirm that the image display is executed through the object (book). It can be easily recognized.

Also, for example, when a user's viewpoint image is continuously recorded by a camera mounted on a wearable display worn by the user, a huge amount of images can be recorded. For example, if the user feels that an object has been seen but cannot remember where it was seen, take a sample image of the object, search for and display the same object as the object included in the sample image, and display that object. The user's situation at the time of viewing can be accurately recalled. When the viewpoint image is shared via the network, other users can access the user's viewpoint image through the object.

In the above description of the embodiment and the display example, it is assumed that both the display position and the posture of the object are maintained during the transition of the image. However, the embodiment of the present disclosure is not limited to such an example, and the image At the time of transition, only the display position or the posture may be maintained. Even in that case, it may be possible to make the user recognize that the display of the image is executed through the object. In this case, the image-object DB may store a record associating the image, the object included in the image, and any one of the display position or the posture in the image of the object.

(3. Hardware configuration)
Next, a hardware configuration of the information processing apparatus according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 12 is a block diagram illustrating a hardware configuration example of the information processing apparatus according to the embodiment of the present disclosure. The illustrated information processing apparatus 900 can realize, for example, a display control apparatus such as a smartphone or a server in the above embodiment.

The information processing apparatus 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Furthermore, the information processing apparatus 900 may include an imaging device 933 and a sensor 935 as necessary. The information processing apparatus 900 may include a processing circuit called DSP (Digital Signal Processor) or ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

The CPU 901 functions as an arithmetic processing device and a control device, and controls all or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. The CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 configured by an internal bus such as a CPU bus. Further, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

The input device 915 is a device operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an external connection device 929 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 900.

The output device 917 is a device that can notify the user of the acquired information visually or audibly. The output device 917 can be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro-Luminescence) display, an audio output device such as a speaker and headphones, and a printer device. . The output device 917 outputs the result obtained by the processing of the information processing device 900 as video such as text or an image, or outputs it as audio such as voice or sound.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing apparatus 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. In addition, the drive 921 writes a record in the attached removable recording medium 927.

The connection port 923 is a port for directly connecting a device to the information processing apparatus 900. The connection port 923 can be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like. The connection port 923 may be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting the external connection device 929 to the connection port 923, various types of data can be exchanged between the information processing apparatus 900 and the external connection device 929.

The communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication. The communication device 925 transmits and receives signals and the like using a predetermined protocol such as TCP / IP with the Internet and other communication devices, for example. The communication network 931 connected to the communication device 925 is a wired or wireless network, such as the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

The imaging device 933 uses various members such as an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and a lens for controlling the formation of a subject image on the imaging element. It is an apparatus that images a real space and generates a captured image. The imaging device 933 may capture a still image or may capture a moving image.

The sensor 935 is various sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, and a sound sensor. The sensor 935 acquires information about the state of the information processing apparatus 900 itself, such as the posture of the information processing apparatus 900, and information about the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900, for example. To do. The sensor 935 may include a GPS sensor that receives a GPS (Global Positioning System) signal and measures the latitude, longitude, and altitude of the apparatus.

Heretofore, an example of the hardware configuration of the information processing apparatus 900 has been shown. Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.

(4. Supplement)
Embodiments of the present disclosure include, for example, a display control device (such as a smartphone or a server) as described above, a system, a display control method executed by the display control device or system, a program for causing the display control device to function, And a non-transitory tangible medium on which the program is recorded.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1) a record reference unit that refers to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. A display control unit.
(2) The display control unit changes the posture of the object to a predetermined posture in displaying the first image, and at least temporarily changes the predetermined posture even after the transition to the display of the second image. The display control device according to (1), wherein the display control device is maintained.
(3) The display control device according to (2), wherein the predetermined posture is a directly-facing posture.
(4) The display control unit changes the display position or orientation of the object so as to correspond to the display position or orientation of the object in the display of the second image in the display of the first image. The display control device according to (1), wherein the display of the first image is changed to the display of the second image.
(5) The display control unit changes the display of the first image so that the display position or orientation of the object corresponds to the display position or orientation of the object in the display of the first image. The display control apparatus according to (1), wherein the display is switched to display of the second image.
(6) The display control apparatus according to any one of (1) to (5), wherein at least one of the first image and the second image is a captured image.
(7) The display control according to any one of (1) to (6), wherein at least one of the first image and the second image is a virtually configured image. apparatus.
(8) The display control according to any one of (1) to (7), wherein the display control unit displays, in the first image, options of an image that can be transitioned as the second image. apparatus.
(9) The display control device according to (8), wherein the display control unit displays the options for each attribute of the image.
(10) The display control unit causes the annotation given to the object in each of the first image and the second image to be displayed in association with the object cumulatively based on the record. The display control apparatus according to any one of 1) to (8).
(11) referring to a record associating an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. And a display control method.
(12) a function of referring to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. A program that causes a computer to realize the functions to be executed.

DESCRIPTION OF SYMBOLS 100 Smartphone 102 Imaging part 104 Recognition part 108 Generation part 110 Reference part 114 Display control part 118 Display part

Claims

A record reference unit that refers to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. A display control unit.
The display control unit changes the posture of the object to a predetermined posture in displaying the first image, and at least temporarily maintains the predetermined posture even after transition to the display of the second image. The display control apparatus according to claim 1.
The display control apparatus according to claim 2, wherein the predetermined posture is a directly-facing posture.
The display control unit changes the display position or orientation of the object so as to correspond to the display position or orientation of the object in the display of the second image in the display of the first image. The display control apparatus according to claim 1, wherein display of one image is transitioned to display of the second image.
The display control unit displays the first image by changing the display position or orientation of the object so that the display position or orientation of the object corresponds to the display position or orientation of the object in the display of the first image. The display control device according to claim 1, wherein the display control device makes a transition to display.
The display control apparatus according to claim 1, wherein at least one of the first image and the second image is a captured image.
The display control device according to claim 1, wherein at least one of the first image and the second image is a virtually configured image.
The display control apparatus according to claim 1, wherein the display control unit displays, in the first image, options of an image that can be transitioned as the second image.
The display control device according to claim 8, wherein the display control unit displays the options for each attribute of the image.
2. The display control unit according to claim 1, wherein, based on the record, the annotation given to the object in each of the first image and the second image is displayed in association with the object in a cumulative manner. Display controller.
Referencing a record associating an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. And a display control method.
A function for referring to a record that associates an image, an object included in the image, and a display position or orientation of the object in the image;
Based on the record, the display of the first image including the object is changed to the display of the second image including the object and different from the first image while maintaining the display position or orientation of the object. A program that causes a computer to realize the functions to be executed.