JP2010165081A - Display apparatus, and system, apparatus and method for creating three-dimensional content - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create and provide a desired three-dimensional content for a user while reducing a load related to acquisition of a required partial element, in creating the three-dimensional content. <P>SOLUTION: When the user images an object from a plurality of directions and positions using a cellular phone 1 with a camera function based on a shooting direction and shooting position of the object such as a building or the like transmitted from a center 5, the captured images, the shooting positions, and shooting directions are transmitted to the center 5. In the center 5, a server 3 creates the three-dimensional content based on the received data and the captured image, the shooting position, and shooting direction of the object stored in a database 4. Then, when the cellular phone 1 requests transmission of the three-dimensional content, the server 3 transmits the created three-dimensional content to the cellular phone 1, and the cellular phone 1 displays the three-dimensional content on a screen based on the three-dimensional content. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

The present invention relates to a display device, a stereoscopic content generation system, a stereoscopic content generation device, and a stereoscopic content generation method.
As a field of use of the present invention, for example, there is provided a stereoscopic content providing service that transmits image data to a center, generates stereoscopic content based on the image data on the center side, and provides it to the user who transmitted the image data.

  A single building is photographed from a plurality of directions with a plurality of cameras, and a three-dimensional reproduction of the building image that has been reproduced in three dimensions based on the photographed images and three-dimensional CAD data prepared in advance. A technique for projecting images is disclosed (for example, see Patent Document 1).

  When restoring camera motion and relative positional relationship (three-dimensional shape) from point correspondence in multiple images, point correspondence is first given by "image coordinates", so it is unified for all cameras. In order to handle it, if a one-to-one mapping relationship between the “pixel coordinates” and the “image coordinates” can be obtained (called “camera calibration”), the “pixel coordinates” can be obtained. "And" Image coordinates "can be freely converted.

  Therefore, as a method for creating three-dimensional shape data from a plurality of photographed images, for example, by inputting image data photographed from a multi-viewpoint angle from 360 degrees all around or from a plurality of cameras arranged around the object, Various software for creating three-dimensional shape data of an object has been developed. In these softwares, a method using a plurality of multi-viewpoint images captured by a camera whose camera position information is previously captured with a camera placed at a predetermined position or angle, or disclosed in Non-Patent Document 1. As described above, there is a method of automatically calculating the camera position at the time of shooting from a plurality of camera images obtained by shooting together the reference marker printed on the turntable and the target object placed on the turntable.

  Then, the three-dimensional information of the shape and texture is automatically calculated from the obtained camera position information and the object image, and the three-dimensional information is appropriately corrected by the user to generate the three-dimensional data from the image hand-held by the digital camera. . In other words, about 10 to 20 or more JPEG-format captured image data with different viewpoints for capturing an object are input, and a series of processes such as camera position calculation, shape information generation, and texture information generation are executed. 3D shape data is created.

More specifically, three-dimensional shape data is obtained by the following procedures (1) to (3).
(1) When calculating the camera position, for example, the three-dimensional position parameters of the camera in the coordinate system (rotation components α, β, γ and translation components (x, y, z)) are calculated based on the Hough transform or the like. That is, a reference marker is extracted from the photographed image, and is determined by combining any three points of the extracted reference marker and any three points in the reference marker registered in advance. The position parameters are calculated by solving the simultaneous equations.
(2) In the generation of shape information, an object is photographed using a monochrome background paper or the like, the object and the background are separated using the specified background color information, and the silhouette (outline) of the object is extracted. .
(3) Next, based on the position information of the camera and the two-dimensional outline information of the object, a three-dimensional shape is reconstructed by voting (voting) processing to the three-dimensional voxel space, and from the obtained voxel data And converted into polygon (polygon) data, and output as three-dimensional shape data such as a triangular polygon representation format and an SSF format.

JP 2004-021848 PR

"Software technology for easily generating 3D data from digital camera images" SANYO TECHNICICAL REVIEW COL. 35 NO. 1 JUN. 2003

  However, with the technology disclosed in Patent Document 1 and the like, when a user (individual) wants to view stereoscopic content, it is necessary to project an image on a wall surface for observation provided in advance and wear stereoscopic glasses. In addition, when the user wants to obtain the desired stereoscopic content, it is necessary to specify a portion that the user wants to view, to shoot a necessary captured image via the system device, and to prepare a plurality of captured images. It was.

  In addition, as disclosed in Non-Patent Document 1, when automatically creating three-dimensional data from an automatic photographing machine, a 360 degree photographing stand or a special photographing machine for photographing a large number of images, active scan images, etc. from 360 degrees around. Since it is necessary, the user needs to go to the place where the dedicated camera is installed and take a picture. However, in order to take a 360-degree image of an existing building or the like desired by the user as a stereoscopic content, a large preparation is required. There was a problem that it was necessary and was not realistic from the viewpoint of economy and time. Further, in order to generate three-dimensional data from a photographic image, it is necessary to perform the calculation of the camera position of (1) above, which requires complicated calculation, and the result also requires adjustment work by an expert. There was an inconvenience.

  In addition, according to the method of creating the three-dimensional shape data of the object by inputting image data taken from a plurality of viewing angles from 360 degrees around the above-mentioned or from a plurality of cameras arranged in the surroundings, Although it is sufficient to input the camera position, the problem (1) does not occur. However, in order to photograph an existing building or the like desired by the user as a three-dimensional content at 360 degrees, a plurality of cameras are used to photograph the entire circumference. If the required number of cameras cannot be installed due to the size of the building or the location conditions (the presence of obstacles such as terrain or standing trees), or the cameras are arranged with the required number of cameras. There was an inconvenience that the cost for doing so increased.

  In order to solve the above-described problem, the invention described in claim 1 includes at least a storage unit that stores a captured image, a shooting position for shooting a target object, or a shooting direction for shooting the target object. A first receiving means for receiving one of them, and a readout for reading out the captured image of the object taken along at least one of the photographing position or the photographing orientation received by the first receiving means from the storage means Means for transmitting the captured image read by the reading means to a center for generating data for displaying the object three-dimensionally, and for transmitting the image of the object by the transmitting means. In response, a second receiving means for receiving the three-dimensional display of the object transmitted from the center, and a data received by the second receiving means. And a stereoscopic display means for stereoscopically displaying the object based on the data.

  According to a second aspect of the present invention, in the first aspect of the present invention, the object is imaged in accordance with at least one of a photographing position or a photographing direction of the object received by the first receiving means. The storage means stores a captured image of the object imaged by the imaging means.

  The invention according to claim 3 is the invention according to claim 2, wherein the first receiving means receives the photographing position of the object and obtains the current position of the device. Means for comparing the shooting position of the object and the current position acquired by the current position acquisition means, and as a result of the comparison by the comparison means, the shooting position of the object and the current position are approximately And a notification means for notifying the fact that they coincide with each other.

  In order to solve the above-mentioned problem, the invention according to claim 4 is still in the database for storing a plurality of subelements as a basis for generating the three-dimensional content of the object, the photographing terminal, and the database. Notification means for notifying the photographing terminal that there is a subelement of the object that is not stored, and a captured image of the object that is photographed and transmitted by the photographing terminal in response to a notification by the notification means Receiving means, extracting means for extracting subelements from the captured image of the object received by the receiving means, subelements extracted by the extracting means, and a plurality of subelements stored in the database And determining means for determining whether or not the three-dimensional content of the object can be generated, and determining that the three-dimensional content can be generated by the determining means. If that, a generation unit for generating stereoscopic contents of the object, characterized in that it comprises a means, for transmitting a stereoscopic content generated by the generation unit to the imaging device.

  In order to solve the above-mentioned problem, the invention according to claim 5 is the database for storing a plurality of partial elements that are the basis for generating the three-dimensional content of the target object. Notification means for notifying the photographing terminal of the presence via the communication network, and receiving means for receiving a captured image of the object photographed and transmitted by the photographing terminal in response to the notification by the notifying means. The extraction unit extracts a partial element from the captured image of the object received by the reception unit, the partial element extracted by the extraction unit, and a plurality of partial elements stored in the database. A determination unit that determines whether or not the three-dimensional content can be generated, and when the determination unit determines that the three-dimensional content can be generated, Characterized in that it comprises generation means for generating a three-dimensional content elephant product, the.

  In order to solve the above-mentioned problem, the invention according to claim 6 is the database for storing a plurality of partial elements that are the basis for generating the three-dimensional content of the target object. A notification step of notifying the imaging terminal of the presence via the communication network, and a reception step of receiving a captured image of the object that is captured and transmitted by the imaging terminal in response to the notification in the notification step An extraction step of extracting a partial element from the captured image of the object received in the reception step, and the partial element extracted in the extraction step and a plurality of partial elements stored in the database. A determination step for determining whether or not the three-dimensional content of the object can be generated, and the three-dimensional content can be generated in this determination step. Judging, characterized in that it comprises a a generation step of generating a three-dimensional content of the object.

  ADVANTAGE OF THE INVENTION According to this invention, the desired three-dimensional content can be produced | generated and provided with respect to a user, reducing the burden concerning acquisition of a required partial element.

It is a figure which shows the structural example of the three-dimensional content provision service system of this invention. It is a process chart which shows the outline | summary of the three-dimensional content provision service in a three-dimensional content provision service system. It is a block diagram which shows the structural example of the electronic circuit of the mobile telephone with a camera provided with the three-dimensional video display function. It is a block diagram which shows the structural example of the server shown in FIG. It is a figure which shows the structural example of the database shown in FIG. 1, FIG. It is a flowchart which shows the operation example of the mobile telephone at the time of 3D content acquisition service selection. It is a figure which shows an example of the service start information etc. which are displayed on the mobile telephone 1 at the time of a three-dimensional content provision service start. It is a figure which shows the operation example of the server 4 at the time of a three-dimensional content provision service.

  FIG. 1 is a diagram showing a configuration example of a stereoscopic content providing service system 100 to which the present invention is applied. The stereoscopic content providing service system 100 is a mobile phone with camera 1 (hereinafter referred to as a display device having a stereoscopic image display function). , “Cell phone 1”), a plurality of cellular base stations 2 (hereinafter referred to as “base station 2”), 3D contents (stereoscopic image data) such as buildings and buildings, and the 3D contents are generated. The server 3 includes a database 4 that stores a plurality of camera information such as shooting images and shooting directions and shooting positions of shooting objects such as buildings and buildings. The base station 2 and the server 3 are connected via a communication network such as the Internet 6 (hereinafter referred to as “Internet 6”). Chikyoku 2 is configured to be connected by a cellular radio communication system. Reference numeral 5 denotes a three-dimensional content providing center (hereinafter referred to as “center 5”) operated by an operator of the three-dimensional content providing service, and reference numeral 7 denotes a structure that is an example of the target of the three-dimensional content requested by the user. Showing things.

  FIG. 2 is a process chart showing an overview of the stereoscopic content providing service in the stereoscopic content providing service system 100 of FIG. In FIG. 2, first, the user activates the mobile phone 1, selects a three-dimensional content providing service from among various services, and performs user registration (input of a registration number or the like if the user has been registered) (process P1).

  From the list of buildings and buildings (hereinafter referred to as “objects”) sent from the center 5 (server 3) to the mobile phone 1, a building or the like (object) scheduled to purchase 3D content is selected (object). Process P2). The database 4 may register not only buildings and buildings but also photographed image identification information in which photographed images of equipment, devices, specimens, and the like are associated.

  When the photographing target is selected, the server 3 of the center 5 obtains a partial element of the target object that is not yet stored in the database 4 in order to obtain a partial element necessary for generating the stereoscopic content for the target object. A list of position information (coordinate data) where the object exists, a shooting direction for shooting an image of the object, and a shooting position (coordinate data) is displayed on the user's mobile phone 1 on the Internet 6 and the base station 2. Mobile phone 1 downloads the data (process P3).

  When the registered user selects a desired object from among the objects that can be photographed displayed on the mobile phone 1, and photographs a plurality of the objects and stores them in the memory, the cellular phone 1 provides stereoscopic content. Images taken on the basis of the shooting direction and shooting position that are insufficient for this purpose, and / or images that have been hit by searching the memory based on these, and camera position information such as the shooting direction and shooting position of the image Are uploaded to the center 5 (process P4).

  The center 5 determines whether or not the captured images necessary for generating the three-dimensional content are prepared, and if it is determined that the images are prepared, the center 5 notifies the mobile phone 1 of the user (one or more) who provided the images to that effect. (Process P5).

  When the user who has received the notification selects whether to receive the provision of stereoscopic image data or the provision of miniature as a specific provision of the stereoscopic content by operating the mobile phone 1 (process P6), The selected content is sent from the mobile phone 1 to the server 3 via the base station 2 and the Internet 6 to provide stereoscopic content according to the user's wishes. At this time, the server 3 charges the user who provided the image free of charge (or a discount price), and charges the other users as a normal price (process P7).

  As a modified example, the process P4 is described as follows: “When the user moves with the mobile phone 1, the mobile phone 1 notifies the user when the user approaches the shooting position, and the user gives guidance. Therefore, a configuration may be adopted in which a photograph is taken and camera information such as a photographed image, a photographing orientation, and a photographing position is uploaded to the center 5 via the base station 2 and the Internet 6 to the server 3 of the center 5.

  FIG. 3 is a block diagram showing an example of the internal configuration of the mobile phone 1 shown in FIG. 1. The mobile phone 3 is configured to be capable of displaying a three-dimensional image, and includes a radio transmission / reception unit 11, a radio signal processing unit 12, and a control. 13, key input unit 14, driver 17, display unit 18, subscriber information storage unit 19, program storage memory 20 including ROM, temporary storage memory 21 including DRAM, recording unit 22, audio signal processing unit 23, In addition to the speaker 24, the microphone 25, the driver 26, and the speaker 27, the imaging unit 15 and the image data processing unit 16, and a camera position acquisition unit 28 that acquires camera position information (shooting position and shooting direction) are provided. The imaging unit 15 and the image data processing unit 16 constitute an imaging system of the present invention.

  The wireless transmission / reception unit 11 transmits and receives audio signals and data via the antenna 111 wirelessly. The wireless signal processing unit 12 performs processing necessary for wireless communication such as demodulation processing of audio signals and data received by the wireless transmission / reception unit 111 and modulation processing of audio signals and data transmitted from the wireless transmission / reception unit 111. In the three-dimensional content acquisition mode, when connected to the Internet 6 under the control of the communication control program, the captured image and the shooting direction and the shooting position of the captured image can be transmitted to the server 3 of the center 5 via the wireless transmission / reception unit 11. In addition, it is possible to receive the shooting direction and shooting position of the captured image of the target object necessary for generating the stereoscopic content and the stereoscopic image data transmitted from the server 3.

  The control unit 13 controls the operation of each internal component shown in FIG. 3 by the control program and the operation program stored in the program storage memory 20. The operation program includes a communication control program, an imaging control program, an image processing program, a mode selection program, a mobile phone side operation program for a stereoscopic content providing service, an image display program for displaying an object image in three dimensions, and the like. include. The communication control program includes a connection control program with the Internet 6.

  The key input unit 14 includes various keys such as a power key, a numeric keypad, a transmission / reception start key, a transmission / reception stop key, a cursor key, a cancel key, an enter key, and a shutter key. When these keys are operated, a corresponding signal is transmitted to the control unit. 13 is sent out.

  The imaging unit 15 has an imaging signal system (not shown) including an imaging element, a signal conversion unit, an imaging timing control unit, an imaging auxiliary light source, and the like, and passes through the imaging lens and is behind the imaging optical axis of the imaging lens. The subject light image incident on the image pickup device arranged at is photoelectrically converted by the image pickup device and output as an image pickup signal (analog). The output signal of the image sensor is correlated double sampling and gain adjustment by a signal conversion unit, and converted to a digital signal by A / D conversion. The A / D converted image signal is sent to the image data processing unit 16 at a predetermined cycle.

  The image data processing unit 16 performs a color process process including an image interpolation process and a γ correction process on the digital signal (image data) output from the imaging unit 15 to obtain a digital luminance signal Y and color difference signals Cb, Cr. And DMA-transferred to a temporary storage memory 21 such as a DRAM via a bus.

  The driver 17 drives the display unit 18 to display an image and data. The subscriber information storage unit 19 stores profile data such as a telephone number for calling another telephone and a user (registrant) ID. In the embodiment, the stereoscopic image display is performed by a three-dimensional image display program. However, hardware capable of displaying a three-dimensional image may be used as the driver 17 and the display unit 18.

  The program storage memory 20 is a memory in which a control program and an operation program of the control unit 13 are recorded. The program storage memory 20 stores various data necessary as a wireless communication terminal, and programs necessary for the operation of the control unit 13 Store the data. For example, a control program, a mode selection program, an operation program in various modes, a mobile phone side operation program for a stereoscopic content providing service, an operation program (application program) such as the image display program, and the like are stored. These programs are called according to the processing stage of the selected mode and executed by the control unit 13. The operation program includes a mode selection program.

  The temporary storage memory 21 is composed of a memory such as a DRAM, stores various data necessary as a wireless communication terminal, and stores programs and data necessary for the operation of the control unit 13. The temporary storage memory 21 is used as a buffer memory for temporarily storing image data from the image data processing unit 16 and a work memory necessary for data processing by various programs in the shooting mode and the stereoscopic content acquisition mode.

  In the three-dimensional content acquisition mode, the temporary storage memory 21 has a work area necessary for displaying the image data of the target object three-dimensionally. In addition, a camera information holding area for holding the shooting position and shooting direction acquired from the center 5 or at the time of shooting, camera information at the time of shooting, and the like is also secured.

  The recording unit 22 stores various data necessary for the wireless communication terminal, captured image data, and a shooting direction and a shooting position corresponding to the shooting of the data. In addition, stereoscopic content (stereoscopic image data) and programs received from the center 5 can be stored. Even when the power switch of the mobile phone 1 is off, the storage unit 22 is supplied with power and the stored contents are maintained.

  The audio signal processing unit 23 encodes the audio signal input from the microphone 24, decodes the audio signal based on the signal output from the wireless signal processing unit 12, and outputs the audio signal from the speaker 25. The driver 26 is a driver for driving the speaker 27.

  The camera position acquisition unit 28 includes a well-known GPS reception positioning device, a three-dimensional acceleration sensor, a tilt sensor, an electronic compass, and other sensor measurement devices that detect the position, angle, and orientation of the camera. The shooting position and shooting direction are acquired from GPS information and an electronic compass.

  FIG. 4 is a block diagram showing a configuration example of the server (server computer) 3 shown in FIG. The server 3 includes a control unit 31, a temporary storage memory 32, a program storage memory 33, a transmission / reception unit 34, a camera position information analysis unit 35, a storage memory 36, a display unit 37, an operation unit 38, and a database 4. The display unit 37 is not essential. Although not shown, a medium recording apparatus for recording data such as CAD for miniature production (not shown) on a recording medium such as a CD may be provided.

  The control unit 31 controls the operation of each internal component shown in FIG. 4 by a control program and an operation program stored in the program storage memory 33. The operation program includes a communication control program, a three-dimensional content generation program, a user registration and authentication program, a billing program, and the like. The communication control program includes a connection control program with the Internet 6.

  The program storage memory 33 is a memory in which a control program and an operation program of the control unit 31 are recorded, stores various control data for functioning as the server 3, and is necessary for the control unit 13 to operate. Store programs and data. For example, a control program and a server-side operation program for a stereoscopic content providing service, that is, a stereoscopic image data generation program, a miniature production action data creation program, a user registration and authentication program, an application program such as a billing program, are stored. . These programs are called to the temporary storage memory 32 and executed by the control unit 31 as appropriate. Further, instead of providing the camera position information analysis unit 35 in the server 3, an image analysis program may be stored in the program storage memory 33 as one of operation programs.

The data generation by the stereoscopic image data generation program is known because the camera position of (1) is acquired at the time of photographing among the procedures (1) to (3) described in the background section. The procedure (2) and (3) described above can be performed with the camera position known. That is, the photographed image is separated into the object image and the background image, the silhouette (contour) of the object image is extracted, and then the position information of the camera and the extracted contour information of the object are used as partial elements. Stereo image data is generated based on a plurality of shooting positions and subelements of shooting directions.
Note that a known technique such as the technique described in Non-Patent Document 1 can be applied to the method of generating stereoscopic image data based on the shooting position, the shooting direction, and the extracted contour information of the object.

  The temporary storage memory 32 is configured by a memory such as a RAM, and stores programs and data necessary for the operation of the control unit 31. The temporary storage memory 32 is used as a buffer memory for temporarily storing image data from the database 4 and the transmission / reception unit, and a work memory necessary for data processing by various programs.

  The transmission / reception unit 34 transmits / receives data to / from the mobile phone 1 via the Internet 6 (and the base station 2) under the control of the communication control program. For example, registration information and authentication information transmitted from the mobile phone 1, camera information including an image, a shooting direction, and a shooting position are received. In addition, information necessary for photographing, stereoscopic content, billing data, and the like are transmitted to the mobile phone 1.

The camera position information analysis unit 35 obtains a plurality of partial elements from a plurality of photographed images associated with the objects already stored in the database 4 as described above, and determines whether or not stereoscopic content (stereoscopic image data) can be generated. Based on the shooting direction and shooting position of the shot image, the shooting direction and shooting position necessary for generation are output if there is a shortage. Note that, instead of the camera position information analysis unit 35, an image analysis program that performs the same function is stored in the program storage memory 20 of the mobile phone 1, and when the target is selected by the user, the control unit 13 sets the database. 4 may be analyzed to determine whether or not the stereoscopic content (stereoscopic image data) can be generated from the image already stored in 4.
Further, the camera position information analysis unit 35 may determine whether or not the captured images for creating the stereoscopic content (stereoscopic image data) are prepared based on the imaging position and the imaging orientation.

The storage memory 36 is composed of a large-capacity storage device such as a hard disk device, and stores a registration file storing user information and authentication data of a stereoscopic content providing service, a charging file for recording charging data of content provided for each user, and the like. To do.
In this embodiment, the memory for storing the database 4 is separate from the storage memory 37, but the database 4 may be constructed in the storage memory 36.

  FIG. 5 is a diagram illustrating a configuration example of the database 4 illustrated in FIGS. 1 and 4. The database 4 includes the object identification information 41, the position information (map coordinates) 42 of the object, and the object as three-dimensional content. A stereoscopic image pointer 43 that is a pointer to the stereoscopic image data storage area 48 that stores stereoscopic image data associated with the object, and a plurality of captured image data storage areas that store image data of a plurality of captured images associated with the object Photographic image pointers 44-1 to 44-m, which are pointers to 49, photographic orientation data 45-1 to 45-m, photographic position data 46-1 to 46-m, registrant identification information (registration file) A plurality of object-specific photographic image records 40 composed of 47 and the like are constructed so as to be searchable.

  When the stereoscopic content is not completed for a certain object (that is, when the partial elements for generating the stereoscopic image data are insufficient), the imaging is insufficient for generating the stereoscopic image pointer 43 and the stereoscopic image data. Invalid values (null) are stored in the photographed image pointer 44-j of the image, the photographing orientation data 45-j of the image, and the photographing position data 46-j (j = 1 to m). Further, when there is no captured image with only the object identification information, an invalid value (null) is stored in the data storage area other than the object identification information.

  FIG. 6 is a flowchart showing an operation example of the mobile phone 1 when the three-dimensional content acquisition service is selected. The mobile phone 1 is selected by a control program, a communication control program, an image processing program, a mode selection program, and various operation programs. Mode operation (function) is executed. The operation of the mobile phone 1 in the 3D content acquisition mode is taken out from the program storage memory 20 by the control unit 13 under the control of the control program, and stored in the execution program area of the temporary storage memory 21. An operation corresponding to the illustrated step is executed based on an operation program such as a side operation program or a three-dimensional image display program.

  In FIG. 6, when the user operates the key input unit 14 to select a three-dimensional content providing service among various application services of the mobile phone 1, the control unit 13 is connected via the wireless signal processing unit 12 and the wireless transmission / reception unit 11. “Service selection notification” and user information are transmitted to the base station 2. The data wirelessly transmitted to the base station 2 is transmitted to the center 5 via the Internet 6 (hereinafter simply referred to as “transmit to the center 5”) (step S1).

  After transmission of “service selection notification” or the like in step S1, the determination result of whether or not the user is a registered user transmitted from the center 5 (see FIG. 1) via the Internet 6 and the base station 2 is checked. The process proceeds to step S3, and if registered, the process proceeds to step S4 (step S2).

  When the user is registered in the center 5, the user operates the key of the key input unit 14 according to the authentication guide displayed on the display unit 18 of the mobile phone 1 to input authentication information such as a password. The unit 13 transmits the input authentication information such as a password and a mobile phone number to the center 5 via the wireless signal processing unit 12 and the wireless transmission / reception unit 11, waits for authentication at the center 5, and when authentication is performed, step S5 is performed. (Step S3).

  When the user is not registered in the center 5, the user operates the keys of the key input unit 14 according to the registration guide displayed on the display unit 18 of the mobile phone 1 to perform user registration of the stereoscopic content acquisition service. The unit 13 transmits the input user identification information, authentication information including a password, and registration information such as a mobile phone number to the center 5 via the wireless signal processing unit 12 and the wireless transmission / reception unit 11, and proceeds to step S5 (step S5). Step S4).

  The control unit 13 transmits, through the wireless transmission / reception unit 11 and the wireless signal processing unit 12, the service-provided target list data transmitted from the center 5 together with the registered notification or the authenticated notification when registration or authentication is performed. The list is displayed on the display unit 18 so as to be selectable, and the user is prompted to select (step S5).

  When the user selects a shooting target, the control unit 13 wirelessly transmits and receives the shooting target object transmitted from the center 5 and map data indicating the current position of the user and information necessary for the center 5 to provide stereoscopic content. The data is received via the unit 11 and the wireless signal processing unit 12 and stored in the temporary storage memory 21 (step S6).

  Further, the control unit 13 displays a map (see FIG. 7A) including the current position of the user and the object based on the received map data (see FIG. 7A) on the display unit 18 (step S7), and the “information browsing” of the user. In response to the operation, a shooting target image (indicated by a landmark) is displayed on the display unit 18 and the shooting direction and shooting position that are insufficient for the center 5 to provide stereoscopic content are displayed. (See FIG. 7B) (step S8).

  When the user shoots an object to be imaged in accordance with the guidance from the shooting position in the illustrated shooting direction and operates the keys of the key input unit 14 to give a recording instruction, the control unit 13 The shooting direction data and shooting position data acquired by the camera position acquisition unit 28 are recorded in the recording unit 22. The user performs this photographing operation until photographing of a necessary image (that is, an image that is insufficient for generating the stereoscopic content) is finished (step S9).

  The control unit 13 uses the shooting position and shooting direction that are insufficient to generate the three-dimensional content of the target acquired in step S6 as a key (the image recorded in the recording unit 22). ) And is transmitted to the center 5 through the wireless signal processing unit 12 and the wireless transmission / reception unit 11 together with the image of the hit object, the shooting position of the image, and the image of the object in which the shooting direction is shot (step S10).

  When the control unit 13 receives a notification that the target image for which the application has been made from the center 5 has been completed, the control unit 13 informs the display unit 18 of the fact and a method for providing the stereoscopic content (stereoscopic image). When a message prompting the user to select whether to receive data or receive a miniature) is displayed and the user selects a providing method, the control unit 13 displays the selection result in the wireless signal processing unit 12 and the wireless Transmission to the center 5 is performed via the transmission / reception unit 11 (step S11).

  When the user selects reception of stereoscopic image data, the control unit 13 receives the stereoscopic image data transmitted from the center 5 via the wireless transmission / reception unit 11 and the wireless signal processing unit 12 and records them in the recording unit 22. At the same time, the object is displayed three-dimensionally on the display unit 18 by the image display program (step S12).

(Modification)
Note that, as a modification of the operation of the mobile phone 1 when the three-dimensional content acquisition service is selected, step S8 is performed as follows: “When the user is moving around the object to be photographed with the mobile phone 1, Compares the shooting position included in the information acquired in step S6 with the current position acquired by the camera position acquisition unit 28. If they substantially match, the speaker 27 (or a vibrator not shown) is driven to notify that effect. In addition, the display unit 18 is configured to display a guidance for photographing ”, and when the object is photographed in the photographing direction indicated in the photographing guidance at the point where the user is notified, the control unit Reference numeral 13 denotes a sensed image, a photographing position (current position) acquired by the camera position acquisition unit 28, and a photographing direction via the wireless signal processing unit 12 and the wireless transmission / reception unit 11. "Send to 5", and repeat step S10 "repeating the operations of steps S8 and S9 until a notification is received from the center 5 that the captured images for creating the 3D content are ready". It may be configured.

  FIG. 7 is a diagram illustrating an example of service start information displayed on the mobile phone 1 when the stereoscopic content providing service is started. FIG. 7A illustrates the current position (indicated by a star) of the user (camera). 7 and an example of a map display including an object 7 to be photographed indicated by a landmark, and FIG. 7B shows a necessary photographing position (represented by # 1 to # 3) in the vicinity of the object 7 to be photographed. It is an example of a shooting guide screen displaying a point 76), a shooting direction (arrow) 77, and a guide message 78.

  FIG. 8 is a flowchart showing an operation example of the server 3 in the center 5. The server 3 is based on an operation program such as a control program, a communication control program, a center-side operation program for a stereoscopic content providing service, and a stereoscopic content generation program. The operation (function) as shown in FIG. 8 is executed.

  In FIG. 8, when the service selection notification and user information are received from the mobile phone 1 via the Internet 6 (step T1), the control unit 31 searches the registration file in the storage memory 36 to check whether or not the user is a registered user. In the case of a registered user, an authentication request for registrant authentication is transmitted to the mobile phone 1 via the Internet 6. If the user is not a registered user, a registration request is transmitted to the mobile phone 1 that is the transmission source of the selection notification, and the process returns to T1 (step T2).

  Next, when registration information is received from the mobile phone 1 (step T3), the control unit 13 registers the received registration information in the registration file in the storage memory 36, ends the user registration, and proceeds to step T7 (step T4). ).

  When the authentication information is received (step T5), the control unit 13 registers the registrant identification information and the authentication data such as a password included in the received authentication information and the registrant identification information stored in the registration file of the storage memory 36. Then, the authentication data is compared, and the authentication result is transmitted to the mobile phone 1 as the authentication information transmission source. At this time, if the authentication is successful, the process proceeds to step T7, and if the authentication is not possible, the process returns to step T1 (step T6).

  In addition, when the registration process in step T4 ends or the registrant can be authenticated in step T7, the server 3 registers the object list data that can be provided by the center 5 with the registration / authentication information transmission source mobile phone. Transmit to the telephone 1 (step T7).

  When the identification information of the target selected from the mobile phone 1 that is the target list data transmission destination is received (step T8), the control unit 31 uses the received identification information of the target to capture the captured image record 40 for each target in the database 4. The three-dimensional image pointer 43 of the object in the object-specific captured image record 40 that has been searched is searched, and if an invalid value is stored in the stereoscopic image pointer 43, the shooting direction data 45-1 to 45-m, The photographing position data 46-1 to 46-m are examined, the photographing direction and the photographing position are taken out based on the pointer that is not an invalid value, and passed to the camera position information analysis unit 35 (step T9).

The camera position information analysis unit 35 checks the combination of the shooting direction and the shooting position extracted from the database 4 in step T9, and satisfies the number of images, the shooting direction, and the shooting position necessary for generating stereoscopic content (stereoscopic image data). If it is insufficient, the shooting direction and the shooting position are calculated. The control unit 31 takes out the map data of the object from the database 4, and analyzes the map data and the analysis result of the camera position information analysis unit 35 (shooting direction and shooting position for shooting an image necessary for stereoscopic content). The information is transmitted to the cellular phone 1 that is the transmission source of the object identification information, and the process returns to Step T1 (Step T10).
Note that the analysis of the shooting direction and shooting position which are insufficient by the camera position information analysis unit 35 is, for example, 2 in (1) to (3) in the procedure of (1) to (3) described in the background art section. Assuming three known camera positions and one unknown camera position, the unknown camera position is calculated by solving an equation determined by combining the positional relationship of the three points. Can be calculated.

  When the photographed image data of the object, the photographing orientation at the time of photographing the image, and the photographing position are received from the mobile phone 1 (step T11), the control unit 31 records the received data in the temporary storage memory 32 and the target The photographed image record 40 classified by object in the database 4 is searched with the object identification information, and the photographed image pointers 44-1 to 44-m of the object of the photographed image record 40 classified by object and the shooting direction data 45-1 to 45-1. 45-m and image position data 46-1 to 46-m, based on a pointer that is not an invalid value, image data, image direction and image position (coordinate data) are sequentially extracted and temporarily stored in a three-dimensional image data generation program. 32, the captured image, the shooting position, and the shooting direction of the target object, and the captured image, the shooting position, and the shooting of the target object extracted from the database 4. Based on the position, the object image and the background image are separated from the photographed image, the silhouette (contour) of the object image is extracted, and then the photographing position and photographing direction, and the extracted contour information are extracted. Based on the above, 3D data is reconstructed by voting to the 3D voxel space, and the obtained voxel data is converted to polygon (polygon) data. A content is generated, and a corresponding part of the stereoscopic image data storage area 48 is overwritten and stored based on the stereoscopic image pointer 43 of the object in the temporary storage memory 32 and the captured image record 40 according to the object in the database 4 to generate a stereoscopic content. Is transmitted to the mobile phone 1 that is the transmission source of the photographed image data, etc., and the process returns to step T1 (step T12).

  When the selection result of the stereoscopic content providing method is received from the cellular phone 1 that is the image data transmission source of the object (T13), the control unit 31 examines the reception selection result and “stereoscopic image data reception” is selected. If this is the case, the process proceeds to step T15. If “miniature reception” is selected, the process proceeds to step T16 (step T14).

  If “receive stereoscopic image data” is selected, the stereoscopic image data stored in the temporary storage memory 32 in step T7 is transmitted to the mobile phone 1 that is the transmission source of the selection result, and charging processing is performed, and the process returns to step T1 ( Step T15).

  If “Miniature Receipt” is selected, the control unit 31 creates production data to be used in the external miniature production device from the data stored in the temporary storage memory 32 in Step T7 by the miniature production data creation program. Then, the processing data created on a recording medium such as a CD is recorded by a medium recording device (not shown), and a charging process is performed, and the process returns to step T1 (step T16).

1 Mobile phone with camera function (display device)
3 Server 4 Database 5 Center 13, 31 Control unit 15 Imaging unit 18 Display unit 22 Recording unit 28 Camera position acquisition unit 32 Storage memory 35 Camera position information analysis unit

Claims (6)

Storage means for storing captured images;
First receiving means for receiving at least one of a shooting position for shooting a target object or a shooting direction for shooting the target object;
Reading means for reading out the captured image of the object taken along at least one of the photographing position or the photographing direction received by the first receiving means from the storage means;
A transmission unit that transmits the captured image read by the reading unit to a center that generates data for displaying the object three-dimensionally;
In response to transmission of the image of the object by the transmission means, second receiving means for receiving data for stereoscopically displaying the object transmitted from the center;
3D display means for stereoscopically displaying the object based on the data received by the second receiving means;
A display device comprising: An imaging unit that images the object according to at least one of an imaging position or an imaging direction of the object received by the first receiving unit;
The display device according to claim 1, wherein the storage unit stores a captured image of the object captured by the imaging unit. The first receiving means receives the shooting position of the object,
Current position acquisition means for acquiring the current position of the device;
Comparison means for comparing the shooting position of the object and the current position acquired by the current position acquisition means;
As a result of the comparison by the comparison means, when the photographing position of the object and the current position substantially coincide with each other, a notification means for notifying that effect,
The display device according to claim 2, further comprising: A database for storing a plurality of subelements that are the basis for generating the stereoscopic content of the object;
A shooting terminal,
Notifying means for notifying the photographing terminal that there is a partial element of the object not yet stored in the database;
Receiving means for receiving a captured image of the object photographed and transmitted by the photographing terminal in response to the notification by the notifying means;
Extracting means for extracting a partial element from the captured image of the object received by the receiving means;
Determining means for determining whether or not the three-dimensional content of the object can be generated by the partial elements extracted by the extracting means and the plurality of partial elements stored in the database;
If the determination means determines that the three-dimensional content can be generated, a generation means for generating the three-dimensional content of the object;
Means for transmitting the stereoscopic content generated by the generating means to the photographing terminal;
A three-dimensional content creation system comprising: A notification means for notifying a photographing terminal via a communication network that there is a partial element that has not yet been stored in a database for storing a plurality of partial elements that are a basis for generating a stereoscopic content of an object;
Receiving means for receiving a captured image of the object photographed and transmitted by the photographing terminal in response to the notification by the notifying means;
Extracting means for extracting a partial element from the captured image of the object received by the receiving means;
Determining means for determining whether or not the three-dimensional content of the object can be generated by the partial elements extracted by the extracting means and the plurality of partial elements stored in the database;
If the determination means determines that the three-dimensional content can be generated, generation means for generating the three-dimensional content of the object;
A three-dimensional content generation device comprising: A notification step of notifying a photographing terminal via a communication network that there is a partial element that is not yet stored in a database for storing a plurality of partial elements serving as a basis for generating the stereoscopic content of the object;
A reception step of receiving a captured image of the object photographed and transmitted by the photographing terminal in response to the notification in the notification step;
An extraction step of extracting a partial element from the captured image of the object received in the reception step;
A determination step of determining whether or not the three-dimensional content of the object can be generated by the partial elements extracted in the extraction step and the plurality of partial elements stored in the database;
If it is determined in this determination step that the three-dimensional content can be generated, a generation step for generating the three-dimensional content of the object;
A three-dimensional content generation method comprising:

Images