JP2011166430A - Terminal apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To properly control reproduction output when a scene is reproduced and output based on scene reproduction information for reproducing a scene related to an image displayed on a display. <P>SOLUTION: The output intensity when scene reproduction information (smell information, temperature information) is reproduced and output is decided, based on 3D depth when displaying an image (Step C18). In addition, whether the decided scene output intensity is "level 0" is determined (Step C19). When the output intensity is "level 0" (YES at the step C19), a fragrance part and a heating and cooling part are not driven (the scene is not reproduced and output). Meanwhile, when the output strength is not "level 0" (NO at the step C19), the fragrance part and the heating and cooling part are driven (Step C20) to control a generation amount of fragrance depending on the decided scene output intensity and control the heating and cooling range, and thereby, the scene (the fragrance and the temperature) is reproduced and output (Step C21). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a terminal device and a program for displaying an image in 3D on a display unit.

  Conventionally, in a terminal device such as a mobile phone, when shooting an image, a surrounding scene such as an odor is detected, and the detection result corresponds to the shot image as scene reproduction information for reproducing the scene. In addition, a technique is known in which when a captured image is reproduced and displayed, the scene at the time of shooting is reproduced and output based on the scene reproduction information stored in association with the captured image. Yes. For example, in a digital camera capable of recording odor information as scene reproduction information, an association between a shooting mode used for shooting and odor information for reproducing the odor is set in advance, and based on the shooting mode selected at the time of shooting. Thus, a technique is disclosed in which odor information is recorded on a recording medium in association with an image (see Patent Document 1).

JP 2003-264730 A

  In the above-described prior art, when a captured image is reproduced, the scene at the time of shooting can be reproduced based on the scent information (scene reproduction information) associated therewith, so that a great sense of realism can be obtained. However, it is not always a good idea to reproduce and output odors every time an image is displayed. Therefore, it is conceivable to reproduce and output the scent by user operation, but for the user it becomes the result of seeing the image after knowing that the scent is reproduced, and the sense of solidarity and power with the image is insufficient, It cannot be said that even a large reproduction effect can be expected.

  An object of the present invention is to appropriately control a reproduction output when a scene is reproduced and output based on scene reproduction information for reproducing a scene related to an image displayed on a display unit.

  In order to solve the above-described problem, the invention according to claim 1 is a terminal device that displays an image in 3D on a display unit, and is based on scene reproduction information for reproducing a scene related to the image displayed on the display unit. Output means for reproducing and outputting the scene, designation means for designating 3D depth indicating the degree of 3D depth or pop-out of the image displayed on the display unit, and display at the 3D depth designated by the designation means A display control means for displaying an image in 3D on the screen, a determination means for determining an output intensity when the scene is reproduced and output by the output means based on the 3D depth specified by the specification means, and a determination by the determination means Output control means for reproducing and outputting a scene from the output means with the output intensity set.

  As an invention dependent on claim 1, the scene reproduction information is information for reproducing a scene recognized by at least one of olfactory sense, touch sense, auditory sense, and taste sense which is a human sense other than vision. And the output means, based on the scene reproduction information, provides a stimulus for recognizing at least one of olfactory sense, touch sense, hearing sense, and taste sense that is a human sense other than vision. According to the second aspect of the present invention, the data may be generated and reproduced and output.

  The invention according to claim 2 is characterized in that the scene reproduction information is information relating to an odor, and an output intensity when the scene is output by the output means is an odor intensity. It may be the invention of this.

  According to a second aspect of the present invention, the scene reproduction information is information relating to temperature, and the output intensity when the scene is output by the output means is a range in which the casing constituting the terminal device is heated or cooled. The invention according to claim 4 may be provided.

  According to a first aspect of the present invention, a photographing means for photographing an image, a scene detecting means for detecting a surrounding scene at the time of photographing an image by the photographing means and obtaining the scene reproduction information, and photographing by the photographing means Storage control means for associating and storing the reproduced image and the scene reproduction information detected by the scene detection means in the storage unit, and the output control means corresponds to the image displayed on the display unit The invention according to claim 5, wherein the attached scene reproduction information is read from the storage unit and the scene is reproduced and output from the output means.

  As an invention dependent on claim 1, when the 3D depth specified by the specifying means is larger than a predetermined 3D depth, the determining means determines an intensity greater than a predetermined intensity. The invention according to claim 6 may be used.

  According to a first aspect of the present invention, the determining means determines the output intensity when reproducing and outputting the scene when the specified 3D depth specified by the specifying means is smaller than the predetermined 3D depth. The invention according to claim 7 is characterized in that it is determined to be smaller than that.

  As an invention dependent on claim 7, when the 3D depth specified by the specifying means is smaller than a predetermined 3D depth, the determining means determines that the output intensity when reproducing and outputting the scene is zero. The invention according to claim 8 may be provided.

  The invention according to claim 7 is characterized in that, when the 3D depth specified by the specifying means is zero, the determining means determines the output intensity when reproducing and outputting the scene to zero. The invention according to claim 9 may be used.

  As an invention dependent on claim 1, it further comprises a change detection means for detecting that the 3D depth of the image displayed on the display unit has been changed, and the determination means has the 3D depth changed by the change detection means. Is detected based on the 3D depth after the change, and the output control means newly determines the output intensity when the scene is reproduced and output, and the output control means outputs the scene reproduced and output from the output means. The invention according to claim 10, wherein the scene is reproduced and output by switching the output intensity to the output intensity newly determined by the determining means.

  In order to solve the above-described problem, an invention according to claim 11 is a terminal device that displays an image in 2D and 3D on a display unit, and reproduces a scene related to the image displayed on the display unit. In a state where the scene reproduction information is associated with the image and stored in the storage unit, output means for reproducing and outputting the scene based on the scene reproduction information stored in the storage unit, and the image displayed on the display unit A designation means for designating whether 2D display or 3D display is performed, a discrimination means for discriminating whether or not 3D display has been designated by the designation means, and a case in which 3D display is designated by the discrimination means A display control means for displaying the image in 3D on the display unit, and scene reproduction information stored in the storage unit corresponding to the image displayed in 3D on the display unit, An output control means for serial reproduced output scene from the output means, characterized in that it comprises a.

  The invention according to claim 11 is characterized in that the display control means displays the image in 2D on the display unit when it is determined that the 2D display is designated by the determination means. The invention described in claim 12 may be provided.

  As an invention dependent on claim 11, the scene reproduction information is information for reproducing a scene recognized by at least one of olfactory sense, tactile sense, auditory sense, and taste sense which is a human sense other than vision. And the output means, based on the scene reproduction information, provides a stimulus for recognizing at least one of olfactory sense, touch sense, hearing sense, and taste sense that is a human sense other than vision. According to the thirteenth aspect of the present invention, the data may be generated and reproduced for output.

  The invention dependent on claim 11 is a photographing means for photographing an image, a scene detecting means for detecting a surrounding scene at the time of photographing an image as the scene reproduction information by the photographing means, and an image photographed by the photographing means. And storage control means for associating and storing the scene reproduction information detected by the scene detection means in a storage unit.

  In order to solve the above-described problem, the invention described in claim 15 has a function of reproducing and outputting a scene based on scene reproduction information for reproducing a scene related to an image displayed on a display unit. A function for designating a 3D depth indicating a 3D depth of the image displayed on the display unit or a degree of popping, a function for displaying a 3D image on the display unit at the designated 3D depth, and the designated A program for realizing a function of determining an output intensity when reproducing and outputting the scene based on a 3D depth and a function of reproducing and outputting a scene with the determined output intensity .

  According to a sixteenth aspect of the present invention, in order to solve the above-mentioned problem, a scene reproduction information for reproducing a scene relating to an image displayed on the display unit is associated with the image and stored in the computer. A function for reproducing and outputting a scene based on the scene reproduction information stored in the storage unit, and a function for designating whether the image is displayed in 2D or 3D on the display unit. A function for determining whether or not the 3D display is specified; a function for displaying the image in 3D on the display unit when it is determined that the 3D display is specified; and a 3D display on the display unit. And a function for reproducing and outputting the scene from the output means based on the scene reproduction information stored in the storage unit corresponding to the existing image. To.

  According to the present invention, when the scene is reproduced and output based on the scene reproduction information for reproducing the scene related to the image displayed on the display unit, the reproduction output can be appropriately controlled, and solidarity with the image is achieved. It is possible to further enhance the sense of presence and the production effect by feeling and powerful feeling.

The block diagram which showed the communication network system which can use the mobile telephone applied as a terminal device. FIG. 2 is a block diagram showing basic components of the mobile phone 1. FIG. The figure for demonstrating the image memory | storage part M3. The figure for demonstrating 3D depth setting information storage part M4. The figure for demonstrating the output condition memory | storage part M5. The flowchart which showed the outline | summary of the whole operation | movement by the side of the mobile telephone 1 by which execution is started with power activation. The flowchart for explaining an image acquisition process (step A6 of FIG. 7) in detail. The flowchart for explaining an image display process (step A8 of FIG. 7) in detail. 10 is a flowchart following the operation of FIG.

Hereinafter, embodiments of the present invention will be described with reference to FIGS.
This embodiment exemplifies a case where the present invention is applied to a mobile phone as a terminal device, and FIG. 1 is a block diagram showing a communication network system in which the mobile phone can be used.
The mobile phone 1 has a call function, an e-mail function, an Internet connection function (Web access function), a TV broadcast reception function capable of receiving digital TV broadcast (terrestrial digital TV broadcast), a camera function, a 3D display function, and the like. Is provided.

  The 3D display function is a function for displaying an image (moving image, still image, television program, etc.) at a 3D depth indicating the depth of 3D display or the degree of popping out by a parallax barrier method using the parallax of the left and right eyes. That is, a right-eye image and a left-eye image that are slightly shifted from each other are prepared, and the two images are displayed simultaneously. At this time, the right-eye image is visible to the right eye and not to the left eye. The parallax barrier that blocks the light path is arranged at an appropriate position so that the image can be seen with the left eye but not with the right eye, so that the image can be seen in three dimensions.

  When the cellular phone 1 is connected to the wireless communication network (mobile communication network) 2 from the nearest base station 2A and exchange 2B, the cellular phone 1 is connected to other cellular phones (not shown) via the wireless communication network 2. The phone is ready to talk. In addition, when the mobile phone 1 is connected to the Internet 3 via the wireless communication network 2, the mobile phone 1 can access and browse a Web site, and images (moving images and still images) from the image distribution device 4 can be viewed on the Internet 3. It can be downloaded via the wireless communication network 2. The cellular phone 1 receives digital terrestrial broadcasts transmitted from the television station 5 via the transmission antenna 6, but the image distribution apparatus via the wireless communication network 2 and the Internet 3 like an Internet radio. 4 can also download and receive digital broadcasts.

FIG. 2 is a block diagram showing basic components of the mobile phone 1.
The central control unit 11 operates by supplying power from the battery unit 12 including a secondary battery, and controls the overall operation of the mobile phone 1 according to various programs in the storage unit 13 and a memory. (Not shown). The storage unit 13 includes a program storage unit M1, various information temporary storage units M2, an image storage unit M3, a 3D depth setting information storage unit M4, an output condition storage unit M5, and the like.

  The program storage unit M1 stores a program and various applications for realizing the present embodiment in accordance with the operation procedure shown in FIGS. 7 to 10, and stores information necessary for the program. Yes. Note that the storage unit 13 may include a removable portable memory (recording medium) such as an SD card or an IC card, or may be on a predetermined external server (not shown). The various information temporary storage unit M2 is a work area for temporarily storing various information necessary for the operation of the mobile phone 1, such as flag information and screen information. The image storage unit M3, 3D depth setting information storage unit M4, and output condition storage unit M5 will be described in detail later.

  The wireless communication unit 14 includes a wireless unit, a baseband unit, a demultiplexing unit, and the like. When the call function is in operation, when the signal is received from the receiving side of the baseband unit, demodulated into a received baseband signal and output to the central control unit 11, the central control unit 11 causes the audio signal processing unit 15 to Via the voice speaker SP, and the voice data input from the voice microphone MC is fetched from the voice signal processing unit 15, encoded into a transmission baseband signal, and then sent to the transmission side of the baseband unit. Transmission is output from the antenna AN1.

  The TV broadcast receiving unit 16 receives, for example, a one-segment terrestrial digital TV broadcast for a communication terminal device. The TV broadcast receiving unit 16 extracts a broadcast signal (analog circuit unit), and the received broadcast signal is OFDM (orthogonal frequency division). A digital circuit unit that performs demodulation, demultiplexes the decoded broadcast signal into video, audio, and data (character data), decompresses the compressed data, and the like. The imaging unit 17 is a component that constitutes a camera function, has an imaging element (for example, a CCD or a CMOS), and can capture moving images in addition to still image shooting. The imaging unit 17 includes a photographic lens, a distance measuring sensor, a light amount sensor, an analog processing circuit, a signal processing circuit, a compression / decompression circuit, and the like, and adjusts and controls the optical zoom, driving control during autofocus, and shutter driving control. , Control exposure, white balance, etc.

  The display unit 18 uses high-definition liquid crystal and displays, for example, character information, a standby image, a television image, and the like. A liquid crystal layer for generating a parallax barrier is formed on the surface of the display unit 18. Is provided. The 3D conversion processing unit 19 generates a right-eye image and a left-eye image according to the 3D depth during 3D display. The parallax barrier control unit 20 generates the right eye generated by the 3D conversion processing unit 19 during 3D display. The parallax barrier is generated on the surface layer of the display unit 18 so that the 3D display can be correctly seen in the state where the image for left and the image for the left eye are displayed on the display unit 18 at the same time.

  The operation unit 21 inputs dial input, character input, command input, and the like. The central control unit 11 executes processing according to an input operation signal from the operation unit 21. For example, power on / off processing is performed in response to a power on / off operation, or image acquisition processing is performed in response to an image acquisition operation such as a download operation instructing image download or a shooting operation instructing shooting. The RTC (real-time clock module) 22 constitutes a clock unit, and the central control unit 11 acquires the current date and time from the RTC 22. The notification unit 23 includes a sound speaker 23A, an LED (light emitting diode) 23B, and a vibration motor 23C. The notification unit 23 is driven at the time of an incoming call to notify an incoming call, and is also driven at the time of an alarm notification.

  The odor detection unit 24 and the temperature detection unit 25 detect an odor (type) and temperature drifting around the image when the imaging unit 17 captures an image, and the central control unit 11 The detection results of the detection unit 24 and the temperature detection unit 25 are acquired as scene reproduction information for reproducing a scene related to the captured image. This scene reproduction information is odor information and temperature information for reproducing a scene recognized by a sense of smell or touch that is a human sense other than vision. The odor detection unit 24 senses the odor (type) drifting around the image pickup unit 17 when an image is taken. The odor detection unit 24 includes a plurality of sensors that detect a voltage change due to the odor substance adhering to the sensitive film. And identifying the odor type based on detection signals from the plurality of sensors to identify the odor.

  The odor detection unit 24 can identify, for example, 10 types of odors except for an unpleasant bad odor, and the type and number are arbitrary and the configuration is also arbitrary. . The temperature detection unit 25 senses the ambient temperature (such as air temperature) when an image is captured by the imaging unit 17, and the temperature of a thermistor or the like that detects an electromotive force or resistance change in response to the temperature. Although it has a sensor and detects temperature change in units of 5 ° C., for example, the unit is arbitrary and the configuration is also arbitrary.

  The fragrance unit 26 and the heating / cooling unit 27 reproduce and output a scene based on scene reproduction information for reproducing a scene related to an image (photographed image, download image, etc.) displayed on the display unit 18. Stimulus (smell and temperature) that is recognized by human senses of smell and touch is generated and reproduced as a scene related to the image. That is, the fragrance portion 26 selectively generates a specified type of fragrance from among various fragrances such as mint and lemon (for example, 10 types), for example, charcoal cotton soaked with a fragrance. A fragrance is generated by energizing. In this case, the amount of fragrance generated (smell output intensity) is controlled by the amount of current supplied to the charcoal cotton. The heating / cooling unit 27 heats or cools the casing that constitutes the mobile phone 1. The heating / cooling unit 27 is a casing that contacts the hand when the user holds the mobile phone 1 in the hand. The heating / cooling output intensity is controlled within the heating / cooling range.

FIG. 3 is an external view of the mobile phone 1.
The mobile phone 1 is a two-axis hinge type that can be changed into a plurality of styles according to the connection state of a plurality of casings constituting the device main body (device main body), and includes an operation unit casing 1A and a display unit casing 1B. Are connected via a hinge 1C so as to be foldable and openable and rotatable. Such a two-axis hinge type mobile phone 1 has a configuration of the mobile phone 1 according to how the mobile phone 1 is connected to the operation unit housing 1A and the display unit housing 1B. The form can be changed to a plurality of styles, and the illustrated example shows an open style in which the operation unit housing 1A and the display unit housing 1B are not folded.

  An operation unit 21 is disposed on the inner surface of the operation unit housing 1A, and a display unit 18 is disposed on the inner surface of the display unit housing 1B. Further, a call microphone MC is disposed on the inner surface side of the operation unit casing 1A, and a call speaker SP is disposed on the inner surface of the display unit casing 1B. In addition, although not shown in FIG. 3, the heating / cooling unit 27 is embedded in a predetermined range on the side surface and the back surface of the operation unit housing 1 </ b> A. For example, in this embodiment, the heating / cooling unit 27 is embedded in the side surface and the back surface of the operation unit housing 1A over a predetermined range, but both side surfaces or one side surface of the operation unit housing 1A. Alternatively, the heating / cooling unit 27 may be embedded over a predetermined range. In this embodiment, the odor generated by the fragrance portion 26 is emitted from the sound hole of the call microphone MC. However, a large number of dedicated holes are provided on the upper surface of the operation unit casing 1A. You may make it discharge | release a fragrance from a hole.

FIG. 4 is a diagram for explaining the image storage unit M3.
The image storage unit M3 stores various images, and is configured to store “image”, “scene reproduction information”, and the like in association with each other. The “image” is an image photographed by the imaging unit 17 or an image downloaded and received from the image distribution device 4, and includes “image ID”, “title”, “type”, “3D depth”, and “actual data”. Have. “Image ID” indicates information for identifying an image, “Title” indicates an image name, and “Type” indicates whether the image is a moving image or a still image. “3D depth” indicates the depth of 3D display or the degree of pop-up (levels “0” to “2”). In order of increasing 3D depth, the relationship of level 2> level 1> level 0 is established. Yes. “Level 0” indicates that the 3D depth is “0 (zero)” and 2D (planar), and an image with “3D depth” being “level 0” is displayed in 2D. “Actual data” indicates actual image data such as a downloaded image and a captured image for each 3D depth.

  For example, if the corresponding image is a captured image, the “scene reproduction information” is detected by the odor detection unit 24 and the temperature detection unit 25 when the image is captured by the imaging unit 17 as described above. If the corresponding image is a download image, it is the odor information and temperature information downloaded and received together with the image. When the “type” is a moving image, the “scene reproduction information” corresponding thereto may be odor information or temperature information that changes over time. In the figure, “-” indicates that “scene reproduction information” does not exist corresponding to the image.

FIG. 5 is a diagram for explaining the 3D depth setting information storage unit M4.
The 3D depth setting information storage unit M4 is a memory in which the 3D depth when an image is first displayed is arbitrarily set, and includes “3D depth” and “setting flag”. The “setting flag” indicates which level of “3D depth” “level 0”, “level 1”, “level 2” is used for display, and when the value is “1” Indicates an effective level to be used, and “0” indicates an invalid level not to be used. In the illustrated example, “1” of the “setting flag” is set corresponding to “level 0” of “3D depth”, and when the image is displayed for the first time, 2D display is performed. The setting contents are as follows.

FIG. 6 is a diagram for explaining the output condition storage unit M5.
The output condition storage unit M5 is a memory in which the output intensity of odor or temperature is arbitrarily set for each level of “3D depth” when displaying an image. “3D depth”, “output intensity (odor, temperature)” ”And“ setting flag ”. That is, the output condition storage unit M5 is a table for determining the output intensity when reproducing the odor and temperature from the fragrance unit 26 and the heating / cooling unit 27, and the central control unit 11 includes the output condition storage unit M5. , The output intensity of odor and temperature is determined according to the “3D depth” when displaying the image.

  That is, in the illustrated example, “level 0” and “level 1” are provided as “output intensity” corresponding to “level 0” of “3D depth”, and “level 1” of “3D depth” is supported. “Level 0”, “Level 1” and “Level 2” as “Output Intensity”, and “Level 0” and “Level” as “Output Intensity” corresponding to “Level 3” of “3D Depth” 1 "and" level 2 ". The magnitude of “output intensity” has a relationship of “level 2”> “level 1”> “level 0”. Note that “level 0” of “output intensity” indicates that no odor or temperature is reproduced and output.

  The “setting flag” is set for each level of “output intensity” and indicates which level of “output intensity” is used to reproduce the odor and temperature, and its value is “1”. When it is, it indicates that it is an effective level to be used, and when it is “0”, it indicates that it is an invalid level that is not used. In the illustrated example, “level 0” of “3D depth” is set to “1” in “setting flag” corresponding to “level 0” of “output intensity”, and “level” of “3D depth”. “1” is set to “1” in the “setting flag” corresponding to “level 1” of “output intensity”, and “level 2” of “3D depth” is set to “level” of “output intensity” This is a case where “1” is set in the “setting flag” corresponding to “2”.

  As described above, a desired “output intensity” can be arbitrarily specified by a user operation from among a plurality of “output intensity” stored for each “3D depth” level. "May be set under the following conditions. That is, when the “3D depth” of the displayed image is larger than the predetermined 3D depth, the “output intensity” of the odor or temperature may be set to be a value larger than the predetermined intensity. . For example, if “3D depth” is “level 1” greater than the predetermined 3D depth “level 0”, “output intensity” is equal to or greater than “level 1” greater than the predetermined output intensity “level 0”. You may make it set as follows.

  On the contrary, when the “3D depth” of the displayed image is smaller than the predetermined 3D depth, the “output intensity” of the odor or temperature may be set to be a value smaller than the predetermined intensity. Good. For example, if “3D depth” is “level 1” smaller than the predetermined 3D depth “level 2”, the “output intensity” is equal to or lower than “level 1” smaller than the predetermined output intensity “level 1”. You may make it set as follows. Further, when “3D depth” is smaller than a predetermined 3D depth, for example, “level 1”, “output intensity” may be set to “0 (zero)”. Further, when “3D depth” is “level 0”, “output intensity” may be set to “0 (zero)”. In the example shown, the odor “output intensity” and the temperature “output intensity” are the same, but the odor “output intensity” and the temperature “output intensity” are set separately. A possible configuration may be used.

  Next, the operation concept of the mobile phone 1 in this embodiment will be described with reference to the flowcharts shown in FIGS. Here, each function described in these flowcharts is stored in the form of a readable program code, and operations according to the program code are sequentially executed. Further, it is possible to sequentially execute the operation according to the above-described program code transmitted via a transmission medium such as a network. In other words, in addition to the recording medium, an operation unique to the present embodiment can be executed using a program / data supplied externally via a transmission medium.

FIG. 7 is a flowchart showing an outline of the overall operation on the mobile phone 1 side which is started to be executed when the power is turned on.
First, when a power-on operation for turning on the power is performed (YES in step A1), the central control unit 11 of the mobile phone 1 executes a power-on process for starting power supply and initializing a predetermined memory or the like. After that, a standby process is performed in which a predetermined standby image is read and displayed, or communication is performed with the base station 2A to register a position (step A2). Whether a setting operation has been performed (step A3), an image acquisition operation has been performed (step A5), an occurrence of an image display instruction has been detected (step A7), or an occurrence of a 3D depth change instruction has been detected. (Step A9), whether an incoming call has been detected (Step A11), whether a power-off operation has been performed (Step A13), or other operations have been performed (Step A15).

  When a setting operation for setting arbitrary information is performed (YES in step A3), for example, settings for the 3D depth setting information storage unit M4 and the output condition storage unit M5 are set as the setting process corresponding to the operation. Is performed (step A4). That is, the 3D depth setting information storage unit M4 and the output condition storage unit M5 are set so that the contents of the “setting flag” can be arbitrarily rewritten. When such setting processing ends, the process returns to step A3 described above. Further, when an image acquisition operation is performed, that is, when an image acquisition operation such as a download operation instructing image download or a shooting operation instructing shooting is performed (YES in step A2), an image acquisition process is performed. After that (step A6), the process returns to step A3 described above. This image acquisition process is a process of acquiring an image and storing it in association with the scene reproduction information in the image storage unit M3.

FIG. 8 is a flowchart for explaining in detail the image acquisition process (step A6 in FIG. 7).
First, the central control unit 11 checks whether the image acquisition operation instructing the start of the image acquisition process is a download operation (step B1), a shooting operation, or another operation (step B4). . If the image acquisition operation is a download operation (YES in step B1), scene reproduction information (scent information and temperature) for reproducing a scene related to the image from the image distribution device 4 together with the image (moving image or still image). Information) is downloaded and received (step B2). Then, the downloaded image and the scene reproduction information are associated with each other and stored in the image storage unit M3 (step B3).

  If the image acquisition operation is a shooting operation (YES in step B4), the imaging unit 17 is driven to acquire a captured image (step B5), and the odor detection unit 24 and the temperature detection unit 25 are used during the shooting. Is acquired (step B6), and scene reproduction information is generated (step B7). Then, the acquired captured image and the scene reproduction information are associated with each other and stored in the image storage unit M3 (step B3). If the image acquisition operation is performed by other operations other than the download operation and the shooting operation (NO in step B4), the scene reproduction information is acquired together with the image according to the other operations (step B8), and then acquired. The captured image and its scene reproduction information are associated with each other and stored in the image storage unit M3 (step B3). For example, if the other operation is a copy operation for copying an image from an external storage medium, after acquiring the scene reproduction information together with the image from the external storage medium (step B8), the acquired captured image and the scene reproduction information are obtained. The image is stored in the image storage unit M3 (step B3). When such an image acquisition process is completed, the process returns to step A3 in FIG.

  Further, when the generation of an image display instruction is detected (YES in step A7 in FIG. 7), that is, when an image display is instructed by a user operation or as in a slide show, a plurality of images are switched and displayed one after another at a predetermined time. When the timing for switching is detected, the image display process is performed (step A8), and the process returns to step A3 described above. This image display process is a display process for displaying an image in 2D / 3D, and scene reproduction information (scent information, temperature information) is reproduced with output intensity corresponding to the 3D depth in 3D display in conjunction with this display process. This is a process to output.

9 and 10 are flowcharts for explaining the image display process (step A8 in FIG. 7) in detail.
First, the central control unit 11 stores, for example, in the image storage unit M3 as other factors as to whether an image display instruction is generated by a television program viewing operation (step C1) or a moving image streaming viewing operation. It is checked whether it is due to the display operation of the displayed image or the switching display (automatic display) to the next image (step C2). Now, if the operation is based on a TV program viewing operation (YES in step C1), the TV function is activated and a receiving operation for receiving an image (TV program) and scene reproduction information related to the image from the TV broadcast receiving unit 16 is performed. After starting (step C3), the received image (television program) and the scene reproduction information are temporarily stored in association with various information temporary storage units M2 (step C6).

  Further, if the image display instruction is generated by a video streaming viewing operation (YES in step C2), scene reproduction information (scent information) for reproducing a scene related to the image together with the image (video) from the image distribution apparatus 4 , Temperature information) is started (step C4), and the received image (moving image) and the scene reproduction information are associated with each other and temporarily stored in the various information temporary storage unit M2 (step C6). Further, if the image display instruction is generated due to other factors (NO in step C2), after the display target image and its scene reproduction information are read from the image storage unit M3 according to the other factors ( In step C5), the display target image and the scene reproduction information are temporarily stored in association with the various information temporary storage unit M2 (step C6).

  Then, the 3D depth whose “setting flag” is “1” is read from the 3D depth setting information storage unit M4 (step C7), and the 3D depth is designated as display control information (step C8). The 3D depth (specified depth) is temporarily stored in the various information temporary storage unit M2 (step C9). Then, it is checked whether or not the 3D depth (specified depth) specified as described above is “level 0” (step C10). If the specified depth is “level 0” (YES in step C10), The process proceeds to 2D display processing (steps C11 and C12). In this case, if a parallax barrier is generated on the display unit 18, the parallax barrier is erased by the parallax barrier control unit 20 (step C11), and then an image temporarily stored in the various information temporary storage unit M2 is read. Then, 2D display is performed on the display unit 18 (step C12).

  If the designated depth is not “level 0” (NO in step C10), that is, if it is “level 1” or “level 2”, the process proceeds to 3D display processing (steps C14 to C16). In this case, it is checked whether the image with the specified depth is temporarily stored in the various information temporary storage unit M2, that is, whether the temporarily stored image is the image with the specified depth (step C13). Here, if the image is not an image of the specified depth (NO in step C13), for example, if the received program (image) for television broadcasting or the received image of the video streaming is not an image of the specified depth, or “actual data” of the specified depth is If the temporarily stored image is not an image of the specified depth, such as when not stored in the image storage unit M3, the 3D conversion processing unit 19 is operated to store the temporarily stored image (different from the specified depth). (3D depth image) is converted into an image with a specified depth (step C14). In this case, the 3D conversion processing unit 19 generates a right-eye image and a left-eye image according to the specified depth. If no parallax barrier is generated on the display unit 18, the parallax barrier is generated by the parallax barrier control unit 20 (step C15), and then the temporarily stored image is displayed in 3D on the display unit 18 at a specified depth. Or 3D display of the 3D converted image (step C16).

  Then, the process proceeds to step C17 in FIG. 10, the output condition storage unit M5 is referred to based on the above-described designated depth, and among the “output strengths” of “3D depth” corresponding to the designated depth, the “setting flag” is Read the “output intensity” of “1”. For example, if the specified depth is “level 0”, “output intensity” of “level 0”, if the specified depth is “level 1,” “output intensity” of “level 1”, and further, the specified depth If “level 2”, “output intensity” of “level 2” is read. Thereby, the “output intensity” read from the output condition storage unit M5 is determined as the output intensity when reproducing and reproducing the scene reproduction information (scent information, temperature information) (step C18).

  Then, it is checked whether or not the “output intensity” determined as described above is “level 0” (step C19). If the “output intensity” is “level 0” (YES in step C19), the fragrance 26 The process returns to step A3 in FIG. 7 without driving the heating / cooling unit 27 (without reproducing and outputting the scene), but if the “output intensity” is not “level 0” (NO in step C19), the fragrance The unit 26 and the heating / cooling unit 27 are driven (step C20). In this case, the central control unit 11 reads the scene reproduction information (smell information, temperature information) temporarily stored in the various information temporary storage unit M <b> 2 and provides the information to the fragrance unit 26 and the heating / cooling unit 27.

  Thereby, the fragrance unit 26 and the heating / cooling unit 27 reproduce and output the scene according to the scene reproduction information (scent information, temperature information) and the “output intensity” determined as described above (step C21). . That is, the fragrance unit 26 generates a kind of odor according to the scene reproduction information (scent information), and controls the amount of fragrance generated according to the determined “output intensity”. In addition, the heating / cooling unit 27 heats / cools a part of the casing (for example, the side surface portion and the back surface portion) at a temperature corresponding to the scene reproduction information (temperature information). ”To control the heating / cooling range. After the scene is reproduced and output in this way (step C21), the process returns to step A3 in FIG. 7. However, if the above-described temporarily stored image is a moving image image such as a TV program or a moving image, the image display ends. 9 and 10 are sequentially repeated until a user operation or an automatic instruction is issued.

  On the other hand, when the occurrence of a 3D depth change instruction is detected (YES in step A9 in FIG. 7), that is, when it is instructed to be changed to a desired 3D depth by a user operation, or the 3D depth is automatically changed. Is designated as the display control information (step A10), the process proceeds to step C9 in FIG. 9, and the changed 3D depth (designated depth) is stored in various information temporary storage units. Temporarily store in M2. Thereafter, the image display processing of FIGS. 9 and 10 is performed (steps C10 to C21). As a result, the image display and the scene reproduction output correspond to the changed 3D depth (designated depth).

  In addition, when an incoming call is detected (YES in step A11 in FIG. 7), a call process is performed according to the response operation (step A12), and the process returns to step A3 described above. When a power-off operation is detected (YES in step A13), the power-off process is performed (step A14), and then this flow ends. However, when other operations such as a calling operation are performed (step S14) After a transmission process is performed as a process corresponding to the operation (step A16), the process returns to the above-described step A3.

  As described above, in this embodiment, the output intensity of the scene related to the image is determined based on the 3D depth when the image is displayed in 3D, and the scene is reproduced and output with this output intensity. It is possible to appropriately control the reproduction output of the scene related to the image, and it is possible to further enhance the sense of presence and the production effect by the sense of solidarity with the image and the sense of power.

  The stimulus (smell and temperature) that is recognized by human senses other than sight and sense of touch is generated and reproduced as a scene related to the image. Tactile) can be used.

  Since the intensity of the odor is controlled based on the 3D depth when the image is displayed in 3D, the intensity of the odor can be increased or decreased according to the depth or pop-out of the 3D display, which has a production effect. .

  Since the range for heating or cooling the housing is controlled based on the 3D depth when displaying the image in 3D, the strength of heating or cooling can be adjusted according to the depth or pop-out of the 3D display. It has an effect.

  The surrounding scene at the time of shooting is detected and stored as scene reproduction information in association with the captured image, and the scene reproduction information associated with this captured image is read out and reproduced and output. It is possible to freely reproduce the scene at any time.

  When the 3D depth is larger than the predetermined 3D depth, the output intensity is determined to be larger than the predetermined intensity. This not only gives the user a visual sense of reality, but also adjusts the sense other than the visual sense accordingly. It is possible to give a sense of realism that is used, and as a result, it is possible to give the user a total sense of realism.

  When the 3D depth is smaller than the predetermined 3D depth, the output intensity is determined to be smaller than the predetermined intensity, so that not only the visual sensation given to the user is reduced, but also the sense other than the visual sense is adjusted accordingly. It is possible to reduce the sense of reality using the computer, reduce the burden of unnecessary processing, and prevent inconvenience to the surroundings.

  When the 3D depth is smaller than the predetermined 3D depth or when it is set to “0 (zero)”, the output intensity is determined to be “0 (zero)”. can do.

  When the 3D depth is changed, the scene is reproduced and output with the output intensity corresponding to the changed 3D depth. Therefore, even if the 3D depth is changed, it is possible to reproduce and output the corresponding scene.

  It is possible to specify whether the image is displayed in 2D or 3D. When 3D display is specified, the image is displayed in 3D and the scene related to the image is reproduced and output. Based on whether or not 3D images are displayed, it is possible to appropriately control the reproduction output of the scene related to the image, and it is possible to enhance the sense of reality and the production effect by the reproduction output of the scene according to the 3D display.

  When the 2D display is designated, the image is displayed in 2D, so that at the time of 2D display, the reproduction output of the scene related to the image can be suppressed.

  In the embodiment described above, “level 0”, “level 1”, and “level 2” are exemplified as the 3D depth and the scene output intensity, but may be “level 3” or higher. Moreover, although the amount of odor generated is shown as the odor output intensity, it may be the odor concentration, and the range of heating or cooling is shown as the temperature output intensity. Good. Moreover, although temperature was illustrated as a tactile sense, vibration, humidity, a touch feeling, etc. may be sufficient. In the embodiment described above, the scene is reproduced and output using the sense of smell and touch, but the sense of hearing and taste may be used. In this case, a microphone or a taste sensor may be provided. In this case, the output intensity of the sound is a volume, and the output intensity of the taste is a taste density. Furthermore, the scene may be reproduced and output by a combination of smell, touch, hearing, and taste.

  In the embodiment described above, when an image is downloaded and acquired, a television program (image) is received and acquired, or a copy of an image is acquired from an external storage medium, scene reproduction information regarding the image is also acquired together with the image. However, the scene reproduction information may be obtained by analyzing the image. That is, the image is analyzed to identify the subject included in the image, and the scene reproduction information is generated by identifying whether the subject generates an odor, a high temperature, or a low temperature. You may make it do. For example, if the subject included in the image is lemon or hot spring, scene reproduction information indicating the smell of lemon and scene reproduction information indicating the smell or temperature of hot spring (sulfur) may be generated and stored. If scene reproduction information is generated and stored in association with the image as described above, the scene reproduction information may be read out and the scene reproduced and output when the image is reproduced and output next time.

  In the above-described embodiment, when a television program (image), a moving image streaming image, or the like is not an image with a specified depth, or when “actual data” with a specified depth is not stored in the image storage unit M3, it is temporarily stored. If the image is not an image of the specified depth, the 3D conversion processing unit 19 is operated to perform 3D conversion of the temporarily stored image (an image of another 3D depth different from the specified depth) to the specified depth. However, a request to transmit an image with a specified depth may be made to the image distribution device 4 and received by receiving an image with the specified depth transmitted from the image distribution device 4 in response thereto.

  In each of the embodiments described above, the 3D image is displayed simultaneously with the right-eye image and the left-eye image corresponding to the 3D depth, and a parallax barrier corresponding to the 3D depth is displayed so that they can be correctly viewed in 3D. Although the method of generating the 3D display on the surface layer of the display unit 18 has been illustrated, for example, any 3D display method such as a method of performing the 3D display by applying a visual effect that makes the image look 3D may be used. . The type of image displayed on the display unit 18 is arbitrary, such as a still image, a moving image, a television program, and an image including text.

  In addition, the display unit 18 may be an arbitrary external display device such as an external monitor. The terminal device is not limited to the mobile phone 1 and may be any terminal device such as a personal computer, a PDA, a digital camera, or a music player. Furthermore, it is not limited to a folding type terminal device, but may be an arbitrary housing structure such as a straight type, a slide type, or a spin top type.

Further, the “device” and “machine” shown in each of the above-described embodiments may be separated into a plurality of housings by function, and are not limited to a single housing. In addition, each step described in the above-described flowchart is not limited to time-series processing, and a plurality of steps may be processed in parallel or separately.

DESCRIPTION OF SYMBOLS 1 Mobile phone 2 Wireless communication network 4 Image distribution apparatus 11 Central control part 13 Storage part 16 Television broadcast receiving part 17 Imaging part 18 Display part 19 3D conversion process part 20 Parallax barrier control part 21 Operation part 24 Odor detection part 25 Temperature detection part 26 Fragrance unit 27 Heating / cooling unit M1 Program storage unit M2 Various information temporary storage M3 Image storage unit M4 3D depth setting information storage unit M5 Output condition storage unit

Claims (16)

A terminal device that displays an image in 3D on a display unit,
Output means for reproducing and outputting the scene based on scene reproduction information for reproducing the scene related to the image displayed on the display unit;
Designating means for designating a 3D depth indicating a 3D depth of the image displayed on the display unit or a degree of popping;
Display control means for 3D displaying an image on the display unit at a 3D depth designated by the designation means;
Determining means for determining an output intensity when reproducing and outputting a scene by the output means based on the 3D depth specified by the specifying means;
Output control means for reproducing and outputting a scene from the output means at the output intensity determined by the determining means;
A terminal device comprising: The scene reproduction information is information for reproducing a scene recognized by at least one of the senses of smell, touch, hearing, and taste, which are human senses other than vision,
The output means generates, as the scene, a stimulus for recognizing at least one of olfactory sense, touch sense, auditory sense, and taste sense that is a human sense other than vision based on the scene reproduction information. Output
The terminal device according to claim 1, which is configured as described above. The scene reproduction information is information relating to odor, and the output intensity when the scene is output by the output means is the intensity of the odor.
The terminal device according to claim 2. The scene reproduction information is information relating to temperature, and the output intensity when the scene is output by the output means is a range in which the casing constituting the terminal device is heated or cooled.
The terminal device according to claim 2. Photographing means for photographing an image;
A scene detection unit that detects a surrounding scene at the time of shooting an image by the shooting unit and acquires the scene reproduction information;
Storage control means for associating the image photographed by the photographing means with the scene reproduction information detected by the scene detection means and storing them in a storage unit;
With
The output control means reads out scene reproduction information associated with an image displayed on the display unit from the storage unit and reproduces and outputs a scene from the output unit.
The terminal device according to claim 1, which is configured as described above. The determining means determines the intensity larger than a predetermined intensity when the 3D depth specified by the specifying means is larger than a predetermined 3D depth.
The terminal device according to claim 1, which is configured as described above. The determining unit determines an output intensity when reproducing and outputting a scene to be smaller than a predetermined intensity when the specified 3D depth specified by the specifying unit is smaller than a predetermined 3D depth;
The terminal device according to claim 1, which is configured as described above. The determining means determines that the output intensity when reproducing and outputting a scene is zero when the 3D depth specified by the specifying means is smaller than a predetermined 3D depth.
The terminal device according to claim 7, which is configured as described above. The determining means determines that the output intensity when reproducing and outputting the scene is zero when the 3D depth specified by the specifying means is zero.
The terminal device according to claim 7, which is configured as described above. A change detecting means for detecting that the 3D depth of the image displayed on the display unit has been changed;
When the change detecting unit detects that the 3D depth has been changed, the determining unit newly determines an output intensity for reproducing and outputting a scene based on the 3D depth after the change,
The output control means switches the output intensity of the scene reproduced and output from the output means to the output intensity newly determined by the determination means, and reproduces and outputs the scene.
The terminal device according to claim 1, which is configured as described above. A terminal device that displays images in 2D and 3D on a display unit,
Based on the scene reproduction information stored in the storage unit in a state where the scene reproduction information for reproducing the scene related to the image displayed on the display unit is stored in the storage unit in association with the image. An output means for reproducing and outputting a scene;
Designating means for designating whether the image is displayed in 2D or 3D on the display unit;
Discrimination means for discriminating whether or not 3D display is designated by the designation means;
Display control means for displaying the image in 3D on the display unit when it is determined that 3D display is designated by the determination means;
Output control means for reproducing and outputting a scene from the output means based on the scene reproduction information stored in the storage unit corresponding to the image displayed in 3D on the display unit;
A terminal device comprising: The display control unit displays the image on the display unit in 2D when it is determined that the 2D display is designated by the determination unit.
The terminal device according to claim 11, which is configured as described above. The scene reproduction information is information for reproducing a scene recognized by at least one of the senses of smell, touch, hearing, and taste, which are human senses other than vision,
The output means generates, as the scene, a stimulus for recognizing at least one of olfactory sense, touch sense, auditory sense, and taste sense that is a human sense other than vision based on the scene reproduction information. Output
The terminal device according to claim 11, which is configured as described above. Photographing means for photographing an image;
A scene detection means for detecting a surrounding scene at the time of shooting an image as the scene reproduction information by the shooting means;
Storage control means for associating the image photographed by the photographing means with the scene reproduction information detected by the scene detection means and storing them in a storage unit;
Comprising
The terminal device according to claim 11, which is configured as described above. Against the computer,
A function to reproduce and output the scene based on the scene reproduction information for reproducing the scene related to the image displayed on the display unit;
A function of designating a 3D depth indicating a 3D depth of the image displayed on the display unit or a degree of protrusion;
A function of displaying a 3D image on the display unit at the specified 3D depth;
A function of determining an output intensity when reproducing and outputting the scene based on the specified 3D depth;
A function to reproduce and output a scene at the determined output intensity;
A program to realize Against the computer,
In a state where scene reproduction information for reproducing a scene related to an image displayed on the display unit is stored in the storage unit in association with the image, the scene is based on the scene reproduction information stored in the storage unit. A function to reproduce and output
A function for designating whether the image is displayed in 2D or 3D on the display unit;
A function of determining whether or not the 3D display is designated;
A function of displaying the image on the display unit in 3D when it is determined that the 3D display is designated;
A function for reproducing and outputting a scene from the output unit based on the scene reproduction information stored in the storage unit corresponding to the image displayed in 3D on the display unit;
A program to realize

Images