JP2007052484A - Image processor, image processing method, program for image processing, and information recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for visually recognizing a desired scenery regardless of weather, and for easily acquiring such accessory information as the designation of natural features in the scenery, and for adding such an added value as an interest to the scenery observation in an observing device which is present in a scenic spot. <P>SOLUTION: This image processor is configured to optically acquire image information Sv corresponding to a visual field image equivalent to a visual field range in a camera 1 of an observer who operates the camera 1 for observing a scenery, and to preliminarily store relevant information Sm relative to an object present in the visual field range, and to generate image information Smx for composition correspondingly to the object in the visual field, and different from the original image information Sv on the basis of the image information Sv and the relevant information Sm, and to make a display part 2 display either an image including only the image information Smx for composition or an image acquired by superimposing the image information Smx for composition on the image information Sv to the observer. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention belongs to the technical field of an image processing device, an image processing method, an image processing program, and an information recording medium. More specifically, for example, an observation device for observing a landscape installed on an observation platform for observing the landscape is provided. The present invention belongs to the technical field of image processing apparatuses and image processing methods, image processing programs used in the image processing apparatuses, and information recording media on which the image processing programs are recorded.

  Traditionally, so-called tourist attractions (castles, high-rise towers, high-rise buildings or summits), art museums, museums, etc., in order to observe the scenery and exhibits that can be seen from there in more detail, for example, binoculars and telescopes (tourist attractions) ) Or an observation device such as a microscope (in the case of a museum, etc.) is installed.

  Also, in recent years when the network environment such as the Internet has developed, a video camera or a TV camera installed on a high place such as a rooftop of a high-rise building is operated via the network, and the video camera is placed on an observer's personal computer. It is also performed to display an image captured by the above.

On the other hand, as a technique for artificially synthesizing another image with an image of an actual feature observed by this type of observation apparatus, for example, there is a technique described in Patent Document 1 below. In the technique described in Patent Document 1, for example, an image of a building in a ruin that is imagined as a thing at the time when the ruins actually existed is displayed at the current position of the ruins in the Asuka period. It is the composition to do.
JP 2003-264740 A

  However, the conventional observation apparatus such as the binoculars and the microscope described above has a problem that the scenery to be observed may not be observed at all in bad weather such as rainy weather or fog. .

  In addition, there is a problem that even if the scenery etc. can be observed in good weather, it may be difficult to distinguish the names of tourist attractions etc. existing in the scenery (view range).

  Furthermore, the above-described observation device has a problem that it is not interesting as a tourist attraction because it only looks at features within the field of view.

  Moreover, in the technique described in Patent Document 1 described above, a virtual image created is only displayed. Therefore, in view of the use as an observation apparatus as described above, there may be cases where it is not always possible to draw out enough interest. There is a problem that there is.

  Therefore, the present invention has been made in view of the above-mentioned problems, and the problem thereof is that the above-described observation device or the like can visually recognize a desired scenery regardless of the weather and Image processing apparatus, image processing method, image processing program used in the image processing apparatus, and image processing program capable of easily adding incidental information such as the name of an object and adding added value such as interest to scenery observation An object is to provide an information recording medium on which an image processing program is recorded.

  In order to solve the above-described problem, the invention according to claim 1 provides image information corresponding to a visual field image corresponding to a visual field range in the observation device of an observer who operates the observation device for observing a landscape. Acquisition means such as an optically acquired camera, storage means such as a storage unit for storing related information related to objects existing in the visual field range, the acquired image information, and the stored related information Based on the generation means, such as a distribution / combination unit for generating correspondence information corresponding to the object and different from the original image information, and only the generated correspondence information or the generated correspondence information Output means such as a display unit for outputting any one of the superimposition information obtained by superimposing the acquired image information to the observer.

  Therefore, correspondence information corresponding to the image within the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the image information corresponding to the image within the visual field range are superimposed. Since any one of the superimposition information is output to the observer, not only a monotonous image within the visual field range but also various information corresponding to the image can be provided to the observer.

  In order to solve the above-described problem, the invention according to claim 2 is the image processing apparatus according to claim 1, wherein the generation unit includes an attitude detection unit such as a control unit that detects an attitude of the observation device; A viewpoint detection unit such as a recognition unit that detects a position of the observer's viewpoint within the field of view based on the detected posture, and a position within the field of view based on the position of the detected viewpoint. An extraction unit such as a recognition unit for extracting the object of interest; and a reading unit such as a processing unit for reading out the related information corresponding to the extracted object from the storage unit. The correspondence information is generated based on related information and the acquired image information.

  Therefore, since the viewpoint of the observer is detected from the posture of the observation apparatus, the correspondence information is generated from the related information corresponding to the object on the viewpoint and output to the observer, so that the current viewpoint of the observer is present. Related information regarding an object (that is, an object in which the observer is interested) can be output to the observer in a timely manner.

  In order to solve the above problem, the invention according to claim 3 is the image processing apparatus according to claim 1 or 2, wherein the storage means stores character information corresponding to the object within the field of view. The related information is stored, and the generation unit is configured to generate the correspondence information by superimposing the character information on a position in the image information where the object corresponding to the character information exists. .

  Therefore, text information corresponding to objects within the field of view is stored as related information, and this information is superimposed on the corresponding object position as correspondence information, so that the observer can easily explain the object currently being viewed. Can be obtained.

  In order to solve the above-mentioned problem, the invention according to claim 4 is the image processing apparatus according to any one of claims 1 to 3, wherein the storage means is the other in the visual field range. Concealed object image information corresponding to an image of the object that is hidden by an object and not visible is stored as the related information, and the generation unit includes the object corresponding to the concealed object image within the field of view. The concealed object image corresponding to the other object being concealed is replaced with the concealed object image to generate the correspondence information.

  Therefore, the concealed object image information corresponding to an object that cannot be viewed within the normal visual field range is stored as related information, and the concealed object image corresponding to another object concealing the object is replaced with the concealed object image. Since the correspondence information is generated, an image of an object that cannot be visually recognized can be provided to the observer.

  In order to solve the above-described problem, according to a fifth aspect of the present invention, in the image processing device according to any one of the first to fourth aspects, the storage unit is configured to store the visual field image at a time different from the current time. Is stored as the related information, and the generation unit generates the other time image corresponding to the same visual field range as the corresponding information instead of the image information at the current time. Configured as follows.

  Therefore, an image in the visual field range at a time different from the current time is stored as related information, and correspondence information is generated using this, so that the current or past state in the visual field range currently being viewed can be easily obtained. Can be provided to the observer.

  In order to solve the above-described problem, the invention according to claim 6 is the image processing apparatus according to any one of claims 1 to 5, wherein the storage unit corresponds to the field of view range and the observation is performed. A far object image corresponding to an object farther from the horizon as viewed from the person is stored as the related information, and the output means displays the far object image corresponding to the image information with respect to the current image information. It is configured to superimpose and output.

  Therefore, since a far object image corresponding to an object that cannot be physically seen by being farther from the horizon is stored as related information, more interesting information can be provided to the observer.

  In order to solve the above-described problem, the invention according to claim 7 is the image processing apparatus according to any one of claims 1 to 6, wherein the generation unit is any of the field-of-view ranges. The correspondence information for displaying the object in a more emphasized manner than the other objects is generated.

  Therefore, correspondence information for displaying any object existing in the field of view with emphasis over other objects is generated.For example, even when it is cloudy, rainy, or at night, the object that the observer wants to see is displayed. It can be clearly distinguished from the object and presented to the observer.

  In order to solve the above problem, an invention according to an eighth aspect is the image processing apparatus according to any one of the first to seventh aspects, wherein the storage means corresponds to the object in the visual field range. Sound information for outputting a sound to be stored is stored as the related information, and the output means uses the sound information corresponding to the sound as the sound corresponding to any one of the currently output sounds. Sound output means.

  Therefore, since sounds related to objects within the current visual field range can be provided to the observer, the observer can be made to observe more interestingly.

  In order to solve the above-described problem, the invention according to claim 9 provides image information corresponding to a visual field image corresponding to a visual field range in the observation device of an observer who operates the observation device for observing a landscape. Corresponding to the object and different from the original image information based on the acquisition step optically acquired, the related information related to the object existing in the visual field range, and the acquired image information Either a generation step of generating information and only the generated correspondence information, or superimposition information obtained by superimposing the generated correspondence information on the acquired image information, to the observer An output step of outputting.

  Therefore, correspondence information corresponding to the image within the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the image information corresponding to the image within the visual field range are superimposed. Since any one of the superimposition information is output to the observer, not only a monotonous image within the visual field range but also various information corresponding to the image can be provided to the observer.

  In order to solve the above-described problem, the invention according to claim 10 is directed to a computer included in an image processing apparatus including an observation apparatus for observing a landscape, and the observation apparatus of an observer who operates the observation apparatus. An acquisition means for optically acquiring image information corresponding to a visual field image corresponding to a visual field range in the storage, a storage means for storing related information related to an object existing in the visual field range, the acquired image information, and the storage Generating means for generating correspondence information corresponding to the object and different from the original image information, and only the generated correspondence information, or the generated correspondence information. One of the superimposition information obtained by superimposing the acquired image information is caused to function as output means for outputting to the observer.

  Therefore, correspondence information corresponding to the image within the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the image information corresponding to the image within the visual field range are superimposed. Since the computer functions so as to output any of the superimposition information to the observer, not only a monotonous image within the field of view but also various information corresponding to the image can be provided to the observer. .

  In order to solve the above-mentioned problem, the invention according to claim 11 is recorded with the image processing program according to claim 10 readable by the computer.

  Therefore, by reading out and executing the image processing program by a computer, correspondence information corresponding to the image within the visual field range of the observation apparatus is generated based on the related information, and only the correspondence information or the correspondence information and the correspondence information are generated. Since the computer functions to output to the observer either superimposition information obtained by superimposing image information corresponding to an image in the field of view range, not only the monotonous image in the field of view but the Various information corresponding to the image can be provided to the observer.

  As described above, according to the first aspect of the present invention, the correspondence information corresponding to the image within the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the correspondence information are generated. Since either superimposition information obtained by superimposing image information corresponding to an image in the field of view range is output to the observer, not only a monotonous image in the field of view but also various information corresponding to the image Can be provided to the observer.

  Therefore, an image rich in added value can be provided to the observer.

  According to the invention described in claim 2, in addition to the effect of the invention described in claim 1, the viewpoint of the observer is detected from the posture of the observation device, and the response is performed from related information corresponding to the object on the viewpoint. Since information is generated and output to the observer, related information regarding an object at the current viewpoint of the observer (that is, an object in which the observer is interested) can be output to the observer at a good timing. it can.

  According to the invention described in claim 3, in addition to the effect of the invention described in claim 1 or 2, the character information corresponding to the object in the visual field range is stored as related information, and this is stored in the corresponding object. Since the corresponding information is superimposed on the position, the observer can easily obtain an explanation of the object currently being viewed.

  According to the invention described in claim 4, in addition to the effect of the invention described in any one of claims 1 to 3, related object image information corresponding to an object that cannot be viewed within the normal visual field range is related. Stores information as information and replaces the concealed object image corresponding to another object concealing the object with the concealed object image to generate correspondence information, thus providing the observer with an image of an object that cannot be normally viewed can do.

  According to the invention described in claim 5, in addition to the effect of the invention described in any one of claims 1 to 4, an image in the visual field range at a time different from the current time is stored as related information. Since the correspondence information is generated using this, it is possible to easily provide the observer with the current or past state within the currently viewed visual field range.

  According to the invention described in claim 6, in addition to the effect of the invention described in any one of claims 1 to 5, a distant place corresponding to an object that cannot be physically seen by being further away from the horizon. Since the object image is stored as the related information, more interesting information can be provided to the observer.

  According to the invention described in claim 7, in addition to the effect of the invention described in any one of claims 1 to 6, any object existing in the visual field range is emphasized more than the other objects. Since the correspondence information for display is generated, for example, even when it is cloudy, rainy, or at night, the object that the observer wants to see can be clearly distinguished from other objects and presented to the observer.

  According to the eighth aspect of the present invention, it is possible to provide the observer with sound related to an object within the current visual field range, so that the observer can make observation more interesting.

  According to the invention described in claim 9, correspondence information corresponding to an image in the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the image in the visual field range are generated. Since either superimposition information obtained by superimposing corresponding image information is output to the observer, not only a monotonous image within the field of view but also various information corresponding to the image is provided to the observer. be able to.

  Therefore, an image rich in added value can be provided to the observer.

  According to the invention described in claim 10, the correspondence information corresponding to the image within the visual field range of the observation device is generated based on the related information, and only the correspondence information or the correspondence information and the image within the visual field range are generated. Since the computer functions to output to the observer one of the superimposition information obtained by superimposing the corresponding image information, various information corresponding to the image as well as the monotonous image within the field of view range. Can be provided to the observer.

  Therefore, an image rich in added value can be provided to the observer.

  According to the eleventh aspect of the present invention, by reading and executing the image processing program by a computer, correspondence information corresponding to an image within the visual field range of the observation device is generated based on the related information, and the correspondence Since the computer functions to output to the observer either the information alone or the superimposition information obtained by superimposing the correspondence information and the image information corresponding to the image in the visual field range, the visual field range In addition to the monotonous image, various information corresponding to the image can be provided to the observer.

  Therefore, an image rich in added value can be provided to the observer.

  Next, the best mode for carrying out the present invention will be described with reference to the drawings. In the embodiment described below, the present invention is applied to an image processing apparatus that presents (displays) an image to an observer via, for example, a telescope or binoculars installed on an observation deck of a high-rise building. It is an embodiment.

  First, the overall configuration and operation of the image processing apparatus according to the embodiment will be described with reference to FIGS. 1 to 3. 1 is a block diagram showing a schematic configuration of the entire image processing apparatus according to the embodiment, FIG. 2 is a block diagram showing a detailed configuration of the image processing apparatus, and FIG. 3 is an overall operation of the image processing apparatus. It is a flowchart which shows.

  As shown in FIG. 1, the image processing apparatus S according to the embodiment includes a camera 1 as an acquisition unit that captures a scene viewed from the observation deck and the like, and an output unit that displays an image or the like corresponding to the captured scene. As a display unit 2, a control unit 3 as an attitude detection unit, an input unit 4 including a keyboard or a touch panel, a distribution / combination unit 5 as a generation unit, and a processing unit 6.

  In this configuration, the camera 1 is more specifically configured by a video camera or a so-called web camera, and acquires the scenery or the like as an optical image and electrically amplifies or encodes it. Then, image information Sv is generated and output to the distribution / synthesis unit 5. It should be noted that the camera 1 may be provided with a microphone so that sound corresponding to the image that is currently converted into the image information Sv is simultaneously collected and output to the distribution / synthesis unit 5 together with the image information Sv as audio information. it can. In addition, the camera 1 can be configured by modifying the above-described conventional telescope and binoculars and installing the microphone and the sensor for generating the image information Sv.

  Next, the control unit 3 attached to the camera 1 detects the current state of the camera 1 (including a three-dimensional posture, a focal length, a connection (operation) state, etc. The same applies hereinafter). Status information Spos indicating the status is generated and output to the processing unit 6. In addition to this function, the control unit 3 controls the operation of the camera 1 based on the content of the control information Sc1 from the processing unit 6, and controls the posture (up / down / left / right angle) to convert the image information into Sv. Switch etc.

  Further, the distribution / combination unit 5 outputs the image information Sv output from the camera 1 to the processing unit 6 or the display unit 2 based on the content of the control information Sc2 from the processing unit 6. In addition to this, the distribution / synthesis unit 5 superimposes (synthesizes) later-described synthesis image information Smx output from the processing unit 6 on the image information Sv from the camera 1 and displays it as display information Sdp. Output to part 2.

  Here, the distribution mode of the image information Sv output from the camera 1 to the processing unit 6 and the display unit 2 and the output mode of the synthesis image information Smx output from the processing unit 6 to the display unit 2 Can be arbitrarily set in the distribution / synthesis unit 5 based on the control information Sc2 from the processing unit 6. More specifically, the output of the image information Sv from the camera 1 to the display unit 2 is completely prohibited, and only the image corresponding to the image information Smx for synthesis from the processing unit 6 is substituted for the image information Sv. The distribution / combination unit 5 can also switch the display information Sdp including the display information 2 to the display unit 2. On the contrary, the distribution / combination can output only the image information Sv from the camera 1 to the display unit 2. Switching in the unit 5 is also possible.

  On the other hand, the display unit 2 is more specifically an image display unit including a liquid crystal display, an HMD (Head Mounted Display), an optical projector, a CRT (Cathode Ray Tube), a half mirror screen, or the like. The image included in at least one of the image information Sv and the display information Sdp output from 5 is displayed so that the viewer can see the image. The display unit 2 may be configured as the display unit 2 by modifying the eyepiece unit in the above-described conventional telescope, binoculars, and the like, and installing each display described above.

  On the other hand, the input unit 4 generates operation information Sin corresponding to the operation performed by the observer performed in the input unit 4 and outputs the operation information Sin to the processing unit 6. Specific examples of the operation include an operation mode designation operation to be described later as the image processing apparatus S, and a designation operation for features in the image currently captured by the camera 1.

  As a result, the processing unit 6 generates the control information Sc1 and Sc2 and outputs the control information Sc1 and Sc2 to the control unit 3 and the distribution / combination unit 5 so that the contents of the operation information Sin are realized, and supervises the entire image processing apparatus S. Control.

  More specifically, the processing unit 6 captures the current camera 1 based on the image information Sv output via the distribution / combination unit 5 and the state information Spos output from the control unit 3. Analyzing the contents of the scenery (such as the shape and size of the features contained in the scene, and so on), and processing the contents of the image information Sv based on the analysis result and the operation signal Sin. The combined image information Smx is output to the distribution / synthesis unit 5.

  In addition, the processing unit 6 generates the control information Sc1 to control the attitude of the camera 1 via the control unit 3 and outputs the control information Sc1 to the control unit 3. Note that the display mode, the screen angle, and the like on the display unit 2 can be controlled by control information (not shown) from the processing unit 6.

  Next, a specific configuration and detailed operation of the processing unit 6 that mainly performs the operation of the present embodiment will be described with reference to FIG.

  As shown in FIG. 2, the processing unit 6 includes a recognition unit 7 as a viewpoint detection unit and an extraction unit, a processing unit 8 as a reading unit, a storage unit 9 as a storage unit, a generation unit 10, and an output unit. 11 and a processing control unit 12. The image information Sv from the distribution / synthesis unit 5 and the status information Spos from the control unit 3 are input to the recognition unit 7, and the operation information Sin from the input unit 4 is input to the processing control unit 12. Further, the image information Smx for synthesis is output from the output unit 11, and the control information Sc1 and Sc2 are output from the processing control unit 12.

  In this configuration, the recognition unit 7 outputs the image information Sv output from the distribution / combination unit 5 to the processing unit 8 based on the control information Sc11 from the processing control unit 12, and the image as the image information Sv. The contents such as the scenery corresponding to are analyzed, recognition processing for recognizing the contents is performed, recognition information Srg including the results is generated, and output to the output unit 11, the processing unit 8, and the generation unit 10, respectively. Here, as the recognition process, specifically, the features included in the image information Sv as the contents (hereinafter referred to as “object”) are separated from each other, extracted, and identified. Processing, spatial coordinate specifying processing, holding processing, and the like, which are realized by comparing the content of the input image information Sv and the later-described related information Sm stored in the storage unit 9. . At this time, by extracting the observer's viewpoint within the visual field range of the camera 1 based on the state information Spos from the control unit 3, the recognition process is speeded up / accurate. Note that the recognition processing can be performed specifically using a so-called logic circuit, or can be performed using artificial intelligence or the like.

  On the other hand, the processing unit 8 processes the image information Sv received through the recognition unit 7 based on the control information Sc10 from the processing control unit 12, the related information Sm from the storage unit 9, and the recognition information Srg from the recognition unit 7. As a result, various visual and auditory effects as described later are added and output to the output unit 11 as processing information Smd.

  Furthermore, the generation unit 10 is recognized using the state information Spos from the control unit 3 based on the control information Sc13 from the processing control unit 12, the related information Sm from the storage unit 9, and the recognition information Srg from the recognition unit 7. A generation process for generating a new image corresponding to an object existing in a region including the observer's viewpoint is performed, and generation information So obtained as a result is output to the output unit 11. Specifically, the generation processing includes, for example, drawing processing of the outline of the object, drawing processing of text information corresponding to the object, generation processing of commentary video or commentary sound indicating the object, or processing of the object. There is a related 3D image drawing process.

  In addition, the storage unit 9 includes all (all omnidirectional images) that are included in the image information Sv and may be output from the camera 1 such as a three-dimensional shape, a moving image, a still image, text (character), voice, or motion. ) A storage device that accumulates an image and information for explaining the image as related information Sm, and outputs the related information Sm to the recognition unit 7, the processing unit 8, or the generation unit 10 as necessary. To do. Here, the storage unit 9 generally stores in advance an image (object) itself included in the image information Sv expected to be input from the camera 1 and related information Sm related to the image. As a matter of course, it is connected via a network to a data center or the like in which various information is recorded, and various related information Sv corresponding to the image processing apparatus S is collected from the data center or the like and stored in the storage unit 9. It can also be configured to keep.

  Further, the output unit 11 receives the recognition information Srg output from the recognition unit 7, the processing information Smd output from the processing unit 8, and the generation information So output from the generation unit 10. Are combined with each other based on the control information Sc12 from 12, and output to the distribution / synthesis unit 5 as the image information for synthesis Smx.

  Finally, the process control unit 12 analyzes the contents of the operation information Sin, generates the control information Sc1, Sc2, Sc10, Sc11, Sc12, and Sc13, respectively, in order to realize the contents, and the corresponding control unit 3 , The distribution / combination unit 5, the recognition unit 7, the processing unit 8, the generation unit 10, the output unit 11, and the like.

  Next, the overall operation of the image processing apparatus S having the above-described configuration and operation will be specifically described with reference to each flowchart shown in FIG. 3A is a flowchart showing the overall operation, FIG. 3B is a flowchart showing details of the recognition process in the overall operation, and FIG. 3C is a display of the overall operation. It is a flowchart which shows an information generation process.

  First, the overall operation of the image processing apparatus S will be described with reference to FIG.

  As shown in FIG. 3A, in the image processing apparatus S having the above-described configuration, when the power is turned on and image information Sv including the object is transmitted from the camera 1 (step S1), the processing is performed. The content of the current operation mode designated in the input unit 4 is confirmed by the unit 6 (step S2).

  Here, the said operation mode shows the kind of operation | movement which should be performed as the image processing apparatus S designated by the observer in the input part 4, and in the case of embodiment, specifically, "explanation mode" , “Delete mode”, “super telephoto mode”, “highlight display mode”, “time change mode”, and “weather change mode”.

  At this time, the “explanation mode” is a mode in which the description of the object based on the character information is displayed superimposed on the image that includes the object and is displayed on the display unit 2. In the “deletion mode”, an object in front is deleted in the field of view of the camera 1 and another object (an object that is normally hidden behind the object in front of the object) is displayed. Mode. Further, the “super-telephoto mode” is to display an object that exists farther than the horizontal line within the field of view of the camera 1 (that is, an object that is actually hidden behind the horizontal line and cannot be seen) on the horizontal line. Mode. Further, the “highlight display mode” is a mode in which an object that is normally dark and invisible is emphasized and displayed, particularly when a night view is observed. In addition, the “time change mode” refers to the current scenery in the field of view of the camera 1, the scenery in the future (future) or previous (past) for the time specified by the input unit 4. In this mode, the scene is displayed on the display unit 2 instead of the scenery. Lastly, the “weather change mode” refers to the scenery in the field of view of the camera 1 that is displayed in the display unit 2 in place of the current scenery etc. If it is fine weather, the same range of scenery, etc. is displayed. If the weather is currently fine, for example, the same range of scenery when foggy is displayed is displayed.

  In addition, in order to realize the above modes, character information indicating the contents (names, features, etc.) of objects included in the landscapes, etc. for landscapes that can be recognized through the camera 1 (execution of the description mode described above) ), Image information for deletion corresponding to an image viewed from the position of the camera 1 alone for each object included in the scenery, etc. (for execution of the deletion mode), an object hidden in a horizontal line in the scenery, etc. Super telephoto image information corresponding to an image of an object that can be viewed for the first time by extending the line of sight of the camera 1 in the direction of the line of sight (for performing the super telephoto mode), and highlighting of each object included in each scene Image information for time change (for executing the above highlight mode), for example, image information for time change corresponding to images of each scenery (images within the field of view of the camera 1) for each hour For changing the weather mode) and, for example, image information for weather change corresponding to images (images within the field of view of the camera 1) of the scenery, etc. during clear weather, cloudy weather, rain, fog and snow The weather change mode execution) is stored and stored in the storage unit 9 as related information Sm.

  Further, when the image processing apparatus S is in operation, the content of the object designated by the input unit 4 is detected from the operation information Sin.

  When the content of the operation or the like in the input unit 4 is confirmed, the processing unit 6 recognizes the content of the object in the image information Sv based on the operation information Sin corresponding to the operation (step S3). ). Details of this recognition processing will be described later with reference to a flowchart shown in FIG.

  Next, based on the content of the recognized image information Sv, the processing unit 6 similarly generates the synthesis image information Smx and outputs it to the distribution / synthesis unit 5 (step S4). The generation process as step S4 will be described later with reference to the flowchart shown in FIG.

  Then, the distribution / combination unit 5 executes a combining process of the output image information Smx for synthesis and the image information Sv according to the contents of the operation information Sin, and displays an image to be displayed on the display unit 2. Display information Sdp is generated (step S5). The composition process includes a process of outputting the image information Sv as it is to the display unit 2.

  Thereafter, the generated image is displayed on the display unit 2 (step S6), and then it is confirmed whether or not an operation for ending the operation as the image processing apparatus S has been performed on the input unit 4 (step S7). When the end operation is not performed (step S7; NO), the process returns to the operation of step S1 and the above-described processing is continued. When the end operation is performed (step S7; YES), the operation is continued. The processing as the image processing apparatus S ends. It should be noted that the processing as step S6 includes outputting audio corresponding to the image to be displayed.

  Next, details of the recognition process (step S3) executed in the recognition unit 7 will be described with reference to FIG.

  As shown in FIG. 3B, in the recognition process (step S3), when the image information Sv is input to the recognition unit 7 (step S10), the next state information Spos is input similarly. Then, the posture of the camera 1 is detected, and the position of the observer's current viewpoint is detected based on the detected posture and the analysis result of the contents of the image input as the image information Sv (step S11). Here, specifically, the position of the current viewpoint can be detected by the collation / comparison with an image stored in advance in the storage unit 9 as the processing of step S11.

  When the position of the viewpoint is detected, an object at that position is extracted, and it is confirmed what object is included in the current image information Sv and the observer is currently paying attention (step S12). ).

  Next, it is searched and confirmed whether or not related information related to the object extracted and confirmed in step S12 is stored in the storage unit 9 (steps S13 and S14). And when the relevant information relevant to the said object does not exist in the memory | storage part 9 (step S14; NO), it transfers to step S17 mentioned later as it is, On the other hand, the relevant information relevant to the said object is stored in the memory | storage part 9. Is present (step S14; YES), the related information Sm related to the object confirmed in the storage unit 9 is read from the storage unit 9 (step S15). It is temporarily stored in a memory (not shown).

  Next, based on the operation mode detected in step S2 in FIG. 3A, if necessary, the display unit 2 displays a three-dimensional shape indicating the extracted and confirmed (step S12) object. The three-dimensional information is also read from the storage unit 9 (step S16), and is also temporarily stored in a memory (not shown) in the recognition unit 7. Here, the read-out three-dimensional information is used, for example, for executing the highlight mode or the like.

  Thereafter, it is confirmed whether or not the recognition process for all objects included in the input image information Sv has been completed (step S17). And when not complete | finished (step S17; NO), it returns to said step S14, and when the series of processes mentioned above are performed with respect to the following object, on the other hand, the recognition process about all the objects is complete | finished (Step S17; YES), each information stored in a memory (not shown) in the recognition unit 7 is output as the recognition information Srg to the recognition unit 7, the processing unit 8, and the generation unit 10 (step S18). The process proceeds to step S4 shown in FIG.

  Next, details of the generation process (step S4) executed mainly by the recognition unit 7, the processing unit 8, and the generation unit 10 will be described with reference to FIG.

  As shown in FIG. 3C, in the generation process (step S4), first, the contents of the operation of the observer detected in the process of step S2 of FIG. 3A are confirmed (step S20). At this time, the content of the operation is an instruction as to which information about which object in the visual field range currently captured by the camera 1 is to be displayed on the display unit 2. It is given by a coordinate value etc. on a typical screen.

  Then, based on the content of the instruction, it is detected which object the observer is indicating (step S21), and whether the related information related to the detected object is stored in the storage unit 9 or not. Search and confirm (steps S22 and S23). And when the relevant information relevant to the said object does not exist in the memory | storage part 9 (step S23; NO), it transfers to step S27 mentioned later as it is, On the other hand, the relevant information Sm relevant to the said object is the memory | storage part 9. If it is present (step S23; YES), then the related information Sm related to the object confirmed in the storage unit 9 (specifically, the moving image and landscape related to the object) Are read out from the storage unit 9 (step S24), and are temporarily stored in a memory (not shown) in the recognition unit 7.

  Next, the recognition unit 7 specifies the position where the stored information (step S24) is to be drawn as the coordinate position on the display screen in the display unit 2 (step S25), and the position is determined at the specified position. An image corresponding to the stored information is drawn, the drawing result is output to the output unit 11 (step S27), and the process proceeds to step S5 shown in FIG.

  Next, specific examples of results obtained by executing the above-described processes in the image processing apparatus S will be described with reference to the respective operation modes.

(I) Specific Example of Explanation Mode First, the explanation mode will be described in detail with reference to FIG. The explanation mode is a mode in which the explanation of the object based on the character information is superimposed and displayed in the image including the object and displayed on the display unit 2 as described above. In executing the explanation mode, the character information described above is read from the storage unit 9 as the related information Sm.

  That is, in the explanation mode, for example, when the image displayed on the display unit 2 is a scenery G1 as shown in FIG. 4A, the observer wants to know the details in the scenery G1. If an input object 4 is specified in the input unit 4 (see step S2 in FIG. 3A), the character information related to the specified object is related information. The character image corresponding to the read character information is read out as Sm from the storage unit 9 (see step S24 in FIG. 3B), for example, as character images T1 and T2 illustrated in FIG. 4B. It is displayed superimposed on the scene G1 in a manner associated with the corresponding object (see steps S25 to S27 in FIG. 3 (c) and steps S5 and S6 in FIG. 3 (a)). . Further, when an operation for requesting a more detailed explanation about any object is performed in the input unit 4 (see step S2 in FIG. 3A), more detailed character information related to the designated object. Is read from the storage unit 9 as the related information Sm (see step S24 in FIG. 3B), and the character image corresponding to the read detailed character information is, for example, the character image T2d illustrated in FIG. As shown in FIG. 3C, the image is superimposed and displayed in the scene G1 in a manner associated with the corresponding object (see steps S25 to S27 in FIG. 3C and steps S5 and S6 in FIG. 3A).

(II) Specific Example of Delete Mode Next, the delete mode will be specifically described with reference to FIG. In the deletion mode, as described above, the object in the foreground within the field of view of the camera 1 is deleted and another object beyond the object (usually an object that is hidden behind the object in front and cannot see part or all of it) ) Is displayed. In executing the deletion mode, the deletion image information corresponding to the image viewed from the position of the camera 1 alone for each object included in the omnidirectional scenery that can be recognized through the camera 1 is provided. The related information Sm is read from the storage unit 9.

  In other words, in the deletion mode, for example, when the image displayed on the display unit 2 is a landscape G2 as shown in FIG. 5A, the entire view of the object OBb behind it in the landscape G2 is displayed. Is specified in the input unit 4 by surrounding an object OBa (a series of buildings in front of the highest building in the case of FIG. 5) with a cursor W as shown in FIG. 5A. Then (see step S2 in FIG. 3A), the designated object OBa is deleted from the image corresponding to the scenery G2 (see steps S25 and S26 in FIG. 3B), and the portion where the object OBa is deleted. In addition, the image information for deletion corresponding to the entire view of the object OBb is displayed in the scenery G2 like the object OBb illustrated in FIG. 5B, for example. (See steps S25 to S27 in FIG. 3 (c) and steps S5 and S6 in FIG. 3 (a)).

(III) Specific Example of Super-Telephoto Mode Next, the super-telephoto mode will be described in detail with reference to FIG. The super-telephoto mode is farther than the horizontal line in the field of view of the camera 1 as described above (if there is no sea in the field of view, for example, a mountain ridge line in the field of view). In this mode, a real object is displayed on the horizontal line. In executing the super-telephoto mode, the super-telephoto image information corresponding to the image viewed from the position of the camera 1 of the object hidden behind the horizon in the omnidirectional scenery that can be recognized through the camera 1 described above. Is read from the storage unit 9 as the related information Sm.

  That is, in the super-telephoto mode, for example, when the image displayed on the display unit 2 is a landscape G3 as shown in FIG. 6A, an object further distant from the horizontal line H in the landscape G3 (for example, When display of an object corresponding to a building existing along the west coast of the United States is designated in the input unit 4 (see step S2 in FIG. 3A), an object that should be far from the designated horizontal line H is displayed. The corresponding super telephoto image information is read out from the storage unit 9 as the related information Sm, and the read super telephoto image information is in the scenery G3, for example, like an object OBc illustrated in FIG. 5B. (See steps S25 to S27 in FIG. 3 (c) and steps S5 and S6 in FIG. 3 (a)).

(IV) Specific Example of Highlight Display Mode Next, the highlight display mode will be described in detail with reference to FIG. The highlight mode is a mode for highlighting and displaying an object that is usually dark and invisible, especially when a night view is observed as described above. In executing the highlight mode, the highlight display image information of each object included in the omnidirectional landscape that can be recognized through the camera 1 is read from the storage unit 9 as the related information Sm.

  That is, in the highlight mode, for example, when the image displayed on the display unit 2 is a scenery G4 corresponding to a night view as shown in FIG. 7A, the roof of the rightmost building in the scenery G4. Is specified in the input unit 4 (see step S2 in FIG. 3 (a)), the above-mentioned for highlighting the designated tower is displayed. The highlighted image information is read out from the storage unit 9 as the related information Sm, and the object included in the read highlighted image information is, for example, the position where the steel tower illustrated in FIG. Is displayed as an emphasized image PP at a position in the scenery G4 corresponding to (see steps S25 to S27 in FIG. 3C and steps S5 and S6 in FIG. 3A).

(V) Specific Example of Time Change Mode Next, the time change mode will be described in detail with reference to FIGS. In the time change mode, as described above, the scenery in the field of view of the camera 1 is displayed in place of the current scenery etc. instead of the scenery in the future (future) or previous (past) for the specified time. This is a mode displayed on the part 2. In executing the time change mode, for example, an omnidirectional landscape that can be recognized through the camera 1 described above, for example, an image of each landscape (an image within the visual field range of the camera 1) every hour. Corresponding time change image information is read from the storage unit 9 as the related information Sm.

  That is, in the time change mode, for example, when the image displayed on the display unit 2 is a scenery G5 as shown in FIG. 8A, for example, a scene of a scene 10 years ago viewed from the same angle as the scenery G5. When display is designated in the input unit 4 (see step S2 in FIG. 3A), the time-changing image information corresponding to the designated scenery is read out from the storage unit 9 as related information Sm, and the read time The scenery G5B corresponding to the image information for change is displayed on the display unit 2 as illustrated in FIG. 8B (see steps S25 to S27 in FIG. 3C and steps S5 and S6 in FIG. 3A).

  On the other hand, when display of a landscape 10 years ahead, for example, viewed from the same angle as the landscape G5 shown in FIG. 8A is specified in the input unit 4 (see step S2 in FIG. 3A), The time change image information corresponding to the designated scenery is read out from the storage unit 9 as the related information Sm, and the view G5F corresponding to the read time change image information is illustrated in FIG. 8C. 2 (see steps S25 to S27 in FIG. 3C and steps S5 and S6 in FIG. 3A).

  In this case, for an object that does not exist in the current scenery G5 (see FIG. 8A), a character image T3 including its explanatory text is superimposed in the scenery G5F as illustrated in FIG. 8C. It is preferable to display them. Further, as shown in FIG. 9, the time change image information corresponding to the scenery G5FF (which is of course an imaginary view) showing a future state further than the time illustrated in FIG. It is also possible to read out from the storage unit 9 and display it on the display unit 2.

(VI) Specific Example of Weather Change Mode Finally, the weather change mode will be specifically described with reference to FIG. The weather change mode is a mode in which, for example, the scenery in the field of view of the camera 1 as described above is displayed on the display unit 2 in place of the current scenery. When the weather change mode is executed, the weather change image information corresponding to the images of the scenery in various weather conditions is associated with the related information for the scenery in all directions that can be recognized through the camera 1 described above. It is read from the storage unit 9 as Sm.

  That is, in the weather change mode, for example, when the image displayed on the display unit 2 is a scenery G6 as shown in FIG. 10, the scenery is in the same field of view as viewed from the same angle as the scenery G6. When it is designated in the input unit 4 that the scenery at the time of other weather different from the current one is displayed (see step S2 in FIG. 3 (a)), the viewing range of the scenery G6 at the designated weather is displayed. The weather change image information for reading is read out from the storage unit 9 as the related information Sm, and the read weather change image information is the same field of view when the fog illustrated in the lower part of FIG. Is displayed on the display unit 2 like the scenery G6W (see steps S25 to S27 in FIG. 3C and steps S5 and S6 in FIG. 3A).

  In the case shown in FIG. 10, for example, when the actual weather is foggy weather as shown in the lower part of FIG. 10, an image corresponding to the corresponding sunny scenery G6 is displayed on the display unit 2. Of course, it is possible.

  As described above, according to the operation of the image processing apparatus S according to the embodiment, the composition image information Smx corresponding to the image within the visual field range of the camera 1 is generated based on the related information Sm, and the composition information Either display information Sdp including only image information Smx or display information Sdp in which image information Smx for synthesis and image information Sv corresponding to an image in the visual field range are superimposed is output to an observer. Therefore, not only a monotonous image within the visual field range but also various information corresponding to the image can be provided to the observer.

  Therefore, an image rich in added value can be provided to the observer.

  Further, since the viewpoint of the observer is detected from the posture of the camera 1, and the image information Smx for synthesis is generated from the related information Sm corresponding to the object on the viewpoint and is output to the observer, the current state of the observer It is possible to output information related to an object at the viewpoint (that is, an object in which the observer is interested) to the observer with good timing.

  Furthermore, the character information corresponding to the object in the visual field range is stored as the related information Sm, and this is superimposed on the position of the corresponding object as the composite image information Smx. An observer can easily obtain it.

  Furthermore, the deletion image information corresponding to the object that cannot be visually recognized within the normal visual field range is stored as the related information Sm, and the image corresponding to the other object concealing the object is replaced with the deletion image. Since the compositing image information Smx is generated, an image of an object that cannot be visually recognized can be provided to the observer.

  In addition, the image in the visual field range at a time different from the current time is stored as the related information Sm, and the image information for synthesis Smx is generated using the related information Sm. The state can be easily provided to the observer.

  Furthermore, since the super-telephoto image corresponding to an object that cannot be physically seen by being further away from the horizontal line or the ridgeline of the mountain is stored as the related information Sm, more interesting information is provided to the observer. Can do.

  In addition, since the composite image information Smx is generated using the highlight display image for displaying any object existing in the field of view with emphasis over the other objects, for example, it is cloudy, rainy, or nighttime. However, the object that the observer wants to see can be clearly distinguished from other objects and presented to the observer.

  In addition, sounds associated with objects within the current field of view can be provided to the observer, allowing the observer to make more interesting observations.

  Each program corresponding to the flowchart shown in FIG. 3 is recorded in an information recording medium such as a rewritable removable memory device such as a flexible disk or a solid-state semiconductor memory, or a hard disk, or is acquired via the Internet or the like. It is also possible to utilize the computer as the processing unit 6 according to the embodiment by reading out and executing these by a general-purpose computer.

  As described above, the present invention can be used in the field of observation devices for observing distant scenery and the like, and particularly remarkable effects when applied to the field of observation devices for observing scenery in scenic locations. Is obtained.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment. It is a block diagram which shows the detailed structure of the process part in the image processing apparatus of embodiment. It is a flowchart which shows the process performed in the image processing apparatus of embodiment, (a) is a flowchart which shows the whole of the said process, (b) is a flowchart which shows the detail of a recognition process among the said processes, ( c) is a flowchart showing details of display information generation processing in the processing. It is a figure which shows the specific example of description mode which concerns on embodiment, (a) is a figure which illustrates original scenery, (b) is a figure which shows the specific example (i) of description mode, (c) is It is a figure which shows the specific example (ii) of description mode. It is a figure which shows the specific example of the deletion mode which concerns on embodiment, (a) is a figure which illustrates an original scenery, (b) is a figure which shows the specific example of deletion mode. It is a figure which shows the specific example of the super telephoto mode which concerns on embodiment, (a) is a figure which illustrates an original scenery, (b) is a figure which shows the specific example of a super telephoto mode. It is a figure which shows the specific example of the highlight display mode which concerns on embodiment, (a) is a figure which illustrates an original scenery, (b) is a figure which shows the specific example of highlight display mode. It is a figure which shows the specific example of the time change mode which concerns on embodiment, (a) is a figure which illustrates an original scenery, (b) is a figure which shows the specific example (i) of time change mode, (c ) Is a diagram showing a specific example (ii) of the time change mode. It is a figure which shows the specific example (iii) of the time change mode which concerns on embodiment. It is a figure which shows the specific example of the weather change mode which concerns on embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS S Image processing apparatus 1 Camera 2 Display part 3 Control part 4 Input part 5 Distribution / combination part 6 Processing part 7 Recognition part 8 Processing part 9 Storage part 10 Generation part 11 Output part 12 Processing control part G1, G2, G3, G4, G5, G5B, G5F, G5FF, G6, G6W Scenery T1, T2, T2d, T3 Character image OBa, OBb, OBc Object W Cursor H Horizontal line PP Enhanced image Sv Image information Spos Status information Sc1, Sc2, Sc10, Sc11, Sc12, Sc13 Control information Smx Composition image information Sdp Display information Sin Operation information Srg Recognition information Sm Related information Smd Processing information So Generation information

Claims (11)

An acquisition means for optically acquiring image information corresponding to a visual field image corresponding to a visual field range of the observer operating the observation device for observing the scenery;
Storage means for storing relevant information related to objects present within the field of view;
Generation means for generating correspondence information corresponding to the object and different from the original image information based on the acquired image information and the stored related information;
Output means for outputting only one of the generated correspondence information or superimposition information obtained by superimposing the generated correspondence information on the acquired image information, to the observer;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The generating means includes
Posture detecting means for detecting the posture of the observation device;
Viewpoint detection means for detecting a position of the viewpoint of the observer within the visual field range based on the detected posture;
Extracting means for extracting the object of interest in the visual field range based on the position of the detected viewpoint;
Reading means for reading out the related information corresponding to the extracted object from the storage means;
With
An image processing apparatus that generates the correspondence information based on the read related information and the acquired image information. The image processing apparatus according to claim 1 or 2,
The storage means stores character information corresponding to the object within the visual field range as the related information,
The image processing apparatus, wherein the generation unit generates the correspondence information by superimposing the character information on a position in the image information where the object corresponding to the character information exists. The image processing apparatus according to any one of claims 1 to 3,
The storage means stores concealed object image information corresponding to an image of the object that is concealed by another object within the visual field range as the related information,
The generating means replaces the concealed object image corresponding to the other object concealing the object corresponding to the concealed object image within the visual field range with the concealed object image, and the correspondence information is obtained. An image processing apparatus that generates the image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The storage means stores other time image information corresponding to the visual field image at a time different from the current time as the related information,
The image processing apparatus, wherein the generating unit generates the other time image corresponding to the same visual field range as the correspondence information instead of the image information at a current time. In the image processing device according to any one of claims 1 to 5,
The storage means stores, as the related information, a far object image corresponding to the visual field range and corresponding to an object far from the horizontal line as viewed from the observer.
The image processing apparatus, wherein the output means superimposes and outputs the far object image corresponding to the image information on the current image information. The image processing apparatus according to any one of claims 1 to 6,
The image processing apparatus, wherein the generation unit generates the correspondence information for displaying any of the objects existing in the visual field range with emphasis over the other objects. In the image processing device according to any one of claims 1 to 7,
The storage means stores sound information for outputting a sound corresponding to the object in the visual field range as the related information,
The image processing apparatus, wherein the output unit includes a sound output unit that outputs the sound corresponding to any one of the currently output using the sound information corresponding to the sound. An acquisition step of optically acquiring image information corresponding to a visual field image corresponding to the visual field range of the observer operating the observation device for observing the scenery;
A generating step of generating correspondence information corresponding to the object and different from the original image information based on the related information related to the object existing in the visual field range and the acquired image information;
An output step of outputting only one of the generated correspondence information or superimposition information obtained by superimposing the generated correspondence information on the acquired image information, to the observer;
An image processing method comprising: A computer included in an image processing apparatus including an observation apparatus for observing a landscape;
An acquisition means for optically acquiring image information corresponding to a visual field image corresponding to a visual field range in the observation device of an observer who operates the observation device;
Storage means for storing relevant information relating to objects present within the field of view;
Generating means for generating correspondence information corresponding to the object and different from the original image information based on the acquired image information and the stored related information; and
Output means for outputting only one of the generated correspondence information or superimposition information obtained by superimposing the generated correspondence information on the acquired image information, to the observer;
An image processing program that is made to function as:   11. An information recording medium in which the image processing program according to claim 10 is recorded so as to be readable by the computer.

Images