WO2016157923A1

WO2016157923A1 - Information processing device and information processing method

Info

Publication number: WO2016157923A1
Application number: PCT/JP2016/050293
Authority: WO
Inventors: 裕司安藤
Original assignee: ソニー株式会社
Priority date: 2015-03-30
Filing date: 2016-01-07
Publication date: 2016-10-06

Abstract

[Problem] To provide an information processing device and an information processing method capable of pseudo-expansion of an apparent field of view. [Solution] An information processing device that comprises an exposure control unit that controls an exposure process so that a captured image acquired via a second acquisition unit and processed on the basis of the relationship of angles of view of an object image acquired by a first acquisition unit and the second acquisition unit is exposed to an eye lens to which at least part of an object image acquired by the first acquisition unit has been exposed.

Description

Information processing apparatus and information processing method

The present disclosure relates to an information processing apparatus and an information processing method.

In optical system devices such as telescopes and binoculars, the subject may be magnified and projected at a high magnification. When the magnification is increased, the subject moves greatly even if the orientation is slightly changed, so that the subject is easily out of sight. Once the subject is off, the surroundings cannot be seen and it is difficult to capture the subject in the sight again. Although the above difficulty can be reduced by increasing the apparent viewing angle, it has been technically and costly difficult to widen the apparent viewing angle. Therefore, as another example of a technique for more easily reducing the above difficulty, a technique for projecting the surrounding state in addition to the state of the observation target has been developed.

For example, Patent Document 1 below discloses a technique for providing a peripheral camera that captures the periphery of an area captured by a main camera in an electron microscope, and synthesizing the video obtained by the main camera with the video obtained by the peripheral camera. Has been.

Further, in Patent Document 2 below, a part of an image projected on an effective imaging region of the image sensor is extracted as a first image, and the periphery of the first image is extracted as a second image and stored as a captured image. A technique for displaying a second image as a through image together with the first image is disclosed.

JP 2011-138096 A JP 2004-343363 A

However, with the technique disclosed in Patent Document 1, the image of the main camera to be synthesized is displayed relatively small as the magnification of the main camera increases. Further, in the technique disclosed in Patent Document 2, the main first image has become narrower in order to extract the second image.

Therefore, the present disclosure proposes a new and improved information processing apparatus and information processing method capable of pseudo-expanding the apparent viewing angle.

According to the present disclosure, the captured image obtained by the second obtaining unit, which has been subjected to the processing based on the relationship between the field angles of the subject images obtained by the first and second obtaining units, is the first image. An information processing apparatus is provided that includes an irradiation control unit that controls an irradiation process so as to irradiate an eyepiece that is irradiated with at least a part of a subject image obtained by an acquisition unit.

Further, according to the present disclosure, the captured image obtained by the second obtaining unit, which has been subjected to the processing based on the relationship between the field angles of the subject images obtained by the first and second obtaining units, There is provided an information processing method executed by a processor, including controlling an irradiation process to irradiate an eyepiece irradiated with at least a part of a subject image obtained by one acquisition unit.

According to the present disclosure, the first acquisition unit, the second acquisition unit, an eyepiece that irradiates at least a part of the subject image obtained by the first acquisition unit, Irradiation for controlling the irradiation process to irradiate the eyepiece with the captured image obtained by the second acquisition unit, which has been processed based on the relationship between the field angles of the subject images obtained by the second acquisition unit. And an information processing apparatus including the control unit.

As described above, according to the present disclosure, the apparent viewing angle can be pseudo-expanded. Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure for demonstrating the outline | summary of the optical system apparatus which concerns on this embodiment. It is a figure which shows an example of the image | video seen from the eyepiece lens of an optical system apparatus. It is a figure which shows an example of the image | video seen from the eyepiece lens of an optical system apparatus. It is a figure for demonstrating a human viewing angle. It is a figure which shows an example of the hardware constitutions of the binoculars concerning this embodiment. It is a block diagram which shows an example of a logical structure of the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a figure for demonstrating an example of the image process by the irradiation control part which concerns on this embodiment. It is a flowchart which shows an example of the flow of the process performed in the binoculars which concern on this embodiment. It is a figure which shows an example of the hardware constitutions of the binoculars which concern on this modification. It is a figure for demonstrating an example of the irradiation process by the irradiation control part which concerns on this modification. It is a figure which shows an example of the hardware constitutions of the binoculars which concern on this modification. It is a figure which shows an example of the hardware constitutions of HMD which concerns on this modification. It is a figure which shows an example of the image | video displayed by HMD by this modification.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Overview 2. Configuration example 2.1. Overall configuration 2.2. 2. Configuration of image processing apparatus Technical features 3.1. Apparent viewing angle expansion function 3.2. Operation mode switching function 3.3. Comparison function Example of operation processing Modification 5.1. Sub optical system-less 5.2. Electronic binoculars 5.3. HMD
6). Summary

<< 1. Overview >>
First, an overview of an information processing apparatus according to an embodiment of the present disclosure will be described with reference to FIGS. Hereinafter, as an example, an example in which the information processing apparatus according to the present embodiment is realized as an optical system apparatus will be described.

FIG. 1 is a diagram for explaining an outline of an optical system apparatus according to the present embodiment. In the present specification, description will be made on the assumption that an area 10 in the real space as shown in FIG.

2 and 3 are diagrams showing an example of an image seen from the eyepiece of the optical system device. In FIG. 2, the image | video of the area | region 10 seen from the eyepiece of a common telescope or binoculars is shown. The range where the image can be seen shows the apparent viewing angle, and the outside is dark (black). In FIG. 3, the image | video of the area | region 10 seen from the finder of a general camera is shown. The visible range of the image shown in FIG. 3 substantially matches the actual range of the photograph. The apparent viewing angle is a viewing angle that represents the spread of an image that can be seen when looking through an eyepiece. In this connection, the viewing angle representing the range of the object included in the video is also referred to as an actual viewing angle. The actual viewing angle has the same meaning as the angle of view indicating the imaging range in the camera.

FIG. 4 is a diagram for explaining a human viewing angle. A range indicated by a symbol A in FIG. 4 indicates a discrimination visual field. The discrimination visual field is a range in which highly accurate information can be received. A range indicated by reference sign B indicates an effective visual field. The effective visual field is a range in which information can be received instantaneously by accompanying eye movement. A range indicated by a symbol C indicates a guidance visual field. The guidance visual field is a range in which the presence of information can be determined. A range indicated by reference sign D indicates an auxiliary visual field. The auxiliary visual field is an auxiliary range for inducing a gaze operation with respect to a strong stimulus. The discrimination visual field and the effective visual field can also be referred to as a central visual field. In addition, the guidance field and the auxiliary field may be referred to as a peripheral field.

Here, in an optical system device such as a telescope and binoculars, if the magnification is increased, the subject will move greatly even if the direction changes slightly, so it will be easily deviated from the field of view. It becomes difficult. If the apparent viewing angle is increased so that a wide range of images for the peripheral vision can be projected, the above difficulties can be reduced. For example, in the example shown in FIGS. 2 and 3, the subject is less likely to be out of the field of view as the apparent viewing angle is widened, and even when the subject has deviated, the subject is easily captured in the field of view again. In addition, if the body is moving while looking into the eyepiece, it causes intoxication and discomfort. Such difficulty becomes more prominent as the magnification increases. Regarding the example shown in FIG. 3, it is not a good idea to widen the view angle of the viewfinder unnecessarily, considering the purpose of confirming every corner of the photographed area and the size of the effective human field of view. I can say that.

However, it may be technically and costly difficult to widen the apparent viewing angle. The technique described in the above-mentioned patent document is an appropriate technique for an apparatus intended to directly view a high-definition image with the naked eye, considering that the main image is relatively small. It can not be said.

Accordingly, the information processing apparatus according to an embodiment of the present disclosure has been created with the above circumstances taken into consideration. The information processing apparatus according to the present embodiment can artificially expand the apparent viewing angle. The information processing apparatus according to this embodiment will be described in detail below with reference to FIGS.

<< 2. Configuration example >>
Below, the structural example of the information processing apparatus which concerns on this embodiment is demonstrated. Here, as an example of the information processing apparatus, binoculars that are optical system apparatuses will be described.

<2.1. Overall configuration>
FIG. 5 is a diagram illustrating an example of a hardware configuration of the binoculars 1 according to the present embodiment. As shown in FIG. 5, the binoculars 1 include a secondary optical system camera block 100, an image processing device 200, a main optical system camera block 300, a display block 400, a beam splitter 500, an objective lens 600, a prism 700, an eyepiece lens 800, and a switch. 900. The binoculars 1 includes a pair of left and right main optical system camera blocks 300, a display block 400, a beam splitter 500, an objective lens 600, a prism 700, and an eyepiece lens 800. In FIG. 5, only one of them is shown and the other is omitted. Further, although the lens 110, the lens 310, and the lens 410 are illustrated as single lenses, they may be designed as an appropriate optical system suitable for the application.

The light incident through the objective lens 600 is applied to the beam splitter 500. The beam splitter 500 has a function of transmitting a part and reflecting the other. For example, most of the incident light travels straight (transmits), and a part is reflected toward the main optical system camera block 300. To do. The light incident on the main optical system camera block 300 passes through the lens 310 and forms an image on the imager 320 and is imaged by the main optical system camera block 300. The captured video signal (captured image) is input to the image processing apparatus 200. Hereinafter, an optical system such as the main optical system camera block 300 related to the light incident through the objective lens 600 is also referred to as a main optical system. On the other hand, for example, an optical system such as the secondary optical system camera block 100 relating to light incident through the lens 110 is also referred to as a secondary optical system.

The light incident through the lens 110 forms an image on the imager 120 and is picked up by the sub optical system camera block 100. The sub optical system camera block 100 is adjusted so that the optical axis is positioned between the left and right main optical systems, and can capture an image in the same range as the image captured by the main optical system in the center. Here, it is assumed that the secondary optical system camera block 100 can capture an image having a wider angle of view than the main optical system. The captured video signal (captured image) is input to the image processing apparatus 200.

The image processing device 200 functions as an arithmetic processing device and a control device, and controls the overall operation in the binoculars 1 according to various programs. The image processing apparatus 200 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example. The image processing apparatus 200 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

Specifically, the image processing apparatus 200 functions as an information processing apparatus that performs image processing on an input video signal. For example, the image processing apparatus 200 performs image processing based on the video signal input from the main optical system camera block 300 on the video signal input from the sub optical system camera block 100 and displays the processed video signal on the display block 400. Output to.

The display block 400 displays the input video signal on an LCD (liquid crystal display) 420. The image displayed on the LCD 420 is adjusted by the lens 410 and irradiated on the back side of the beam splitter 500. The image irradiated on the back side of the beam splitter 500 is reflected by the beam splitter 500 and enters the user's eye through the prism 700 and the eyepiece lens 800 together with the light transmitted through the beam splitter 500. In other words, the eyepiece 800 that is irradiated with at least a part of the subject image obtained by the main optical system (first acquisition unit) is processed by the image processing apparatus 200 obtained by the secondary optical system (second acquisition unit). The captured image that has been subjected to is irradiated. Thus, an image obtained by combining the image of the main optical system and the image of the sub optical system is displayed on the eyepiece 800.

The hardware configuration of the binoculars 1 has been described above. Next, an internal configuration of the image processing apparatus 200 will be described with reference to FIG.

<2.2. Configuration of Image Processing Device>
FIG. 6 is a block diagram illustrating an example of a logical configuration of the image processing apparatus according to the present embodiment. As shown in FIG. 6, the image processing apparatus 200 includes an irradiation control unit 210 and an operation mode control unit 220, and is connected to the main optical system camera block 300, the sub optical system camera block 100, the display block 400, and the storage unit 230. The

The irradiation control unit 210 has a function of performing image processing based on video signals input from the sub optical system camera block 100 and the main optical system camera block 300. For example, the irradiation control unit 210 performs image processing on the captured image input from the sub optical system camera block 100 based on the captured image input from the main optical system camera block 300. Then, the irradiation control unit 210 outputs the captured image after image processing to the display block 400, and controls the irradiation process in which the display block 400 irradiates the eyepiece 800 with the captured image.

The operation mode control unit 220 has a function of controlling the operation mode of the irradiation control unit 210.

The storage unit 230 is a part that records and reproduces data with respect to a predetermined recording medium. For example, the storage unit 230 stores captured images captured by the sub optical system camera block 100 and the main optical system camera block 300. The storage unit 230 may be built in the binoculars 1 or may be formed as a removable medium such as a memory card.

The example of the internal configuration of the image processing apparatus 200 according to the present embodiment has been described above. Next, technical features of the binoculars 1 according to this embodiment will be described.

<< 3. Technical features >>
<3.1. Apparent viewing angle expansion function>
The binoculars 1 (for example, the irradiation control unit 210) according to the present embodiment has a function of artificially expanding the apparent viewing angle of the main optical system.

For example, the irradiation control unit 210 irradiates the second region on the outer edge of the first region irradiated with the subject image obtained by the main optical system with the captured image obtained by the sub optical system. Thereby, the apparent viewing angle is expanded in a pseudo manner. At that time, the apparent viewing angle is effectively reduced by performing processing based on the relationship between the angle of view of the subject image obtained by the main optical system and the sub optical system on the captured image obtained by the sub optical system. Can be extended. Hereinafter, specific examples of processing based on the relationship between the angles of view will be described with reference to FIGS.

FIG. 7 is a diagram for explaining an example of image processing by the irradiation control unit 210 according to the present embodiment. FIG. 7 shows an example of an image captured on the eyepiece lens 800. The region denoted by reference numeral 21 is a first region in which an image obtained by the main optical system (in other words, light incident through the objective lens 600 and transmitted through the beam splitter 500) is reflected. . An area indicated by reference numeral 22 is a second area in which an image obtained by the sub optical system (in other words, light displayed by the display block 400 and reflected by the beam splitter 500) is reflected. The second region may correspond to a human peripheral vision. Note that the boundary line between the first area and the second area may be explicitly displayed or may not be displayed. Referring to FIG. 7, the apparent viewing angle of the original main optical system is the first region, whereas the second region is located at the outer edge of the first region, so that the apparent appearance angle of the main optical system is increased. The viewing angle is pseudo-expanded to the second region.

In the example shown in FIG. 7, the magnifications of the first area video and the second area video are the same. In other words, the captured image obtained by the sub optical system is processed so as to represent the subject image included in the captured image at the same magnification as the subject image irradiated on the first region. As a result, the subject image shown in the first area matches the subject image shown in the second area.

Furthermore, in the example shown in FIG. 7, the video of the first area and the video of the second area are seamlessly connected. In other words, the captured image obtained by the sub optical system is processed so that the angle of view at the boundary between the first area and the second area matches between the main optical system and the sub optical system. As a result, the subject image shown in the first area matches the subject image shown in the second area.

The process by the irradiation control unit 210 will be described in detail. First, the irradiation controller 210 enlarges / reduces the captured image obtained by the sub optical system so that the magnification of the sub optical system and the magnification of the main optical system coincide. Then, the irradiation control unit 210 generates an image (in other words, a masked image) in which a portion corresponding to the first region is blacked out from the enlarged / reduced captured image, and irradiates the eyepiece 800 with the image. Thereby, nothing is reflected by the beam splitter 500 in the masked portion, and the image of the main optical system that has passed through the beam splitter 500 appears to fit into the masked portion.

FIG. 8 is a diagram for explaining an example of image processing by the irradiation control unit 210 according to the present embodiment. FIG. 8 shows an example of an image captured on the eyepiece lens 800. An area indicated by reference numeral 21 indicates a first area, and an area indicated by reference numeral 22 indicates a second area. In the example shown in FIG. 8, the magnification of the image in the second area is lower than the image in the first area. In other words, the captured image obtained by the sub optical system is processed to represent the subject image included in the captured image at a lower magnification than the subject image irradiated on the first region. As a result, a video having a wider actual viewing angle (viewing angle) than that of the example shown in FIG. 7 is provided to the user.

Here, humans rely not only on the semicircular canal but also on visual information when grasping their posture. For this reason, when the body movement and the field of view are significantly different, there may be a problem such as causing a symptom similar to that of motion sickness or swaying. Therefore, it is desirable that the image shown in the second area has a magnification close to 1. In that case, since it is possible to project the same image as when the outside world is seen with the naked eye in the peripheral visual field, the sense of posture of the user can be stabilized, and sickness and discomfort can be prevented or reduced. it can.

FIG. 9 is a diagram for explaining an example of image processing by the irradiation control unit 210 according to the present embodiment. FIG. 9 shows an example of an image captured on the eyepiece 800. An area indicated by reference numeral 21 indicates a first area, and an area indicated by reference numeral 22 indicates a second area. In the example shown in FIG. 9, the magnification of the image in the second region gradually decreases from the center side toward the outside. In other words, the captured image obtained by the sub optical system is processed so that the angle of view increases non-uniformly from the boundary portion between the first region and the second region toward the outside. As a result, a video having a wider actual viewing angle (viewing angle) than that of the example shown in FIG. 7 is provided to the user. Furthermore, since the angle of view at the boundary between the first region and the second region is the same between the main optical system and the sub optical system, the user feels uncomfortable compared to the example shown in FIG. It is reduced. Here, an example of a method for increasing the angle of view will be described with reference to FIGS. 10 and 11.

10 and 11 are diagrams for explaining an example of image processing by the irradiation control unit 210 according to the present embodiment. 10 and 11 show an example of an image captured on the eyepiece 800. FIG. An area indicated by reference numeral 21 indicates a first area, and an area indicated by reference numeral 22 indicates a second area.

FIG. 10 shows an example in which the angle of view increases non-uniformly in the second region. For example, the irradiation control unit 210 increases the increasing rate of the angle of view of the sub optical system image as it approaches the outside of the image. Specifically, the irradiation control unit 210 makes the interval 24 until the angle of view increases from 30 degrees to 40 degrees shorter than the interval 23 until the angle of view increases from 20 degrees to 30 degrees. In addition, the irradiation controller 210 makes the interval 25 until the angle of view increases from 40 degrees to 50 degrees shorter than the interval 24 until the angle of view increases from 30 degrees to 40 degrees. In contrast to the example shown in FIG. 10, the irradiation control unit 210 may decrease the increasing rate of the angle of view of the image of the sub optical system as it approaches the outside of the image.

Such a geometrical change of the image allows the binoculars 1 to be seamlessly displayed outward from the image of the main optical system and compressed and presented within the apparent viewing angle. As a result, compared to a case where the apparent viewing angle is simply expanded, the subject is less likely to lose sight of being out of the viewing angle, and a subject that jumps out of the field of view can be quickly detected. Although the distortion increases and the image quality worsens toward the outside of the image, the image quality of the image of the main optical system is maintained, so that the user can comfortably view the subject.

FIG. 11 shows an example in which the change rate of the angle of view increases evenly. For example, the irradiation control unit 210 increases the angle of view of the image of the sub optical system evenly as it approaches the outside of the image. Specifically, the irradiation controller 210 makes the interval 26 until the angle of view increases from 20 degrees to 25 degrees and the interval 27 until the angle of view increases from 25 degrees to 30 degrees have the same length. The angle of view values shown in FIGS. 10 and 11 are examples.

Here, the magnification of the image in the second area may change as the magnification of the image in the first area changes. In other words, the captured image obtained by the sub optical system may be processed so that the relationship of magnification between the subject image irradiated on the first region and the subject image included in the captured image is maintained. For example, when the main optical system has a zoom function, the irradiation control unit 210 enlarges the image of the sub optical system when the image obtained by the main optical system is enlarged, and the image obtained by the main optical system. If is reduced, the image of the sub optical system is also reduced. As a result, the angle of view of the video in the second area automatically changes in accordance with the change in the angle of view of the video in the first area. Therefore, the consistency between the video in the first area and the video in the second area is maintained. In addition, when the lenses of the main optical system or the sub optical system are exchanged, the same processing may be performed so as to maintain the relationship of magnification.

If the zoom ratio of the main optical system is large, the image in the second area may become excessively rough if the image of the sub optical system is simply enlarged by image processing. Therefore, it is desirable that the sub optical system also has an optical zoom function. In the example shown in FIG. 9, if the difference between the angle of view of the main optical system (the angle of view of the boundary portion) and the angle of view of the sub-optical system (the angle of view of the most peripheral portion) is excessively large, the image is distorted. May become excessively large, difficult to see, and uncomfortable video may be provided to the user. Therefore, the irradiation control unit 210 may adjust the angle of view of the secondary optical system according to a user operation, and the secondary optical system by image processing so that the difference does not become excessively large based on the zoom ratio of the primary optical system. The angle of view may be automatically adjusted.

The irradiation control unit 210 may control the size of the first area and the second area. For example, the irradiation control unit 210 decreases the first area when the user is searching for the target subject, and increases the second area when the user is observing the target subject. You may make 2nd area | region small. Control of the size of the first region can be realized by a diaphragm (not shown) provided in the main optical system. Further, the control of the size of the second region can be realized by controlling the size of the masked region in the captured image obtained by the sub optical system.

<3.2. Operation mode switching function>
The binoculars 1 (for example, the operation mode control unit 220) according to the present embodiment has a function of switching an operation mode related to processing for artificially extending the apparent viewing angle of the main optical system.

For example, the operation mode control unit 220 switches the content of the irradiation process controlled by the irradiation control unit 210. Specifically, the operation mode control unit 220 may switch which of the processes shown in FIGS. 7 to 11 is executed. In addition, the operation mode control unit 220 may switch the sizes of the first area and the second area.

The operation mode control unit 220 may perform switching according to a user operation. For example, the operation mode control unit 220 can control the operation mode in accordance with the switching operation to the switch 900.

The operation mode control unit 220 may perform switching according to the user's state. For example, the binoculars 1 may include sensors such as an acceleration sensor and a gyro sensor, and the operation mode control unit 220 may control the operation mode according to the user's exercise state detected by the sensor. For example, when it is detected that the user continues to exercise in the operation mode in which the process shown in FIG. 7 or 9 is performed, the operation mode control unit 220 automatically performs the process shown in FIG. You may switch to the operation mode performed. Thereby, a user's sickness and discomfort can be prevented or relieved.

<3.3. Comparison function>
The binoculars 1 (for example, the irradiation control unit 210) according to the present embodiment has a function of providing an image for comparing the current and the past.

For example, the irradiation control unit 210 synthesizes an image captured by the main optical system or the sub optical system in the past, stored in the storage unit 230, with an image obtained by the main optical system. The video stored in the storage unit 230 may be a video captured by the binoculars 1 or a video captured by another device and input to the binoculars 1. Hereinafter, a specific description will be given with reference to FIGS. 12 and 13.

12 and 13 are diagrams for explaining an example of image processing by the irradiation control unit 210 according to the present embodiment. FIG. 12 shows an image obtained by the main optical system. FIG. 13 shows an image obtained by combining the past image 32 with the image 31 obtained by the main optical system. The irradiation control unit 210 may synthesize the past image on the entire surface of the image obtained by the main optical system, or may synthesize, for example, only half as shown in FIG.

The technical features of the binoculars 1 according to this embodiment have been described above. Next, with reference to FIG. 14, an example of operation processing of the binoculars 1 according to the present embodiment will be described.

<< 4. Example of operation processing >>
FIG. 14 is a flowchart illustrating an example of a flow of processing executed in the binoculars 1 according to the present embodiment.

As shown in FIG. 14, first, the binoculars 1 obtain a captured image by the sub optical system (step S102). More specifically, the image processing apparatus 200 acquires a captured image captured by the sub optical system camera block 100.

Next, the irradiation controller 210 enlarges / reduces the captured image (step S104). At this time, the irradiation control unit 210 enlarges / reduces the captured image according to the operation mode setting by the operation mode control unit 220. For example, the irradiation control unit 210 may enlarge or reduce the captured image so that the magnification of the sub optical system and the magnification of the main optical system match, or the magnification of the sub optical system is larger than the magnification of the main optical system. You may expand / contract so that it may become low, and you may expand / contract so that magnification may become low gradually toward the outer side.

Next, the irradiation controller 210 masks the central portion (in other words, the portion corresponding to the first region) of the captured image after the enlargement / reduction (step S106). At this time, the irradiation control unit 210 may control the size of the masked area depending on whether the user is searching for the target subject or the user is observing the target subject.

Then, the irradiation controller 210 outputs the masked mask image to the display block 400 and causes the display block 400 to irradiate the eyepiece 800 with the mask image (step S108).

The operation processing example of the binoculars 1 according to the present embodiment has been described above.

<< 5. Modification >>
<5.1. No secondary optical system>
The present technology can also be applied to an apparatus that does not have a secondary optical system. Hereinafter, with reference to FIG. 15 to FIG. 16, an application example of the present technology to an apparatus that does not have a sub optical system will be described.

FIG. 15 is a diagram illustrating an example of a hardware configuration of the binoculars 1-1 according to the present modification. The binoculars 1-1 shown in FIG. 15 has a configuration in which the sub optical system camera block 100 is omitted from the binoculars 1 shown in FIG. The binoculars 1-1 also includes a memory card 1000 (in other words, the storage unit 230 formed as a removable medium).

In the binoculars 1-1 shown in FIG. 15, the image processing apparatus 200 (for example, the irradiation control unit 210) stores the captured image captured by the main optical system camera block 300 in the memory card 1000. Then, the image processing apparatus 200 outputs the captured image read from the memory card 1000 to the display block 400, and controls the irradiation process in which the display block 400 irradiates the eyepiece 800 with the captured image. As a result, the past image of the main optical system captured by the main optical system camera block 300 is combined with the current image of the main optical system that has passed through the objective lens 600 and the beam splitter 500 and reached the eyepiece lens 800. . At that time, as shown in FIG. 16, the irradiation control unit 210 irradiates a past captured image on a region outside the current field of view of the main optical system according to the movement of the binoculars 1-1 detected by the sensor. May be.

FIG. 16 is a diagram for explaining an example of irradiation processing by the irradiation control unit 210 according to the present modification. FIG. 16 shows an example of an image captured on the eyepiece lens 800. An area denoted by reference numeral 41 is an area in which a current image obtained by the main optical system (in other words, light incident through the objective lens 600 and transmitted through the beam splitter 500) is reflected. The area indicated by reference numeral 42 is an area in which a past video (in other words, a captured image captured by the main optical system camera block 300) obtained by the main optical system is reflected. The upper diagram of FIG. 16 shows an example of an image when the binoculars 1-1 are stationary, and the lower diagram of FIG. 16 shows an example of an image when the binoculars 1-1 are moved to the left.

For example, when the binocular 1-1 is stationary, the irradiation control unit 210 does not irradiate the past main optical system with an image. Therefore, as shown in the upper diagram of FIG. 16, only the current image of the main optical system is displayed on the eyepiece 800. When the binoculars 1-1 are moved, the part that has been visible until now is out of the field of view. Therefore, the irradiation control unit 210 irradiates a captured image corresponding to the region acquired from the memory card 1000 to a region out of the field of view of the subject image obtained by the main optical system. For example, the irradiation control unit 210 irradiates the area 42 with a captured image of a part of the subject image in the area 41 illustrated in the upper diagram of FIG. 16 that is not included in the area 41 illustrated in the lower diagram. Therefore, when the binoculars 1-1 move to the left, the past video extends to the right (peripheral visual field). As described above, when the binoculars 1-1 move up, down, left, and right, the apparent field of view widens.

<5.2. Electronic binoculars>
In the above description, binoculars have been described as an example. However, the present technology can be applied to various apparatuses. For example, the present technology can be applied to various optical system devices such as a telescope, a field scope, a single-lens camera, a microscope, a binocular, a surveying instrument, and a gun sight. In the above example, the image of the main optical system is analog and the image of the sub optical system is once digitized. However, the image of the main optical system is also digitized. Also good. As an example, FIG. 17 shows a configuration example when the present technology is applied to electronic binoculars.

FIG. 17 is a diagram illustrating an example of a hardware configuration of the binoculars 1-2 according to the present modification. A binocular 1-2 shown in FIG. 17 includes a secondary optical system camera block 100, an image processing apparatus 200, a main optical system camera block 300, an electronic viewfinder 1100, and a switch 900.

The light incident through the objective lens 311 forms an image on the imager 320 and is imaged by the main optical system camera block 300. Further, the light incident through the lens 110 forms an image on the imager 120 and is imaged by the sub optical system camera block 100.

The image processing apparatus 200 synthesizes the captured images captured by the main optical system camera block 300 and the sub optical system camera block 100 and outputs them to the electronic viewfinder 1100. The electronic viewfinder 1100 displays the image synthesized by the image processing apparatus 200 on the LCD 420. The image displayed on the LCD 1120 passes through the eyepiece 1110 and enters the user's eyes.

The binoculars 1-2 according to this modification performs the same processing as the binoculars 1 described above. In addition, since the binoculars 1-2 according to this modification is data in which each of the images to be synthesized is digitized, the ratio of the field of view of the main optical system and the sub optical system (in other words, the first region and the first region) It is possible to more easily control the ratio of the size to the area 2).

Note that when the binoculars 1-2 can output an image to the outside, the imaging device and the display device may be formed separately. For example, an image may be output by an HMD (Head Mounted Display) instead of the electronic viewfinder 1100.

In addition, an endoscope is an example of a system in which an imaging device and a display device are separately formed. For example, a main camera that functions as a main optical system and a sub camera that functions as a sub optical system may be included in one endoscope. In this case, although the size of the endoscope is increased, it is possible to adjust the visual field and parallax of both the main camera and the sub camera at the manufacturing stage. In addition, images of a plurality of endoscopes having different angles of view and resolutions may be handled as images of the main optical system or images of the sub optical system, respectively. In that case, it is possible to keep the size of each endoscope small. However, the endoscope system detects the positional relationship of a plurality of endoscopes in real time based on information of sensors provided in each of the endoscopes or feature amounts of images, and based on the detection results. To combine the images.

<5.3. HMD>
The present technology can also be applied to information processing apparatuses other than the optical system apparatus. For example, the present technology can be applied to an information processing apparatus capable of outputting video, such as an HMD, a PC, a smartphone, and a car navigation apparatus. As an example thereof, FIG. 18 illustrates a configuration example when the present technology is applied to an HMD.

FIG. 18 is a diagram illustrating an example of a hardware configuration of the HMD 2 according to the present modification. As shown in FIG. 18, the HMD 2 includes a pair of left and right camera blocks 2100, a display element 2200, and an eyepiece lens 2300.

The light incident through the lens 2110 forms an image on the imager 2120 and is imaged by the camera block 2100. The HMD 2 includes an image processing apparatus (not shown), synthesizes the captured images captured by the camera block 2100, and displays the combined image on the display element 2200. The image displayed by the display element 2200 passes through the eyepiece lens 2300 and enters the user's eye.

Any one of the pair of left and right camera blocks 2100 may function as a main optical system, and the other may function as a sub optical system. In addition, the HMD 2 simulates each of the video obtained by the main camera and the video obtained by the sub camera by performing image processing such as enlargement / reduction on the captured image captured by the pair of left and right camera blocks 2100. May be generated automatically.

Here, when the HMD 2 is used, the user's field of view is completely blocked by the display. For this reason, if the actual movement of the body and the video do not match, there is a risk of video sickness and discomfort. Therefore, as shown in FIG. 19, the HMD 2 may synthesize the user's external image captured by the camera block 2100 with the peripheral visual field. As a result, the actual movement of the body matches the image of the peripheral visual field, and it is possible to suppress the user's image sickness and discomfort.

FIG. 19 is a diagram showing an example of an image displayed by the HMD 2 according to this modification. An area denoted by reference numeral 51 is an area in which a main video is shown. For example, a video of the virtual world may be shown in the area indicated by reference numeral 51. In addition, the image obtained by the endoscope may be reflected in the area indicated by reference numeral 51. An area indicated by reference numeral 52 is an area in which an image of the user's outside world is shown at the same magnification. In the example shown in FIG. 19, the user's external image is shown at the same magnification on the outside (the area indicated by reference numeral 52) of the two divided areas, but the inner area (the area indicated by reference numeral 51). It is hard to say that the apparent viewing angle is expanded. Therefore, the HMD 2 projects the user's external image at the same magnification on the outermost side of the three-divided region, and performs the above-described processing for extending the apparent viewing angle using the two inner regions. Good.

As a use case of HMD2, there is a use of viewing a spherical image prepared in advance. For example, HMD2 senses the movement of the user's head and moves the image according to the movement of the head, thereby providing an image as if the user is entering the virtual world and watching the surroundings. Can do. The HMD 2 may perform the above-described processing for extending the apparent viewing angle when the user uses an optical system device such as virtual binoculars and a telescope in the virtual world. Specifically, when zooming, the HMD 2 may perform image processing shown as an example in FIGS. Further, the HMD 2 may suppress the motion sickness of the user by synthesizing a virtual world video or a real world video of the same size at the outermost periphery.

<< 6. Summary >>
The embodiment of the present disclosure has been described in detail above with reference to FIGS. As described above, the image processing apparatus 200 according to this embodiment performs imaging based on the relationship between the angles of view of the subject images obtained by the main optical system and the sub optical system, and obtains an image obtained by the sub optical system. The irradiation process is controlled so that the image is irradiated to an eyepiece that irradiates at least a part of the subject image obtained by the main optical system. Since the apparent viewing angle of the main optical system is pseudo-expanded, the optical system apparatus can be reduced in size and weight compared with the case where the apparent viewing angle of the main optical system is physically expanded. In addition, it can be configured at a low cost. In addition, since the image obtained by the sub optical system corresponds to the peripheral visual field, the image quality can be lowered, and the optical system apparatus can be configured more inexpensively.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

For example, each device described in this specification may be realized as a single device, or a part or all of the devices may be realized as separate devices. For example, in the functional configuration example of the binoculars 1 illustrated in FIG. 5, the image processing apparatus 200 may be provided in a device such as a server connected to other components via a network or the like.

Further, the series of processing by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. For example, the program constituting the software is stored in advance in a storage medium (non-transitory medium) provided inside or outside each device. Each program is read into a RAM when executed by a computer and executed by a processor such as a CPU.

In addition, the processes described using the flowcharts and sequence diagrams in this specification do not necessarily have to be executed in the order shown. Some processing steps may be performed in parallel. Further, additional processing steps may be employed, and some processing steps may be omitted.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A subject obtained by the first acquisition unit using a captured image obtained by the second acquisition unit, which has been processed based on the relationship between the angles of view of the subject images obtained by the first and second acquisition units. An irradiation control unit that controls the irradiation process so as to irradiate an eyepiece to which at least a part of the image is irradiated;
An information processing apparatus comprising:
(2)
The information processing apparatus according to (1), wherein the irradiation control unit irradiates the captured image to a second region at an outer edge of the first region irradiated with the subject image obtained by the first acquisition unit. .
(3)
The information processing apparatus according to (2), wherein the captured image is processed to represent a subject image included in the captured image at a lower magnification than a subject image irradiated on the first region.
(4)
The information processing apparatus according to (3), wherein the captured image is processed such that the angle of view increases non-uniformly outward from a boundary portion between the first region and the second region.
(5)
The information processing apparatus according to any one of (2) to (4), wherein the captured image is processed so that an angle of view of a boundary portion between the first area and the second area matches. .
(6)
Any one of (2) to (5), wherein the captured image is processed so that a magnification relationship between a subject image irradiated on the first region and a subject image included in the captured image is maintained. The information processing apparatus according to one item.
(7)
The information processing apparatus according to (2), wherein the captured image is processed to represent a subject image included in the captured image at the same magnification as a subject image irradiated on the first region.
(8)
The information processing apparatus according to any one of (2) to (7), wherein the irradiation control unit controls the size of the first region and the second region.
(9)
The information processing apparatus according to any one of (1) to (8), further including an operation mode control unit that switches contents of the irradiation process.
(10)
The information processing apparatus according to (9), wherein the operation mode control unit performs the switching according to a user operation.
(11)
The information processing apparatus according to (9) or (10), wherein the operation mode control unit executes the switching according to a user state.
(12)
The irradiation control unit stores the captured image obtained by the first obtaining unit in a storage unit, and obtains the subject image obtained by the first obtaining unit from the storage unit in an area out of the field of view. The information processing apparatus according to any one of (1) to (11), wherein the captured image corresponding to the region is irradiated.
(13)
A subject obtained by the first acquisition unit using a captured image obtained by the second acquisition unit, which has been processed based on the relationship between the angles of view of the subject images obtained by the first and second acquisition units. Controlling the irradiation process to irradiate an eyepiece to which at least a part of the image is irradiated;
An information processing method executed by a processor.
(14)
A first acquisition unit;
A second acquisition unit;
An eyepiece that irradiates at least a part of the subject image obtained by the first acquisition unit;
Irradiation processing to irradiate the eyepiece with the captured image obtained by the second acquisition unit, which has been processed based on the relationship between the field angles of the subject images obtained by the first and second acquisition units. An irradiation control unit for controlling
An information processing apparatus comprising:

DESCRIPTION OF SYMBOLS 1 Binoculars 100 Secondary optical system camera block 110 Lens 120 Imager 200 Image processing device 210 Irradiation control part 220 Operation mode control part 230 Memory | storage part 300 Main optical system camera block 310 Lens 320 Imager 400 Display block 410 Lens 420 LCD
500 Beam splitter 600 Objective lens 700 Prism 800 Eyepiece 900 Switch 1000 Memory card 1100 Electronic viewfinder 1110 Eyepiece 1120 LCD
2 HMD
2100 Camera block 2110 Lens 2120 Imager 2200 Display element 2300 Eyepiece

Claims

A subject obtained by the first acquisition unit using a captured image obtained by the second acquisition unit, which has been processed based on the relationship between the angles of view of the subject images obtained by the first and second acquisition units. An irradiation control unit that controls the irradiation process so as to irradiate an eyepiece to which at least a part of the image is irradiated;
An information processing apparatus comprising:
The information processing apparatus according to claim 1, wherein the irradiation control unit irradiates the captured image to a second region at an outer edge of the first region irradiated with the subject image obtained by the first acquisition unit.
The information processing apparatus according to claim 2, wherein the captured image is processed to represent a subject image included in the captured image at a lower magnification than a subject image irradiated on the first region.
The information processing apparatus according to claim 3, wherein the captured image is processed such that a field angle increases non-uniformly outward from a boundary portion between the first area and the second area.
The information processing apparatus according to claim 2, wherein the captured image is processed so that an angle of view of a boundary portion between the first area and the second area matches.
The information processing apparatus according to claim 2, wherein the captured image is processed so that a magnification relationship between a subject image irradiated on the first region and a subject image included in the captured image is maintained.
The information processing apparatus according to claim 2, wherein the captured image is processed to represent a subject image included in the captured image at the same magnification as a subject image irradiated on the first region.
The information processing apparatus according to claim 2, wherein the irradiation control unit controls the sizes of the first area and the second area.
The information processing apparatus according to claim 1, further comprising an operation mode control unit that switches contents of the irradiation process.
The information processing apparatus according to claim 9, wherein the operation mode control unit executes the switching in accordance with a user operation.
The information processing apparatus according to claim 9, wherein the operation mode control unit performs the switching according to a user state.
The irradiation control unit stores the captured image obtained by the first obtaining unit in a storage unit, and obtains the subject image obtained by the first obtaining unit from the storage unit in an area out of the field of view. The information processing apparatus according to claim 1, wherein the captured image corresponding to the region is irradiated.
A subject obtained by the first acquisition unit using a captured image obtained by the second acquisition unit, which has been processed based on the relationship between the angles of view of the subject images obtained by the first and second acquisition units. Controlling the irradiation process to irradiate an eyepiece to which at least a part of the image is irradiated;
An information processing method executed by a processor.
A first acquisition unit;
A second acquisition unit;
An eyepiece that irradiates at least a part of the subject image obtained by the first acquisition unit;
Irradiation processing to irradiate the eyepiece with the captured image obtained by the second acquisition unit, which has been processed based on the relationship between the field angles of the subject images obtained by the first and second acquisition units. An irradiation control unit for controlling
An information processing apparatus comprising: