JP2020010111A - Display device, imaging device, and control method, program, and imaging system for the devices - Google Patents

Abstract

To enable easy grasping of an exposure situation.SOLUTION: An operation/display terminal unit 1100, in displaying received omnidirectional video, makes a display unit 1103 display extracted video 401 obtained by extracting one or some angles of view from all angles of view and omnidirectional reduced video 402 of all angles of view in parallel and, when receiving exposure control operation from a user, transmits an exposure control request to a camera unit 1000. When receiving omnidirectional video in which exposure control corresponding to exposure control request information has been reflected and inappropriate coordinates are received from the camera unit 1000, the operation/display terminal unit 1100 superposes and displays a marker 407 on the omnidirectional reduced video 402 and also superposes and displays a marker 407 on the extracted video 401 when the inappropriate coordinates are included in angles of view of the extracted video 401.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a display device that handles omnidirectional images, an imaging device, and a control method, a program, and an imaging system thereof.

  Recently, techniques for capturing panoramic images and 360 ° omnidirectional images have attracted attention. The imaging device according to this technology arranges a plurality of sensors or moves the imaging device itself so as to cover a wide angle of view, and synthesizes a plurality of acquired images to form a wide-angle image. Generate The obtained video is transferred to a display device such as a terminal different from the imaging device and displayed using, for example, wireless technology. When displaying the transferred video on the display device, it is expected that the reduction ratio is too high and the viewing is difficult if the angle of view is wide. Therefore, many display devices are provided with a user interface that can be arbitrarily enlarged or reduced so as to have an angle of view for user viewing.

  As described above, when viewing the angle of view for user viewing on the display device at a location distant from the imaging location, the user cannot know the exposure status of the angle of view that is not displayed. Here, there is known a technique of detecting that extremely dazzling light such as sunlight enters an angle of view that is not viewed and automatically changing exposure control (Patent Document 1). However, since the exposure is automatically controlled, it is difficult to set the full angle of view of the omnidirectional image to the exposure state desired by the user.

  On the other hand, there is also a usage form in which an image having an angle of view for user viewing and an image obtained by reducing and displaying the entire angle of view are simultaneously displayed. In this use mode, a technology that makes it easy for a user to edit an image by superimposing some drawing effect such as blurring on an angle of view for user viewing and reflecting the drawing effect on the entire angle of view. Is also known (Patent Document 2).

JP 2013-198070 A JP-A-2017-84422

  By the way, some display devices allow the user to arbitrarily change the exposure of the angle of view being viewed. In such a display device, the exposure state at an angle of view invisible to the user may not be appropriate due to the change in the exposure according to the user operation. Therefore, it is desirable for the user to be able to grasp the exposure state at all angles of view. However, even if the user tries to confirm the exposure condition by displaying the full angle of view, it is difficult to confirm the entire exposure condition because the omnidirectional image is reduced and displayed. As described above, there is a problem that it is not easy for the user to grasp the exposure state of the omnidirectional image at all angles of view.

  An object of the present invention is to make it easy to grasp an exposure situation.

  In order to achieve the above object, the present invention provides a receiving unit for receiving an omnidirectional image from an imaging device, a display unit, and displaying the omnidirectional image received by the receiving unit. Control means for displaying a first image obtained by cutting out a part of the angle of view from the angle of view and the second image having the full angle of view in parallel on the display unit, and a receiving means for receiving an exposure control operation Transmitting means for transmitting an exposure control request to the imaging device based on the exposure control operation received by the receiving means, wherein the control means reflects the exposure control request transmitted by the transmitting means. When the receiving unit receives from the imaging device a notification indicating that an improperly exposed portion is present in the omnidirectional video that has been reflected and the omnidirectional video in which the exposure control request is reflected, based on the notification, , Serial exposure and wherein the displaying the second image by adding the visual effect on the display unit in improper positions.

  According to the present invention, it is possible to easily grasp the exposure situation.

It is a control block diagram of an imaging system. It is a control block diagram of the imaging system of the comparative example. It is a flowchart of a camera side process and a display side process. It is a figure showing the example of a display of a display part. It is a figure showing an example of a picture of a marker center angle of view. It is a figure showing the example of a display of the display part where the image of the marker center angle of view was displayed. It is a flowchart of a display side process. FIG. 14 is a diagram illustrating a display example of a display unit according to a modification.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First Embodiment)
FIG. 1 is a control block diagram of the imaging system according to the first embodiment of the present invention. This imaging system is configured such that a camera unit 1000 that can capture an omnidirectional image (omnidirectional image) and an operation / display terminal unit 1100 that performs user operations and image display are communicably connected.

  FIG. 2 is a control block diagram of the imaging system of the comparative example. In the comparative example (FIG. 2), the same components as those of the imaging system of the present embodiment are denoted by the same reference numerals. Components common to both of them will be mainly described in the description of the imaging system of the present embodiment.

  In the imaging system of the present embodiment, an improper location detection unit 2020 and an improper location notification unit 2021 are added to the imaging system of the comparative example. The imaging system according to the present embodiment further includes an inappropriate location receiving unit 2120, a marker superimposing unit 2121, a marker specifying unit 2122, a magnification calculating unit 2123, and another marker detecting unit 2124. Note that the marker designation unit 2122, the magnification calculation unit 2123, and the other marker detection unit 2124 are used in the second embodiment described later, and therefore, it is not essential to provide them in this embodiment.

  First, the configuration and operation of the present embodiment, including portions common to the comparative example, will be described with reference to FIG. A plurality of sensors need to be arranged in the camera unit 1000 so as to capture images in all directions of 360 °. In the example of FIG. 1, image sensors 1003 and 1004 are arranged as two sensors (imaging means). The subject light forms an image on image sensors 1003 and 1004 through lens groups 1001 and 1002 including a shift lens, respectively. The imaging elements 1003 and 1004 photoelectrically convert the formed subject optical image into an electric signal. The captured images are sequentially input to the signal processing units 1005 and 1006 line by line. The signal processing units 1005 and 1006 perform video correction processing and phenomenon processing to optimize the video, and the generated video is stored in the memory 1007.

The photometric value acquiring unit 1008 acquires a photometric value from the output of the signal processing units 1005 and 1006, and sends the photometric value to the evaluation value generating unit 1009. The evaluation value generation unit 1009 calculates an evaluation value based on the sent photometric value, and compares the evaluation value with a predetermined reference to determine whether the exposure is appropriate. Regardless of the method of acquiring the evaluation value, an example is shown here. The evaluation value generation unit 1009 divides the entire angle of view of the omnidirectional video into an N × M matrix, and acquires the light amount value for each frame, which is each of the divided angle of view. Each frame is assigned a predetermined weight (multiplication coefficient) according to the location. The evaluation value generation unit 1009 calculates an evaluation value by multiplying the light amount value of each frame by the weight of the frame and averaging the calculation results of all the frames. For the weighting, there may be various patterns depending on the design, such as a large value near the center and a large value near the subject. The values and formulas for calculating the evaluation value are as follows.
Number of frames (N horizontal, M vertical): N × M
Photometric value of a certain frame (nth horizontal, mth vertical): Lnm
Weight (multiplication coefficient) of a certain frame (nth horizontal and mth vertical): Knm
Evaluation value of a certain frame (nth horizontal and mth vertical): Ynm = Lnm × Knm
Evaluation value = (Y11 + Y12 +... YNM) / (N × M)
Note that this evaluation value calculation method is merely an example, and another calculation method may be used.

  If the evaluation value generation unit 1009 determines that the exposure is not proper, the exposure control unit 1010 issues an operation command to the gain control unit 1011 and the shutter control unit 1012 so as to obtain proper exposure. In response to the operation command, the gain control unit 1011 changes the operation of the signal processing unit 1005 and the signal processing unit 1006, and the shutter control unit 1012 changes the operation of the electronic shutters of the imaging devices 1003 and 1004. As a result, the images exposed by the imaging elements 1003 and 1004 are properly exposed. Each video is combined by the encoding processing unit 1013 into one outputtable video (omnidirectional video), and sent to the network communication unit 1014.

  Various types of information including video are transmitted and received between a network communication unit 1014 in the camera unit 1000 and a network communication unit 1101 (transmission unit, reception unit) in the operation / display terminal unit 1100. Regardless of the communication standard, it may be wireless or wired. The video received by the operation / display terminal unit 1100 is stored in the memory 1102.

  In the present embodiment, as a display mode on display unit 1103, a mode in which both a clipped image and an omnidirectional image are displayed in parallel is assumed. Note that only one of these may be displayed by mode switching or the like. Here, with respect to the omnidirectional video, the cutout enlargement / reduction control unit 1104 displays the omnidirectional reduced video 402 (second video; see FIG. 4) obtained by reducing the 360 ° omnidirectional view angle (all view angles). On the other hand, with respect to the cut-out image 401 (first image; see FIG. 4), the cut-out enlargement / reduction control unit 1104 cuts out a part of the angle of view according to the user's request from the full angle of view in the omnidirectional image, and The image is enlarged or reduced and displayed as a cutout image 401. During shooting, the cutout video 401 is a live view video (LV).

  On the screen on which the cut-out image 401 (see FIG. 4) of the angle of view requested by the user is displayed, the user operates the exposure operation unit 1105 as an accepting unit that receives an exposure control operation to perform exposure correction so that desired brightness is obtained. Can be operated. The content of the exposure correction operation is transmitted from the network communication unit 1101 to the network communication unit 1014 of the camera unit 1000 as exposure control request information. In the camera unit 1000, the exposure control request information is passed to the exposure control unit 1010 by the controller 1015, and the exposure control unit 1010 executes the exposure control according to the exposure control request information so that the exposure desired by the user is obtained.

  The camera unit 1000 includes a RAM and a ROM (not shown) in addition to the CPU 1200. The ROM stores a control program executed by the CPU 1200. The RAM serves as a work area when the CPU 1200 executes the control program. The function of each functional unit in the camera unit 1000 is realized by cooperation of the CPU 1200, RAM, ROM, and the like. Each functional unit here includes at least signal processing units 1005 and 1006, photometric value acquisition unit 1008, evaluation value generation unit 1009, exposure control unit 1010, gain control unit 1011, shutter control unit 1012, and encoding processing unit 1013. included. In addition, each functional unit includes an inappropriate location detection unit 2020 and an inappropriate location notification unit 2021.

  The operation / display terminal unit 1100 includes a RAM and a ROM (not shown) in addition to the CPU 1300 as a control unit. A control program executed by the CPU 1300 is stored in the ROM. The RAM is a work area when the CPU 1300 executes the control program. The function of each functional unit in the operation / display terminal unit 1100 is realized by cooperation of the CPU 1300, RAM, ROM, and the like. Each functional unit includes at least a cutout enlargement / reduction control unit 1104, a marker superimposition unit 2121, a marker designation unit 2122, a magnification calculation unit 2123, and another marker detection unit 2124.

  In the camera unit 1000, an inappropriate portion detection unit 2020 as a detection unit calculates an evaluation value Ynm of each frame according to the photometry value acquired by the photometry value acquisition unit 1008. If the calculated evaluation value Ynm is out of the range of the predetermined threshold, the inappropriate position detection unit 2020 determines that the frame is not a proper exposure, and determines that the evaluation value Ynm is equal to or smaller than the predetermined threshold. If it is within the range, it is determined that the frame is properly exposed. The improper location detection unit 2020 passes the coordinate information (referred to as “inappropriate coordinates”) of all the frames determined not to be the proper exposure to the improper location notification unit 2021. The inappropriate location notification unit 2021 transmits the passed inappropriate coordinates to the operation / display terminal unit 1100 via the network communication unit 1014. The transmission of the inappropriate coordinates is also a notification indicating that there is a portion where the exposure is inappropriate in the omnidirectional video. Inappropriate coordinates are information for specifying a portion of the omnidirectional image where the exposure is inappropriate. The position of the frame and the angle of view corresponding to the frame can be specified by the coordinate information of the frame.

  In the operation / display terminal unit 1100, when the network communication unit 1101 receives the inappropriate coordinates, it passes it to the inappropriate location receiving unit 2120. The incorrect position receiving unit 2120 converts the inappropriate coordinates into coordinates of the omnidirectional angle of view, and passes the converted coordinate information to the marker superimposing unit 2121. The marker superimposing unit 2121 superimposes the marker on the passed coordinates in the omnidirectional angle of view of the omnidirectional video and writes the marker into the memory 1102. Here, the marker is an example of a visual effect for visually recognizing that the exposure is not appropriate. The image with the marker written in the memory 1102 is displayed on the display unit 1103 (described later with reference to FIG. 4). An operation including a process of superimposing a marker will be described with reference to FIG.

  FIG. 3A is a flowchart of camera-side processing executed by the camera unit 1000. FIG. 3B is a flowchart of display-side processing executed by the operation / display terminal unit 1100. The camera-side processing is realized by the CPU 1200 expanding a program stored in the ROM of the camera unit 1000 into a RAM (both not shown) and executing the program. The display-side processing is realized by the CPU 1300 expanding a program stored in the ROM of the operation / display terminal unit 1100 into a RAM (both not shown) and executing the program. The camera-side processing and the display-side processing are started when communication between the camera unit 1000 and the operation / display terminal unit 1100 is established, and ended when the communication is disconnected. Both processes are performed in parallel. It is assumed that the omnidirectional video is transmitted from the camera unit 1000 to the operation / display terminal unit 1100 at the start of the camera-side processing and the display-side processing.

  First, the camera-side processing (FIG. 3A) will be described. In step S3001, the CPU 1200 waits until it determines that the exposure control request information transmitted from the operation / display terminal unit 1100 has been received by the network communication unit 1014. This is because if the exposure control request information is received, it can be determined that an exposure operation has been performed by the user. If CPU 1200 determines that the exposure control request information has been received, it advances the process to step S3002.

  In step S3002, the CPU 1200 causes the exposure control unit 1010 to perform exposure control (exposure correction) so as to achieve proper exposure according to the exposure control request information. In step S3003, the CPU 1200 determines whether or not there is a frame (location) that is determined by the inappropriate location detection unit 2020 as not being properly exposed. If the CPU 1200 determines that there is no frame determined not to be the proper exposure, the process proceeds to step S3005. On the other hand, if the CPU 1200 determines that there is a frame determined not to be the proper exposure, it executes step S3004 and then advances the processing to step S3005.

  In step S3004, the CPU 1200 transmits the coordinate information (inappropriate coordinates) of the frame determined not to be the appropriate exposure to the operation / display terminal unit 1100 through the network communication unit 1014 by the inappropriate position notification unit 2021. In step S3005, the CPU 1200 causes the encoding processing unit 1013 to transmit an omnidirectional video image of all angles of view to the operation / display terminal unit 1100 via the network communication unit 1014. After step S3005, CPU 1200 returns the process to step S3001.

  Next, the display-side processing (FIG. 3B) will be described. In step S3101, the CPU 1300 waits until it is determined that the user has performed an exposure correction operation on the exposure operation unit 1105. If the CPU 1300 determines that an exposure correction operation has been performed, the CPU 1300 transmits exposure control request information corresponding to the content of the exposure correction operation to the camera unit 1000 via the network communication unit 1101. Note that the exposure control request information is received by the camera unit 1000 in step S3001.

  In step S3102, the CPU 1300 receives an omnidirectional video image at all angles of view from the camera unit 1000 via the network communication unit 1101. The received omnidirectional image of all angles of view is the image on which the exposure control in step S3002 of FIG. 3A has been performed. That is, the omnidirectional video is the omnidirectional video transmitted in step S3005 after the exposure control request information transmitted in step S3101 is reflected.

  In step S3103, the CPU 1300 cuts out the received omnidirectional video image of all angles of view and displays the cutout video image 401. That is, the CPU 1300 cuts out / enlarges the omnidirectional video by the cutout / enlargement / reduction control unit 1104 so that the LV (live view) angle of view requested by the user is obtained. Here, a display example of the display unit 1103 of the operation / display terminal unit 1100 will be described with reference to FIG.

  FIG. 4 is a diagram illustrating a display example of the display unit 1103. The display unit 1103 can simultaneously display an omnidirectional reduced image 402 obtained by reducing an omnidirectional image having an omnidirectional angle and a clipped image 401 having an LV (live view) angle of view requested by the user. The angle of view of the cutout video 401 can be arbitrarily specified by the user. The omnidirectional reduced image 402 is obtained by separately generating the image of the first surface angle of view obtained by the image sensor 1003 and the image of the second surface angle of view obtained by the image sensor 1004, and displaying the combined image. is there. The omnidirectional reduced image 402 is composed of an image 402a having an angle of view of 180 ° on the front side and an image 402b having an angle of view of 180 ° on the rear side. Note that the omnidirectional reduced image 402 may be displayed by another method as long as the whole can be looked down on. The markers 407 (407a, 407b, 407c, etc.) superimposed on the omnidirectional reduced image 402 remain undisplayed until the inappropriate coordinates transmitted in step S3004 are received in step S3104.

  In the screen of the cut-out image 401, an icon 405 for notifying the presence of the improperly exposed portion may be displayed in addition to the exposure operation UI 404. The exposure operation UI 404 is an operation unit for the user to perform an exposure correction operation, and corresponds to the exposure operation unit 1105. Note that the exposure operation unit 1105 may be provided as a physical operation unit different from the exposure operation UI 404. The icon 405 is displayed when the inappropriate coordinates are received in step S3104 and a notification indicating that an improperly exposed portion is present is received. The icon 405 makes it easy for the user to notice that there is a portion where the exposure is inappropriate.

  In step S3104, when CPU 1300 determines that the incorrect coordinates have been transmitted, CPU 1300 receives the incorrect coordinates by network communication unit 1101. Next, in step S3105, CPU 1300 determines whether or not there is a frame (location) that is not properly exposed, based on whether or not inappropriate coordinates have been received. If the incorrect coordinates have not been received, the CPU 1300 determines that there is no frame (location) that is not properly exposed, and returns the process to step S3101. On the other hand, when receiving the inappropriate coordinates, the CPU 1300 determines that there is a frame (location) that is not properly exposed, and advances the process to step S3106.

  In step S3106, the CPU 1300 causes the marker superimposing unit 2121 to superimpose the marker 407 at a position corresponding to the incorrect coordinates. Then, in step S3107, CPU 1300 causes display unit 1103 to display an image on which marker 407 is superimposed (FIG. 4). Here, the markers 407 are displayed by the number of the received incorrect coordinates. When the incorrect coordinates are received, the marker 407 is always displayed on the omnidirectional reduced image 402. In the cutout video 401, if the angle of view of the cutout video 401 includes incorrect coordinates, the marker 407 is also displayed on the cutout video 401. After that, the CPU 1300 returns the process to step S3101.

  Note that the marker 407 may be added to a part of the frame or may be added to the entire area of the frame. Note that the addition of the visual effect by the marker superimposing unit 2121 may be in the form of changing the color or luminance in the designated frame or superimposing the pattern. The CPU 1300 may directly refer to the corresponding memory information within the frame of the displayed video, and superimpose a pattern such as a zebra pattern according to the value.

  In addition, the operation / display terminal unit 1100 directly determines the pixel value of the entire video without using the coordinate information, and superimposes a pattern such as a zebra pattern as a visual effect according to the value. May be adopted. In this case, the camera unit 1000 does not necessarily need to transmit the inappropriate coordinates to the operation / display terminal unit 1100, and informs the user that there is a portion where the exposure is inappropriate in the omnidirectional video reflecting the exposure control request. Only the indicated notification may be transmitted. In the case where the inappropriate coordinates are transmitted, the inappropriate coordinates are information for specifying a portion where the exposure in the omnidirectional image is inappropriate, so that the processing load on the operation / display terminal unit 1100 can be reduced.

  According to the present embodiment, when operation / display terminal unit 1100 transmits exposure control request information, camera unit 1000 controls the exposure of the omnidirectional video in accordance with the exposure control request information, and also controls the exposure after the exposure control. Detects improperly exposed areas. Then, the camera unit 1000 transmits the omnidirectional image after the exposure control according to the exposure control request information is reflected to the operation / display terminal unit 1100, and also notifies the operation / display terminal unit 1100 that there is a location where the exposure is inappropriate. Send incorrect coordinates. Then, based on the received notification, the operation / display terminal unit 1100 superimposes and displays the marker 407 on the omnidirectional reduced image 402 and, if the angle of view of the cutout image 401 includes incorrect coordinates, the cutout image A marker 407 is also superimposed on 401. The user can view the omnidirectional reduced image 402 not only from the angle of view being viewed but also from the marker 407, and can visually recognize the inappropriately exposed portion from a bird's-eye view. Therefore, it is possible to easily grasp the exposure state.

  In addition, from the viewpoint of facilitating the confirmation of the exposure in the omnidirectional reduced image 402, it is not essential to superimpose the marker 407 on the cut image 401 even if the angle of view of the cut image 401 includes improper coordinates. .

(Second embodiment)
In the first embodiment, a marker 407 is superimposed on the omnidirectional reduced image 402. However, since the omnidirectional reduced image 402 is a reduced image, it is not easy for the user to check the detailed exposure status corresponding to each marker 407. Therefore, in the second embodiment of the present invention, the individual markers 407 can be displayed in an enlarged manner to make it easier to check the exposure status. In the present embodiment, the marker designation unit 2122, the magnification calculation unit 2123, and the other marker detection unit 2124 in FIG. 1 are also used.

  The user can designate (select) a desired one of the markers 407 superimposed on the omnidirectional reduced image 402 by the marker designation unit 2122 as a selection unit. The mode of designation is, for example, by moving and pressing a cursor. Information indicating the designated marker 407 is sent to the inappropriate location receiving unit 2120. The inappropriate location receiving unit 2120 holds a table that defines the correspondence between the marker 407 and the coordinate information on the omnidirectional reduced image 402 of each of the markers 407. The inappropriate location receiving unit 2120 can acquire the coordinate information on the omnidirectional reduced image 402 of the designated marker 407 with reference to the table. The acquired coordinate information is sent to the magnification calculator 2123.

The magnification calculation unit 2123 calculates the magnification for setting the designated marker 407 at a predetermined position on the screen, for example, at the center, and making the “marker center view angle” further enlarged and displayed. At this time, the magnification calculation unit 2123 determines the angle of view (360 degrees) in all directions so that the frame in which the marker 407 is present has an angle of view (area B) within, for example, 20% of the display screen of the display unit 1103. The magnification R (%) for enlargement is calculated. The value of 20% is a fixed value, but is not limited to the value of 20%. The omnidirectional video transferred from the camera unit 1000 to the operation / display terminal unit 1100 is referred to as an “original image”. The values and formulas for calculating the magnification R (%), which is the magnification of the marker center angle of view with respect to the original image, are as follows. Z is a value indicating the size of the marker in the marker center angle of view, and is, for example, the ratio of the area occupied by the marker.
Number of frames with markers superimposed: X
Number of frames (N horizontal, M vertical): N × M
Original image area: A
Area of view angle displayed on operation / display terminal unit 1100: B
Marker size (%): Z
Area of the angle of view of the marker center image C = A × {X / (N × M)} (Z / 100)
R = B / C = BNMZ / 100AX

  FIGS. 5A and 5B are diagrams illustrating an example of an image of a marker center angle of view. FIG. 6 is a diagram illustrating a display example of the display unit 1103 on which the image of the marker center angle of view is displayed. For example, it is assumed that the user selects the marker 407a from the plurality of markers 407 superimposed on the image by the marker specifying unit 2122 while the omnidirectional reduced image 402 shown in FIG. 4 is displayed. Then, as shown in FIG. 6, instead of the omnidirectional reduced image 402, an image of a marker center angle of view (hereinafter referred to as a marker center angle of view image 403) is cut out as a third image including a visual effect. It is displayed in parallel with the image 401.

  An example of the marker center view angle image 403 is shown in FIG. In displaying the marker center angle of view image 403, the display size of the marker 407 is changed such that the display size of the selected marker 407 is larger than the display size of the omnidirectional reduced image 402, so that the marker 407 becomes easier to see. . It is desirable that the Z value be set so that the magnification R is such that the entire selected marker 407 appears in the marker center view angle image 403. The Z value may be a fixed value, or may be specified by the user. In that case, if the user wants to display the selected marker 407 in a large size, the user may set the Z value to a larger value.

  Note that the method of calculating the magnification R is not limited to the illustrated one. From the viewpoint of making the selected marker 407 easier to see, the magnification R may be calculated so that the display size of the selected marker 407 is larger than the display size of the omnidirectional reduced image 402.

  FIG. 7 is a flowchart of display-side processing executed by the operation / display terminal unit 1100. The display-side processing is realized by the CPU 1300 expanding a program stored in the ROM of the operation / display terminal unit 1100 into a RAM (both not shown) and executing the program. The camera-side processing executed by the camera unit 1000 is the same as that described with reference to FIG.

  Steps S3101 to S3107 are the same as those described in FIG. However, after step S3107, the process proceeds to step S701. In step S701, the CPU 1300 of the operation / display terminal unit 1100 determines whether any of the markers 407 on the omnidirectional reduced image 402 has been selected by a user's selecting operation on the marker specifying unit 2122. When determining that marker 407 has not been selected, CPU 1300 returns the process to step S3101. On the other hand, when determining that any marker 407 has been selected, CPU 1300 advances the process to step S702.

  In step S702, as described above, the CPU 1300 causes the magnification calculator 2123 to calculate the magnification R for enlarging and displaying the selected marker 407 as the marker center view angle image 403. Next, in step S703, the CPU 1300 displays the marker center view angle image 403 in place of the omnidirectional reduced image 402 according to the calculated magnification R (FIG. 6). After that, the CPU 1300 returns the process to step S3101.

  According to the present embodiment, the same effects as in the first embodiment can be obtained with respect to facilitating the grasp of the exposure situation. Further, in displaying the marker center view angle image 403, the selected marker 407 is displayed in a larger scale than when the omnidirectional reduced image 402 is displayed. Detailed understanding of the exposure situation is facilitated.

  Note that the marker center angle of view is set in frame units set by the camera unit 1000, but is not limited thereto. For example, the CPU 1300 may cut out a predetermined angle of view around the selected marker.

  By the way, the marker center view angle image 403 may include a part of a marker different from the selected marker. In such a case, the CPU 1300 may change the magnification R such that the entirety of the selected marker and the entirety of the another marker appear in the marker center view angle image 403. The operation for this will be described.

  In step S702, after calculating the magnification R, the CPU 1300 determines whether or not a part of another marker different from the one selected by the user enters the marker center angle-of-view image 403 by the other marker detection unit 2124. Is determined based on For example, in the omnidirectional reduced image 402 shown in FIG. 4, three markers 407b are displayed overlapping each other. When one of the three markers 407b is selected, a part of the other one or two markers 407b may enter the marker center view image 403 when the magnification R calculated at the first time is applied. There is.

  Therefore, in step S702, if the CPU 1300 determines that a part of another marker does not enter after calculating the magnification R, the process proceeds to step S703. However, if the CPU 1300 determines that a part of another marker enters, the CPU 1300 sets the magnification R so that the whole of the another marker corresponding to the entering marker is included in the marker center view angle image 403 and displayed. change. At that time, the CPU 1300 obtains a new magnification R by multiplying the current magnification R by a predetermined coefficient y (smaller than 1). If the whole of the other marker does not fit in the marker center view angle image 403 to which the newly obtained magnification R has been applied, the current magnification R is again multiplied by a predetermined coefficient y, and a new coefficient R is obtained. Obtain the magnification R. The CPU 1300 repeats the magnification R by multiplying by the predetermined coefficient y until the whole of the another marker appears in the marker center angle-of-view image 403, and then advances the process to step S703. As a result, for example, as shown in FIG. 5B, a marker center view angle image 403 showing three markers 407b is displayed.

  In the present embodiment, the marker center angle of view image 403 is displayed instead of the omnidirectional reduced image 402, but the present invention is not limited to this. For example, as shown in a modified example in FIG. 8, the CPU 1300 may separately display the marker center angle of view image 403 while maintaining the display of the omnidirectional reduced image 402. When there are two or more selected markers, CPU 1300 may display a plurality of marker center angle-of-view images 403 corresponding to the number of selected markers.

  In each of the above embodiments, the above-described various controls described as being performed by the operation subject of the flowchart may be performed by one piece of hardware, or the processing may be shared by a plurality of pieces of hardware. May be controlled.

  Further, in the above-described embodiment, the case where the present invention is applied to the imaging device and the display device has been described as an example. However, the present invention can be applied to any imaging device as long as it has a function of capturing an omnidirectional image. The present invention can be applied to any display device as long as it has a function of cutting out and displaying an omnidirectional image. Therefore, as a display device, a personal computer, a PDA, a mobile phone terminal, a portable image viewer, and a printer device having a display can be applied. Further, as the display device, a digital photo frame, a music player, a game machine, an electronic book reader, a tablet terminal, a smartphone, a projection device, a home appliance including a display, a vehicle-mounted device, and the like can be applied.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or an apparatus via a network or various storage media, and a computer (or a CPU or an MPU) of the system or the apparatus reads out the program code and This is the process to be performed. In this case, the program and a storage medium storing the program constitute the present invention.

  As described above, the present invention has been described in detail based on the preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms that do not depart from the gist of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

401 cutout image 402 omnidirectional reduced image 407 marker 1000 camera unit 1100 operation / display terminal unit 1101 network communication unit 1103 display unit 1105 exposure operation unit 1300 CPU

Claims (19)

Receiving means for receiving an omnidirectional image from the imaging device;
A display unit,
When displaying the omnidirectional image received by the receiving means, a first image obtained by cutting out a part of the angle of view from an entire angle of view of the omnidirectional image, and a second image of the omnidirectional image are displayed. Control means for displaying in parallel on the display unit,
Receiving means for receiving an exposure control operation;
Transmitting means for transmitting an exposure control request to the imaging device based on the exposure control operation received by the receiving means,
The control unit is an omnidirectional image in which the exposure control request transmitted by the transmitting unit is reflected, and a notification indicating that an improperly exposed portion exists in the omnidirectional image in which the exposure control request is reflected. When the receiving means receives from the imaging device, based on the notification, the second video is displayed on the display unit by adding a visual effect to the inappropriately exposed portion. Display device. The notification includes information for specifying a location where the exposure is inappropriate in the omnidirectional video,
The display device according to claim 1, wherein the control unit adds the visual effect to a position based on the information.   The control unit, when the inappropriately exposed portion is included in the partial angle of view, adds the visual effect to the inappropriately exposed portion and displays the first image on the display unit. The display device according to claim 1, wherein the display device displays the information. Selecting means for selecting a visual effect from the second video to which at least one of the visual effects is added;
The control means may control the third video including the visual effect selected by the selection means so that a display size of the selected visual effect is larger than a display size of the second video. 4. The display device according to claim 1, wherein a display size of the selected visual effect is changed and displayed on the display unit. 5.   The control means sets a magnification so that the entirety of the selected visual effect appears in the third image, and changes the display size of the selected visual effect at the set magnification to change the display size of the selected visual effect. The display device according to claim 4, wherein the image of No. 3 is displayed on the display unit.   The control means, when displaying the third video, if the third video includes a part of a visual effect different from the selected visual effect, The display device according to claim 5, wherein the magnification is changed such that the entire visual effect and the entire other visual effect appear in the third video.   7. The display device according to claim 4, wherein the control unit displays the third image instead of the second image. 8.   7. The display device according to claim 4, wherein the control unit displays the third image in parallel with the second image. 8.   5. The display according to claim 4, wherein when the plurality of visual effects selected by the selection unit are plural, the control unit displays the third video image by the number of the selected visual effects. 6. apparatus.   2. The control unit, in response to receiving the notification, displaying an icon for notifying the presence of the inappropriately exposed portion in the first video. 3. 10. The display device according to any one of claims 9 to 9.   The display device according to any one of claims 1 to 10, wherein the addition of the visual effect is to superimpose a predetermined marker. Imaging means for capturing an omnidirectional image;
Receiving means for receiving the exposure control request;
Control means for controlling the exposure of the omnidirectional video imaged by the imaging means, according to the exposure control request received by the receiving means,
In the omnidirectional video after the exposure control according to the exposure control request by the control unit is reflected, a detection unit that detects a portion where exposure is inappropriate,
The omnidirectional image after the exposure control according to the exposure control request is reflected by the control unit, and when the detection unit detects the improper exposure, the exposure is improper. And a transmission unit for transmitting a notification indicating that there is a unique part.   13. The imaging apparatus according to claim 12, wherein the notification includes information for specifying a portion where the exposure is inappropriate in the omnidirectional video.   The detection unit acquires an evaluation value for each of the divided angle of view obtained by dividing the angle of view of the omnidirectional video imaged by the imaging unit into a plurality of parts, and, based on the acquired evaluation value, for each of the divided angle of view. 14. The imaging apparatus according to claim 12, wherein whether the exposure is appropriate or not is determined, and a divided angle of view for which the exposure is determined to be inappropriate is detected as the location where the exposure is incorrect.   A program for causing a computer to function as each unit of the display device according to claim 1.   A program for causing a computer to function as each unit of the imaging device according to any one of claims 12 to 14. A method for controlling a display device, comprising:
Receiving omnidirectional video from the imaging device,
When displaying the received omnidirectional image, a first image obtained by cutting out a part of the angle of view from an entire angle of view of the omnidirectional image, and a second image having the omnidirectional angle are displayed on a display unit. To be displayed in parallel,
Accept exposure control operation,
Based on the received exposure control operation, transmit an exposure control request to the imaging device,
When the transmitted omnidirectional video reflecting the exposure control request and the notification indicating that there is an improperly exposed portion in the omnidirectional video reflecting the exposure control request are received from the imaging device, A control method for a display device, wherein a visual effect is added to a portion where the exposure is inappropriate based on the notification, and the second image is displayed on the display unit. A method for controlling an imaging device that captures an omnidirectional video,
Receiving an exposure control request,
According to the received exposure control request, controlling the exposure of the captured omnidirectional video,
In the omnidirectional image after the exposure control according to the exposure control request is reflected, the portion where the exposure is inappropriate is detected,
While transmitting the omnidirectional image after the exposure control according to the exposure control request is reflected, and when the location where the exposure is inappropriate is detected, it indicates that the location where the exposure is inappropriate exists. A method for controlling an imaging device, comprising: transmitting a notification. An imaging system having an imaging device and a display device capable of communicating with the imaging device,
The imaging device,
Imaging means for capturing an omnidirectional image;
First receiving means for receiving an exposure control request;
First control means for controlling the exposure of the omnidirectional video imaged by the imaging means in response to the exposure control request received by the receiving means;
Detecting means for detecting a portion where exposure is inappropriate in the omnidirectional image after the exposure control according to the exposure control request is reflected by the first control means;
The first control means transmits an omnidirectional image after the exposure control according to the exposure control request is reflected, and the exposure means detects the improperly-exposed portion and detects the exposure. And a first transmission unit that transmits a notification indicating that an inappropriate portion exists,
The display device,
Second receiving means for receiving an omnidirectional image from the imaging device;
A display unit,
When displaying the omnidirectional image received by the second receiving means, a first image obtained by cutting out a part of the angle of view from an entire angle of view of the omnidirectional image, and a second image of the omnidirectional angle Second control means for displaying an image in parallel with the display unit;
Receiving means for receiving an exposure control operation;
And a second transmission unit that transmits an exposure control request to the exposure control operation based on the exposure control operation received by the reception unit.
The second control means includes an omnidirectional image on which the exposure control request transmitted by the second transmission means is reflected, and an inappropriately exposed portion in the omnidirectional image on which the exposure control request is reflected. When the second receiving means receives a notification from the imaging device, the second image is displayed on the display unit by adding a visual effect to the inappropriately exposed portion. Imaging system.