JP2022041716A - Image processing apparatus and control method for the same - Google Patents

Abstract

To reduce an effect of changes in an optical axis position due to changes of a focal length of a photographing device on accuracy of a posture adjustment of a photographing device.SOLUTION: An image processing apparatus includes: a lens setting value obtaining unit (105) configured to obtain a first focal length used for photographing at the time of a posture adjustment of a photographing device, and a second focal length used for photographing at the time of use of the photographing device; a correction value computing unit (107) configured to obtain an optical axis shift amount indicating a difference between an optical axis position at the time of the photographing at the first focal length and an optical axis position at the time of the photographing at the second focal length of the photographing device; a target image storage unit (106) configured to store an image as a target image, corresponding to a photographed image photographed by the photographing device at the adjusted posture and the first focal length, generated on the basis of a three-dimensional model of the adjusted posture, the first focal length, and photographing space of the photographing device; and a target image correction unit (109) configured to generate a target image corrected by shifting the target image on the basis of the optical axis shift amount.SELECTED DRAWING: Figure 1A

Description

The present invention relates to an image processing apparatus and a control method thereof.

Generally, a technique for adjusting the posture of a camera by superimposing and displaying a target image and a captured image is known. For example, there is known a method of detecting a change in posture of a fixed surveillance camera over time by superimposing a past shot image as a target image on a current shot image, and correcting and adjusting the posture change of the camera. In Non-Patent Document 1, the position and orientation of the camera are identified based on the position of the landmark (specific object) in the image taken by the camera and the position coordinates of the landmark calculated in advance by computer graphics. The technology is disclosed. Further, in Patent Document 1, the position and orientation of each of a plurality of cameras for photographing a subject from multiple viewpoints are determined, and information on the determined position and orientation is provided to the user (installer) by the user or the like. The technology that supports the installation work of the camera is disclosed.

Takahiro Kawamura, Junichi Tatemura, Masao Sakauchi, "Augmented Reality Generation Method Using Landmark Information in Real Images", Proceedings of the National Convention 52nd (System), Information Processing Society of Japan, March 1996 6th, p.177-178

Japanese Unexamined Patent Publication No. 2019-092007

When adjusting the shooting direction (camera posture) of the camera so as to shoot the desired shooting range, if the focal length of the camera is long, there is no characteristic part in the image shot by the camera, and the camera is desired. You may not know if it is installed in a posture. In such a case, adjust the posture of the camera using the captured image obtained by shortening the focal length of the camera so that the characteristic part is included in the captured image, and then use the focal length for shooting. It is done to return to the focal length for. Also in Non-Patent Document 1 and Patent Document 1, when a characteristic portion does not exist in the captured image, it is required to shorten the focal length of the camera to acquire a captured image in which the characteristic portion exists. .. After the position and orientation of the camera are identified using the acquired captured image, the focal length is returned to the focal length at the time of use.

However, when the focal length of the camera is changed, the optical axis shifts due to the shape error of the lens, the assembly accuracy, and the like. That is, when the focal length is changed, the position of the optical axis moves without changing the posture of the camera. Therefore, even if the posture of the camera is adjusted so that the captured image obtained by shooting at the focal length changed from the focal length when using the camera matches the target image, the camera's focal length when used is adjusted. The posture is not adjusted correctly. In addition, the amount and direction of the optical axis shift varies depending on the focal length before and after the change, and differs depending on the individual lens.

The present disclosure provides a technique for reducing the influence of a change in the optical axis position due to a change in the focal length of a photographing device on the accuracy of posture adjustment of the photographing device.

The image processing apparatus according to one aspect of the present disclosure has the following configuration. That is,
A first acquisition means for acquiring a first focal length used for photographing when adjusting the posture of the photographing device and a second focal length used for photographing when using the photographing device.
A second acquisition means for acquiring an optical axis shift amount representing the difference between the optical axis position at the time of shooting at the first focal length and the optical axis position at the time of shooting at the second focal length of the photographing device. When,
Imaging taken by the imaging device in the adjusted posture and the first focal length, generated based on a three-dimensional model of the adjusted posture, the first focal length, and the imaging space of the imaging device. A storage means for storing an image corresponding to an image as a target image,
It has a generation means for generating a corrected target image by shifting the target image based on the optical axis shift amount.

According to the present disclosure, it is possible to reduce the influence of the change in the optical axis position due to the change in the focal length of the photographing device on the accuracy of the posture adjustment of the photographing device.

The block diagram which shows the configuration example of the posture adjustment system by 1st Embodiment. The block diagram which shows the hardware configuration example of an image processing apparatus. The flowchart which shows the posture adjustment processing by the image processing apparatus of 1st Embodiment. The figure which shows the measurement example of the optical axis shift amount. The figure which shows the data structure example of the look-up table. The figure explaining the correction of the target image by 1st Embodiment. The figure explaining the superimposition display at the time of posture adjustment by 1st Embodiment. The block diagram which shows the configuration example of the posture adjustment system by 2nd Embodiment. The flowchart which shows the posture adjustment processing by the image processing apparatus of 2nd Embodiment. The block diagram which shows the configuration example of the posture adjustment system by 3rd Embodiment. The figure explaining the superimposition display at the time of posture adjustment by the 3rd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

<First Embodiment>
In the following, the posture adjustment method and its system (posture adjustment) for adjusting the posture of the photographing device by presenting the target image, which is an index of the attitude adjustment of the photographing device, and the real-time photographed image taken by the photographing device to the user. System) will be described. According to each of the following embodiments, the decrease in the accuracy of the posture adjustment of the photographing apparatus due to the optical axis shift of the lens due to the change of the focal length can be improved.

In the first embodiment, at the time of posture adjustment, the focal length of the photographing device is set shorter than the focal length used for photographing at the time of use, and shooting is performed to obtain a captured image for adjustment. That is, at the time of posture adjustment, the focal length of the photographing device is set to the wide-angle side of the photographing at the time of use. In the posture adjustment, the posture of the photographing device is adjusted by using the photographed image obtained from the photographing device and the target image corresponding to the focal length at the time of the attitude adjustment. After the posture adjustment is completed, the focal length of the photographing device is set to the focal length at the time of use, and the imaging of the desired imaging area is executed. The target image is an image used as a target at the time of posture adjustment, and corresponds to a photographed image obtained from a photographing device after the attitude adjustment at the focal length at the time of adjustment. In the present embodiment, the target image corresponds to a captured image obtained by setting the focal length on the wide-angle side of the captured image at the time of use. Therefore, the target image includes more feature points (for example, a soccer goal, a baseball base, a part of a line or a stand, etc.) that are adjustment targets at the time of posture adjustment than a photographed image obtained at the time of use. In the posture adjustment, the posture of the photographing device is adjusted so that the positions of the feature points included in the captured image and the target image match.

FIG. 1A is a block diagram showing a configuration example of a posture adjustment system according to the first embodiment. The posture adjustment system includes a camera 101 as a photographing device, and an image processing device 102 that generates an image for assisting the posture adjustment of the camera 101. The camera 101 is provided with a photographing element inside, and a zoom lens capable of changing the focal length is provided in front of the photographing element. The camera 101 is held by a posture adjusting mechanism (not shown) that adjusts and fixes the direction of the camera. The image processing device 102 takes a target image for adjusting the posture of the camera 101 and displays it on the display 111 together with the captured image from the camera 101. The camera 101 and the image processing device 102 are connected by a camera control cable and an image transmission cable.

In the image processing device 102, the user interface 103 receives various operations from the user. For example, the user interface 103 accepts user input of information necessary for adjusting the posture of the camera 101. Information necessary for posture adjustment includes, for example, the focal length used for posture adjustment of the camera 101 (hereinafter, focal length at the time of adjustment), the focal length used for shooting when the camera 101 is used (hereinafter, focal length at the time of shooting), and the like. Includes identification information for the lens mounted on the camera 101. For example, the individual number of the lens assigned to each individual lens may be used as the identification information of the lens.

The camera control unit 104 is connected to the camera 101 via a camera control cable, and controls pan, tilt, roll, and zoom of the camera 101. For example, the camera control unit 104 controls the focal length of the lens of the camera 101 according to the focal length (focal length at the time of adjustment, focal length at the time of shooting) input from the user interface 103. Further, the camera control unit 104 changes the posture (pan, tilt, roll) of the camera 101 by controlling a pan head (not shown) that supports the camera 101. The lens setting value acquisition unit 105 acquires and stores information necessary for posture adjustment input in the user interface 103. As described above, the information required for posture adjustment includes, for example, the individual number of the lens, the focal length at the time of adjustment, and the focal length at the time of shooting. The target image storage unit 106 stores a target image that is a target when adjusting the posture of the camera 101. The look-up table 107 stores the focal length of the lens and the amount of deviation of the optical axis position (optical axis shift amount) in association with each other for each individual lens. The look-up table 107 records the amount of optical axis shift for each focal length that can be set using the lens to which the camera 101 is mounted. The details of the look-up table 107 will be described later with reference to FIG. The look-up table 107 stores the correspondence between the focal length and the optical axis shift amount, but is not limited to this. For example, instead of the focal length, other indicators corresponding to the focal length may be used.

The correction value calculation unit 108 acquires the optical axis shift amount of the camera 101 that occurs between the time of adjustment and the time of use by referring to the look-up table 107, and corrects the target image based on the acquired optical axis shift amount. Calculate the amount of correction to do. The target image correction unit 109 corrects the target image stored in the target image storage unit 106 using the correction amount calculated by the correction value calculation unit 108, and generates a corrected target image. The image superimposition unit 110 superimposes the target image layer for displaying the target image and the captured image layer for displaying the captured image to generate a superimposed display image. The target image corrected by the target image correction unit 109 is used as the target image layer. As the captured image layer, a real-time captured image input from the camera 101 via the image transmission cable is used. The display 111 displays the superimposed image generated by the image superimposing unit 110.

FIG. 1B is a block diagram showing a hardware configuration example of the image processing device 102. The image processing device 102 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218.

The CPU 211 realizes each function of the image processing device 102 shown in FIG. 1A by controlling the entire image processing device 102 using computer programs and data stored in the ROM 212 and the RAM 213. The image processing device 102 may have one or more dedicated hardware different from the CPU 211, and the dedicated hardware may execute at least a part of the processing by the CPU 211. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 212 stores programs and the like that do not require changes. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I / F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 215 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 102. The display unit 215 can function as the display 111 described above. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 211 in response to an operation by the user. The operation unit 216 can function as the above-mentioned user interface 103. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216.

The communication I / F 217 is used for communication with an external device of the image processing device 102. For example, when the image processing device 102 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the image processing device 102 has a function of wirelessly communicating with an external device, the communication I / F 217 includes an antenna. The bus 218 connects each part of the image processing device 102 to transmit information.

In the present embodiment, it is assumed that the display unit 215 and the operation unit 216 exist inside the image processing device 102, but at least one of the display unit 215 and the operation unit 216 exists as another device outside the image processing device 102. You may be doing it.

FIG. 2 is a flowchart illustrating the posture adjustment process of the camera 101 performed by the posture adjustment system (image processing device 102) according to the first embodiment. Hereinafter, the camera posture adjustment process according to the first embodiment will be described with reference to the flowchart of FIG.

In step S201, the lens setting value acquisition unit 105 acquires information (including the focal length at the time of adjustment, the focal length at the time of shooting, and the lens solid number) necessary for adjusting the posture of the camera 101, and the RAM 213 or the auxiliary storage device 214 (hereinafter referred to as the auxiliary storage device 214). , Described as memory). As described above, the information required for posture adjustment is input by the user, for example, via the user interface 103. The focal length at the time of adjusting the camera posture and the focal length at the time of shooting are limited to the range of the focal length that can be set by the lens created in the camera 101.

In step S202, the target image storage unit 106 acquires the target image and stores it in the memory. The target image is an image used as an index when adjusting the posture of the camera 101, and corresponds to a captured image taken by the camera 101 after adjusting the posture at the focal length at the time of adjustment. The target image is generated based on the position and target posture of the camera 101, the focal length at the time of adjustment, and a three-dimensional model (hereinafter, 3D model) of the shooting space. For example, the target image is 3D based on a 3D model of the structure in the shooting range, the position coordinates where the camera 101 is installed, the gazing point position coordinates of the camera 101, the focal length of the camera 101, the size of the shooting element, and the number of pixels. Created using model simulation software. That is, for the target image, a 3D model space for adjusting the posture of the camera 101 is created, the camera 101 is virtually arranged in the 3D model space, the image taken by the camera 101 is simulated, and computer graphics are used. It is created by drawing with a computer. In this embodiment, the configuration in which the target image storage unit 106 stores the target image created in advance is described, but the present invention is not limited to this. For example, the image processing device 102 may generate the target image by using the information for creating the target image (the above-mentioned 3D model space, the arrangement position (coordinates) of the camera 101, the focal length at the time of adjustment, etc.). ..

In step S203, the correction value calculation unit 108 moves the optical axis position of the lens of the camera 101 with respect to the photographing element of the camera 101 based on the adjusted focal length and the shooting focal length input in step S201. (Optical axis shift amount) is acquired. Then, the correction value calculation unit 108 calculates the correction value of the target image based on the optical axis shift amount. As described above, the image processing apparatus 102 has a look-up table 107 that records the correspondence between the focal length of the lens used and the amount of optical axis shift. The correction value calculation unit 108 acquires the optical axis shift amount when the focal length changes from the focal length at the time of shooting to the focal length at the time of adjustment by referring to the look-up table 107.

Here, an example of measuring the amount of optical axis shift for creating the look-up table 107 will be described with reference to FIG. FIG. 3 is a diagram showing a measurement example of the optical axis shift amount. In the illustrated measurement example, the image captured by the camera 101 is acquired while changing only the focal length without changing the position / posture of the camera 101 and the position of the target object with the focal length of 70 mm as a reference. FIG. 3A shows a photographed image when the focal length is set to 70 mm as a reference and the target 301 is placed at a position separated from the camera 101 by a predetermined distance. A cross guide line 302 is displayed in the center of the captured image. The posture of the camera 101 has been adjusted so that the lower right corner on the front side of the target 301 comes to the center of the captured image (the intersection of the guide lines 302). The pixel position (x, y) corresponding to the lower right corner on the front side of the target 301 at this time is used as a reference (0,0). The coordinate of the pixel position corresponding to the lower right corner on the front side of the target 301 when the focal length is changed from this state is the optical axis shift amount. That is, the number of pixels in the x-direction and the y-direction between the front right lower corner portion of the target 301 and the intersection of the guide lines 302 is recorded as the optical axis shift amount. The index representing the optical axis shift amount is not limited to the number of pixels, and for example, the actual distance (mm) on the photographing element of the camera 101 may be used.

By such a method, the amount of optical axis shift at each focal length when the focal length is changed is measured. FIG. 3B shows a captured image when the focal length of the camera 101 is set to 100 mm without changing the installation position, posture, and target position of the camera 101 from the state of FIG. 3A. When the right side in the horizontal direction of the captured image is the positive direction in the x direction and the lower side in the vertical direction is the positive direction in the y direction, the pixel position (x, y) = (-0) in the lower right corner on the front side of the target 301. .8,0.7) is recorded as the optical axis shift amount. Similarly, in FIGS. 3 (c) and 3 (d), the focal length of the camera 101 is set to 120 mm and 200 mm, respectively, without changing the installation position, posture, and target position of the camera 101 from the state of FIG. 3 (a). Shows the shot image (shooting range) when set. Similar to FIG. 3B, the optical axis shift amount is recorded for each focal length. In this way, a look-up table 107 in which the optical axis shift amount is recorded for each of the plurality of focal lengths that can be set by the camera 101 is generated. Since the optical axis shift amount differs for each individual lens, the optical axis shift amount is measured for each lens to be used, a look-up table 107 is generated, and the look-up table 107 is recorded for each individual lens number.

FIG. 4 shows an example of the look-up table 107 in which the optical axis shift amount measured as described above is recorded. The look-up table 107 records the amount of optical axis shift in the x-direction and y-direction for each lens individual number and the focal length that can be set with the lens (in FIG. 4, it can be set in 1 mm increments in the range of 70 mm to 200 mm). It has the tables 1070 and 1071 that have been made. The optical axis shift amount is the amount of movement of the image when the focal length is changed while using the lens. In the tables 1070 and 1071, the amount of movement in the x-direction and the y-direction of the image when the reference focal length (70 mm in FIG. 4) is changed to each focal length is recorded as the optical axis shift amount. That is, the tables 1070 and 1071 are correspondence tables in which a plurality of focal lengths and the amount of movement of the image at each focal length are recorded. The look-up table 107 includes tables 1070 and 1071 prepared for each lens, table 1070 is a correspondence table for lenses with lens solid number = 1234567, and table 1071 is a correspondence table for lenses with solid number = 1234568. It is a table.

In step 203, the correction value calculation unit 108 refers to the look-up table 107 and calculates the optical axis shift amount when the focal length at the time of shooting is changed to the focal length at the time of adjustment. More specifically, the correction value calculation unit 108 calculates the difference between x and y, which represent the optical axis shift amounts of the focal length at the time of shooting and the focal length at the time of adjustment, respectively. For example, for a lens having a lens solid number = 1234567, the optical axis shift amount is calculated as follows when the focal length at the time of adjustment is 75 mm and the focal length at the time of shooting is 200 mm. That is, the correction value calculation unit 108 refers to the table 1070 and sets the shift amount δx in the x direction when the focal length changes from 75 mm to 200 mm, δx = (-2.0)-(-1.5). = -0.5 is calculated. Further, the correction value calculation unit 108 calculates the optical axis shift amount δy in the y direction when the focal length changes from 75 mm to 200 mm as δy = (1.5)-(1.0) = 0.5. ..

In step S204, the target image correction unit 109 reads out the target image acquired by the target image storage unit 106 in step S202 and stored in the memory. FIG. 5A is an example of a target image stored in the memory, and is computer graphics obtained by simulating a captured image captured at a focal length = 75 mm from the camera 101 to be adjusted. .. In step S205, the target image correction unit 109 corrects the target image read in step S204 according to the optical axis shift amount calculated in step S203. The target image correction unit 109 creates a new target image by translating the target image in the x-direction and the y-direction based on the optical axis shift amount calculated in step S203.

FIG. 5B shows a target image after correction according to the first embodiment. When the focal length is changed from 75 mm to 200 mm, the center of the image (optical axis position) moves by the optical axis shift amount (δx, δy) = (-0.5, 0.5) calculated in step S203. It ends up. Therefore, the target image correction unit 109 shifts the center of the angle of view of the target image shown in FIG. 5A in the direction opposite to the direction (δx, δy) of the optical axis shift amount calculated in step S203. Move only the amount. This movement is indicated by the arrow 501. That is, the target image correction unit 109 moves the angle of view of the target image acquired in step S202 by − (δx, δy) = (0.5, −0.5), and is shown in FIG. 5 (b). A new target image (corrected target image) is generated and stored. In this process, only the optical axis shift amount (δx, δy) = (0.5-0.5) when the focal length changes from the focal length at the time of shooting (200 mm) to the focal length at the time of adjustment (75 mm). , Is equivalent to shifting the range (angle of view) of the target image. Further, the movement of the range of the target image shown in FIG. 5 corresponds to (δx, δy) = (-0.5, 0.5), and the direction and length of the arrow 501 are (δx, δy). It goes without saying that it changes according to the absolute value of) and the sign (positive or negative).

The camera posture adjustment after step S206 is performed using the target image corrected as described above.

In step S206, the camera control unit 104 sets the lens of the camera 101 to the focal length at the time of adjustment (75 mm in this example). In step S207, the image superimposing unit 110 displays on the display 111 a superposed image in which a layer for displaying the target image corrected by the target image correction unit 109 and a layer for displaying the captured image taken by the camera 101 are superimposed. Let me. For example, the image superimposing unit 110 displays the layer of the target image on the front side of the layer of the captured image and displays it as semi-transparent. The transparency of the layer of the target image may be set by the user. The display form of the superimposed image is not limited to this example, and for example, the layer of the captured image may be displayed on the front side of the layer of the target image and displayed as semi-transparent. At this time, the adjusted focal length (75 mm in this example) is set to a focal length closer to the wide angle, which is shorter than the shooting focal length (200 mm). This prevents the occurrence of a situation in which a characteristic portion that serves as a reference for adjusting the posture of the camera does not enter within the shooting range.

In step S208, the posture of the camera 101 is changed according to the user operation. The user operates and adjusts the posture of the camera 101 from the user interface 103 while confirming the superimposed display of the display 111 in step S207. For example, the user can operate the pan, tilt, and roll of the camera 101 via the user interface 103. In addition, when the posture of the camera changes, the captured image also changes in real time according to the change. Since the layer of the target image is displayed as translucent and is displayed superimposed on the layer of the captured image, the user can adjust the posture of the camera so that the captured image matches the target image in the superimposed image. ..

FIG. 6 is a diagram illustrating posture adjustment according to the first embodiment. FIG. 6A shows a layer for displaying the image 61 captured by the camera 101, and FIG. 6B shows a layer for displaying the target image 62. As mentioned above, the target image is displayed semi-transparently. In addition, the target image is shown by a broken line to distinguish it from the captured image. In FIG. 6A, the frame 60 shows a shooting range in shooting with a focal length at the time of shooting. Since the target 301 for posture adjustment does not exist in the captured image based on the focal length at the time of shooting, it is difficult to accurately adjust the posture of the camera 101. The captured image 61 shows a captured image captured at the adjusted focal length, and the target 301 is present in the captured image. The target image 62 in FIG. 6B corresponds to the focal length at the time of adjustment, and the optical axis shift amount is corrected by the target image correction unit 109. The target image 62 corresponds to the corrected target image described in FIG. 5 (b).

FIG. 6C is a diagram showing a superimposed image 63 in which a layer for displaying a captured image 61 and a layer for displaying a target image 62 are superimposed. As described above, the superimposed image 63 is generated by the image superimposing unit 110 and displayed on the display 111. In the superimposed image 63, the layer displaying the target image 62 is superimposed on the front side of the layer displaying the captured image 61. Since the layer displaying the target image 62 is displayed semi-transparently, the layer displaying the captured image 61 behind the target image 62 is also visibly displayed on the display 111. If the posture of the camera 101 is changed in that state, the captured image 61 changes according to the change in the camera posture. The user can adjust the posture of the camera 101 so that the characteristic portions of the target image 62 and the captured image 61 overlap each other, so that the posture of the camera 101 can be accurately adjusted. FIG. 6D shows a superimposed image 63 displayed on the display 111 after adjusting the posture of the camera 101.

When the posture adjustment of the camera 101 is completed as described above, the user instructs the completion of the adjustment via the user interface 103. When the adjustment completion is instructed, in step S209, the camera control unit 104 sets the lens of the camera 101 to the focal length at the time of shooting. When the focal length is changed from the adjusted focal length to the shooting focal length, the shooting center (optical axis position) is only the shift amount (δx, δy) = (-0.5, 0.5) calculated in step S203. Moving. However, since the target image is moved in advance in the direction opposite to the optical axis shift amount in step S205, a desired angle of view (a photographed image taken by a camera 101 whose posture is adjusted correctly at a set shooting focal length). Can be obtained.

As described above, according to the first embodiment, in the posture adjustment using the image taken by the camera 101 at the adjusted focal length different from the shooting focal length, the optical axis shift due to the change of the focal length is caused. It is possible to adjust the posture with less influence. Therefore, even if the posture of the camera is adjusted by the focal length at the time of adjustment and then changed to the focal length at the time of shooting, it is possible to obtain a shot image with the intended camera posture (a shot image in the intended shooting range).

<Second Embodiment>
In the first embodiment, a method of superimposing a target image and a captured image on one display 111 to adjust the posture has been described. In the second embodiment, a configuration for adjusting the posture of the camera by displaying the output image of the camera and the target image on separate display devices and comparing them will be described.

FIG. 7 is a block diagram illustrating a configuration example of the posture adjustment system according to the second embodiment. The monitor 120 is added to the configuration described in the first embodiment (FIG. 1). The monitor 120 is connected to the camera 101 with an HDMI (registered trademark) cable or the like, and displays images taken from the camera 101 in real time. Further, in the image processing device 102 of the second embodiment, the camera control unit 104 and the image superimposing unit 110 are omitted as compared with the first embodiment.

FIG. 8 is a flowchart illustrating the camera posture adjustment process performed by the posture adjustment system (image processing device 102) according to the second embodiment. Steps S201 to S206 and steps S209 are the same processes as in the first embodiment (FIG. 2).

In step S307, the target image correction unit 109 displays the target image corrected in step S205 on the display 111. If the image processing device 102 is a portable device such as a laptop computer or a tablet, the user can bring the image processing device 102 to the installation position of the camera 101. Therefore, the user can adjust the posture of the camera 101 while comparing the captured image of the camera 101 displayed on the monitor 120 with the target image displayed on the display 111. If the image processing device 102 is not portable, the corrected target image may be printed on paper or the like by a printing device (not shown), and the printed matter may be brought to the installation position of the camera 101.

In step S308, the posture of the target image displayed on the display 111 (or printed on paper or the like) and the captured image of the camera 101 displayed on the monitor 120 are adjusted. The image taken from the camera is displayed on the monitor 120 in real time, and when the posture of the camera changes, the image displayed on the monitor 120 also changes according to the change. The user can adjust the posture of the camera 101 while comparing the target image displayed on the display 111 (or the target image printed on paper or the like) with the captured image displayed on the monitor 120 in step S307. The user adjusts the posture of the camera 101 until the target image and the captured image displayed on the monitor 120 become equal to each other, thereby completing the posture adjustment of the camera 101.

In the configuration of the present embodiment, when adjusting the posture of the camera 101, the user sets the focal length of the lens of the camera 101 to the adjusted focal length, but the present invention is not limited to this. By adding the camera control unit 104 to the configuration shown in FIG. 7, the focal length of the camera 101 may be set to the adjusted focal length under the control of the image processing device 102.

Further, in the first embodiment and the second embodiment, the target image correction unit 109 obtains a corrected target image by shifting the target image stored in the target image storage unit 106 based on the optical axis shift amount. However, it is not limited to this. For example, a target image in a range larger than the shooting range of the camera 101 is stored in the target image storage unit 106, and the target image correction unit 109 sets the range of the same size as the shooting range of the camera 101 and the optical axis shift amount. In consideration, it may be cut out from the target image.

As described above, according to the second embodiment, the focal length is changed in the posture adjustment using the image taken by the camera 101 at the adjustment focal length different from the shooting focal length as in the first embodiment. It is possible to adjust the posture while reducing the influence of the optical axis shift caused by. Therefore, even if the posture of the camera is adjusted by the focal length at the time of adjustment and then changed to the focal length at the time of shooting, it is possible to obtain a shot image with the intended camera posture (a shot image in the intended shooting range).

<Third Embodiment>
In the first embodiment and the second embodiment, the target image is moved based on the optical axis shift amount to correct the deviation of the optical axis position. In the third embodiment, a superimposed display for posture adjustment is provided by shifting and superimposing a layer of a target image or a layer of a captured image based on an optical axis shift amount.

FIG. 9 is a block diagram showing a configuration example of the posture adjustment system according to the third embodiment. The difference from the configuration of the first embodiment (FIG. 1) is that the target image correction unit 109 is omitted, and the optical axis shift amount and the target image (uncorrected target image) from the correction value calculation unit 108 and the target image storage unit 106. ) Is sent to the image superimposing unit 110. The image superimposing unit 110 superimposes the captured image from the camera 101 and the target image from the target image storage unit 106 based on the optical axis shift amount calculated by the correction value calculation unit 108. Further, the posture adjustment process according to the third embodiment is a flow in which step S205 is omitted from the flowchart of FIG. Then, in step S207, the superimposition processing described below (superimposition processing in consideration of the optical axis shift amount) is performed.

FIG. 10 is a diagram illustrating the superimposition processing of the target image and the captured image by the image superimposing unit 110 of the third embodiment. FIG. 10A shows a layer for displaying the captured image 61, and FIG. 10B shows a layer for displaying the target image 62a. In the superimposed display, the target image 62a is displayed semi-transparently. In FIG. 10, it is shown by a broken line to distinguish it from the captured image 61. In the third embodiment, since the target image correction unit 109 is omitted, the target image 62a in FIG. 10B can be corrected based on the target image (correction based on the optical axis shift amount) stored in the target image storage unit 106. It is a target image before being applied), and corresponds to the target image shown in FIG. 5 (a).

FIG. 10C is a diagram showing a state in which a layer for displaying the captured image 61 and a layer for displaying the target image 62a are superimposed. When superimposing these two layers, the image superimposing unit 110 shifts the position of the target image 62a on the layer by the correction value (δx, δy) based on the optical axis shift amount calculated by the correction value calculation unit 108. It is superimposed on the captured image 61. Similar to the first embodiment, the user can adjust the posture of the camera 101 using the superimposed image displayed on the display 111 to adjust the posture corresponding to the deviation of the optical axis position due to the change of the focal length. .. The image of the display after adjusting the camera posture is shown in FIG. 10 (d). The range displayed as the superimposed image may be, for example, the range of the captured image 61. In this case, in FIGS. 10 (c) and 10 (d), the portions protruding from the captured image 61 on the left side and the lower side of the target image 62a are not displayed as superimposed images.

As described above, according to the third embodiment, the posture of the camera is adjusted without creating a new target image in which the deviation of the optical axis is corrected with respect to the target image stored in the memory by the target image storage unit 106. It will be possible to adjust. In the third embodiment, only the layer displaying the target image is moved, but the present invention is not limited to this. For example, the position of the layer displaying the captured image 61 may be shifted by the optical axis shift amount calculated by the correction value calculation unit 108, that is, shifted by (−δx, −δy) and superimposed on the target image. Alternatively, both the layer displaying the target image and the layer displaying the captured image may be shifted in opposite directions. That is, the captured image at the time of adjustment and the target image may be superimposed and displayed with a relative shift by the amount of the optical axis shift.

In each of the above embodiments, the case of "focal length at the time of shooting> focal length at the time of adjustment" has been described, but it goes without saying that the present disclosure can be applied to the case of "focal length at the time of shooting <focal length at the time of adjustment". The case of "focal length at the time of shooting <focal length at the time of adjustment" is, for example, a case where the marker is photographed larger at the time of posture adjustment and the posture is adjusted more accurately.

As described above, according to each of the above embodiments, it is possible to reduce the influence on the posture adjustment caused by the individual difference of the lens and the variation of the optical axis due to the change of the focal length. Therefore, even if the focal length of the camera is changed to the focal length at the time of use after adjusting the posture of the camera using the target image at the focal length at the time of adjustment, the intended shooting range can be taken. As a result, for example, in the posture adjustment of the camera performed by setting the focal length at the time of adjustment shorter than the focal length at the time of use in order to include a characteristic part in the posture adjustment of the camera in the shooting range, the influence of the optical axis shift is exerted. Reduced and more accurate adjustments are achieved. Alternatively, the effect of the optical axis shift is reduced in the posture adjustment of the camera performed by setting the focal length at the time of adjustment longer than the focal length at the time of use so that the characteristic part can be obtained as a larger image on the captured image. More accurate adjustment can be achieved.

<Fourth Embodiment>
The photographing apparatus according to the first to third embodiments can be applied to a system for generating a virtual viewpoint image, a system for generating three-dimensional shape data of an object, or a surveillance camera system.

For example, in a system that generates a virtual viewpoint image, a plurality of photographing devices according to the above-described embodiment are arranged so as to surround the photographing space. Then, this system generates a virtual viewpoint image corresponding to a designated virtual viewpoint based on a plurality of images obtained by taking pictures with a plurality of cameras 101. In such a system, in addition to the above-mentioned photographing device, a generation device for generating a virtual viewpoint image is provided. Even if the focal length used for shooting when adjusting the posture of the shooting device is different from the focal length of the shooting device when acquiring the image used to generate the virtual viewpoint image, the above-mentioned image processing device affects the posture adjustment. Can be reduced.

Alternatively, in the system that generates the three-dimensional shape data of the object, as in the system that generates the virtual viewpoint image, a plurality of photographing devices in the above-described embodiment are arranged so as to surround the photographing space. .. Then, this system generates three-dimensional shape data of an object existing in the shooting space based on a plurality of images obtained by shooting with a plurality of shooting devices. In such a system, in addition to the above-mentioned imaging device structure, a generation device for generating three-dimensional shape data is provided. Even if the focal length used for shooting when adjusting the posture of the shooting device and the focal length of the shooting device when acquiring the image used to generate 3D shape data are different, the above-mentioned image processing device can be used for posture adjustment. The impact can be reduced. The system for generating the virtual viewpoint image may include a system for generating the three-dimensional shape data. In this case, for example, a virtual viewpoint image may be generated based on the three-dimensional shape data of the object.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to publicize the scope of the present invention, the following claims are attached.

101: Camera, 102: Image processing device, 103: User interface, 104: Camera setting value acquisition unit, 105: Lens setting value acquisition unit, 106: Target image storage unit, 107: Look-up table, 108: Correction value calculation unit , 109: Target image correction unit, 110: Image superimposition unit, 111: Display

Claims (15)

A first acquisition means for acquiring a first focal length used for photographing when adjusting the posture of the photographing device and a second focal length used for photographing when using the photographing device.
A second acquisition means for acquiring an optical axis shift amount representing the difference between the optical axis position at the time of shooting at the first focal length and the optical axis position at the time of shooting at the second focal length of the photographing device. When,
Imaging taken by the imaging device in the adjusted posture and the first focal length, generated based on a three-dimensional model of the adjusted posture, the first focal length, and the imaging space of the imaging device. A storage means for storing an image corresponding to an image as a target image,
An image processing apparatus comprising: a generation means for generating a target image corrected by shifting the target image based on the optical axis shift amount. The image according to claim 1, further comprising a superimposing means for superimposing the corrected target image and the captured image captured by the photographing apparatus at the first focal length and outputting the image to a display. Processing device. The image processing apparatus according to claim 1, further comprising an output means for displaying the corrected target image on a display or printing the corrected target image on a printing apparatus. A first acquisition means for acquiring a first focal length used when adjusting the posture of the photographing device and a second focal length used for photographing when the photographing device is used, and a first acquisition means.
A second acquisition means for acquiring an optical axis shift amount representing the difference between the optical axis position at the time of shooting at the first focal length and the optical axis position at the time of shooting at the second focal length of the photographing device. When,
Imaging taken by the imaging device in the adjusted posture and the first focal length, generated based on a three-dimensional model of the adjusted posture, the first focal length, and the imaging space of the imaging device. A storage means for storing an image corresponding to an image as a target image,
In order to adjust the posture of the imaging device, the image captured by the imaging device at the first focal length and the target image at the time of adjustment are displayed on the display with the optical axis shift amount relatively shifted. An image processing device comprising: a superimposing means for superimposing and displaying. The image processing apparatus according to claim 4, wherein the superimposing means superimposes the layer for displaying the target image on the layer for displaying the captured image at the time of adjustment. The image processing apparatus according to claim 5, wherein the superimposing means superimposes the layer for displaying the captured image at the time of adjustment by shifting the layer for displaying the target image. The image processing apparatus according to claim 5, wherein the superimposing means superimposes both a layer for displaying the captured image at the time of adjustment and a layer for displaying the target image by shifting them in opposite directions. The image processing apparatus according to any one of claims 1 to 7, wherein the first acquisition means receives inputs of the first focal length and the second focal length by a user. Further having a look-up table in which the optical axis positions at each of the first focal length and the plurality of focal lengths including the second focal length are recorded.
The image processing apparatus according to any one of claims 1 to 8, wherein the second acquisition means acquires the optical axis shift amount by referring to the look-up table. The image processing apparatus according to claim 9, wherein the look-up table is prepared for each lens that can be attached to the photographing apparatus. The image processing according to claim 9 or 10, wherein the look-up table records the amount of deviation of the optical axis positions of the plurality of focal lengths from the optical axis positions at a predetermined focal length. Device. The image processing apparatus according to any one of claims 9 to 11, wherein the plurality of focal lengths are a plurality of focal lengths that can be set by the photographing apparatus. A first acquisition step of acquiring a first focal length used for photographing when adjusting the posture of the photographing device and a second focal length used for photographing when using the photographing device, and a first acquisition step.
A second acquisition step of acquiring an optical axis shift amount representing the difference between the optical axis position at the time of shooting at the first focal length and the optical axis position at the time of shooting at the second focal length of the photographing device. When,
Imaging taken by the imaging device in the adjusted posture and the first focal length, generated based on a three-dimensional model of the adjusted posture, the first focal length, and the imaging space of the imaging device. A storage process in which an image corresponding to an image is stored in a storage unit as a target image,
A posture adjusting method comprising: a generation step of generating a corrected target image by shifting the target image based on the optical axis shift amount. The first acquisition step of acquiring the first focal length used when adjusting the posture of the photographing device and the second focal length used for photographing when the photographing device is used, and the first acquisition step.
A second acquisition step of acquiring an optical axis shift amount representing the difference between the optical axis position at the time of shooting at the first focal length and the optical axis position at the time of shooting at the second focal length of the photographing device. When,
Imaging taken by the imaging device in the adjusted posture and the first focal length, generated based on a three-dimensional model of the adjusted posture, the first focal length, and the imaging space of the imaging device. A storage process in which an image corresponding to an image is stored in a storage unit as a target image,
In order to adjust the posture of the imaging device, the image captured by the imaging device at the first focal length and the target image at the time of adjustment are displayed on the display with the optical axis shift amount relatively shifted. A posture adjustment method characterized by having a superimposition step of superimposing and displaying. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images