JP6450604B2 - Image acquisition apparatus and image acquisition method - Google Patents

Description

The present invention relates to an image acquisition device and an image acquisition method .

  In a digital camera, light coming from a subject is converged by a photographing lens. As a result, a subject image is formed on the imaging surface of the image sensor. The imaging element generates image data based on the subject image formed on the imaging surface. This image data is subjected to various image processing for recording and display.

  In general, a fisheye lens forms an image of a subject covering the entire hemisphere having an extent of about 180 ° in one circular image circle. There are a circumferential fisheye and a diagonal fisheye in the positional relationship between the image circle and the imaging surface. For example, the lens barrel described in Patent Document 1 is made to be a diagonal fisheye with respect to an APS-C, APS-H, or full-size image sensor by rotating the zoom operation ring of the fisheye zoom lens. And a circular fisheye for a full-size image sensor can be easily switched.

JP2012-22108A

  For example, when a lens having a wide angle of view, such as a fisheye lens, is used, images of many subjects can be acquired at once. A new image can be created by cutting out and rearranging a necessary image from the image thus obtained.

It is an object of the present invention to provide an image acquisition apparatus and an image acquisition method capable of creating a new image by cutting out a plurality of images from one image, arranging their orientations, and rearranging them. .

  According to an aspect of the present invention, an image acquisition device captures a first subject and a second subject located around the first subject at the same time, and generates a captured image; A trimming unit that cuts out a first image including the first subject and a second image including the second subject from the captured image, and a direction determination for determining an image direction that is an orientation of the first image An image that creates an edited image by composing the first image and the second image by aligning the orientation of the first image and the second image based on the image direction and the first image And a synthesis unit.

  According to one aspect of the present invention, an image acquisition method includes simultaneously capturing a first subject and a second subject located around the first subject to generate a captured image; and Cutting out a first image including the first subject and a second image including the second subject from the image, determining an image direction which is an orientation of the first image, Adjusting an orientation of the first image and the second image based on an image direction, and synthesizing the first image and the second image to create an edited image.

According to the present invention, it is possible to provide an image acquisition device and an image acquisition method capable of creating a new image by cutting out a plurality of images from one image, arranging their orientations, and rearranging them.

1 is a block diagram illustrating an outline of a configuration example of an imaging apparatus according to an embodiment. FIG. 10 is a diagram for describing an example of a usage mode of an imaging device. FIG. 10 is a diagram for describing an example of a usage mode of an imaging device. FIG. 10 is a diagram for describing an example of a usage mode of an imaging device. The figure which shows an example of the picked-up image acquired when an imaging device is used as a document camera. The figure which shows an example of the edit image produced based on a picked-up image. It is a flowchart for demonstrating an example of operation | movement of an imaging device. The figure for demonstrating an example of how to determine the internal diameter of a peripheral region. The figure for demonstrating an example of how to determine the internal diameter of a peripheral region. The figure for demonstrating the outer diameter and internal diameter of a peripheral region. The figure for demonstrating producing a peripheral image from the image of a peripheral region. The figure for demonstrating producing a peripheral image from the image of a peripheral region. The figure for demonstrating a periphery image. The figure for demonstrating a center image.

  An imaging apparatus as an image acquisition apparatus according to an embodiment of the present invention will be described. FIG. 1 is a diagram illustrating an outline of a configuration example of an imaging apparatus 1 as an image acquisition apparatus according to the present embodiment. The imaging device 1 includes a control unit 2, an imaging unit 4, a display unit 5, a touch panel 6, a recording unit 7, an operation unit 8, an attitude detection unit 9, a communication unit 10, and a temporary storage unit 11. And a microphone 12.

  The imaging unit 4 includes a fisheye optical system 41 and an imaging element 42. The fish-eye optical system 41 has a plurality of lenses, converges a light beam incident from a subject, and forms a subject image on the imaging surface of the image sensor 42. The fish-eye optical system 41 is a circumferential fish-eye lens that can capture a subject having an angle of view of 180 ° or more on the imaging surface of the imaging device 42. The fisheye optical system 41 preferably has an angle of view greater than 180 °. The imaging element 42 includes, for example, a CCD or CMOS image sensor, and generates analog image data related to the subject image formed on the imaging surface. The imaging unit 4 includes an AD conversion circuit, and converts analog image data generated by the imaging element 42 into captured image data that is digital image data. Hereinafter, an image acquired by the imaging unit 4 will be referred to as a captured image.

  The display unit 5 includes, for example, a liquid crystal display (LCD) or an organic EL display. The display unit 5 displays a captured image on the display surface based on, for example, captured image data. Further, the display unit 5 may display information related to the captured image, information related to the state and operation of the imaging device 1, and the like. The display unit 5 may have a plurality of display surfaces.

  The touch panel 6 is provided to overlap the display surface of the display unit 5. The touch panel 6 detects position information on the panel touched by the user with a fingertip or a pen tip. The detection method of the touch panel 6 may be any detection method among a resistive film method, a capacitance method, an optical method, an electromagnetic induction method, and the like.

  The recording unit 7 is a recording medium in which contents are stored even when the power is turned off, for example, a FLASH memory. The recording unit 7 records various information such as a program for operating the imaging apparatus 1, captured image data, various parameters such as an enlargement rate described later, and an edited image created from the captured image described later. Note that a detachable recording unit may be provided separately from the recording unit 7, and the data of the captured image and the edited image may be recorded in the detachable recording unit. The temporary storage unit 11 temporarily stores information such as a captured image, an image obtained by performing predetermined image processing on the image, a calculation result by the control unit 2, and the like.

  The operation unit 8 includes, for example, various buttons, a dial, a slider, and the like. The operation unit 8 includes, for example, a release button and a mode dial. When an operation by the user is performed on the operation unit 8, this operation is detected by the control unit 2.

  The posture detection unit 9 includes an acceleration sensor. The posture detection unit 9 detects an acceleration applied to the imaging device 1 that is used to calculate the posture of the imaging device 1. The posture detection unit 9 transmits the detected acceleration information to the control unit 2. This posture detection unit 9 can determine the orientation and angle of the device using gravitational acceleration and the like, and can determine the shooting direction with a geomagnetic sensor or the like, but when the device is directed directly above or below, It is difficult to detect the top and bottom of what is photographed in the center. The present invention is effective in the case of correctly displaying and recording an object that must be determined up and down separately from the posture of the apparatus.

  The communication unit 10 transmits / receives data to / from an external device by wired connection or wireless connection with the external device. As a communication method, various generally known methods can be adopted.

  The microphone 12 converts sound into an electrical signal. The imaging device 1 preferably includes a plurality of microphones 12 so that a sound source can be specified based on the direction of arrival of sound.

  A magnet or a hook is provided on the exterior portion of the imaging apparatus 1. The imaging device 1 can be attached to a stand or other equipment by a magnet or a hook.

  The control unit 2 includes a central processing unit (CPU) or an application specific integrated circuit (ASIC). The control unit 2 includes an image processing unit 21, a state determination unit 22, a face detection unit 23, a sound source identification unit 24, a trimming unit 25, a distortion correction unit 26, a direction determination unit 27, and an image composition unit 28. Including. The image processing unit 21, the state determination unit 22, the face detection unit 23, the sound source identification unit 24, the trimming unit 25, the distortion correction unit 26, the direction determination unit 27, and the image composition unit 28 are each 1 It may be realized by one circuit, or a plurality of them may be realized by one circuit.

  The image processing unit 21 performs demosaic processing on captured image data, image processing for adjusting various shooting parameters such as noise, white balance, gradation, and color related to the captured image data.

  The state determination unit 22 determines the posture state of the imaging device 1 based on the acceleration information acquired from the posture detection unit 9.

  The face detection unit 23 detects a face included in the captured image. For face detection, for example, pattern matching based on facial features such as eyes, nose or mouth is used. The face detection unit 23 acquires information related to the face position, face orientation, and the like. The face detection unit 23 may detect a subject other than the face. For example, the face detection unit 23 may detect characters by performing pattern matching based on character characteristics on the captured image. The face detection unit 23 acquires information related to the direction of characters.

  The sound source specifying unit 24 analyzes sound data acquired by a plurality of microphones and specifies the direction of the sound source. That is, the sound source specifying unit 24 specifies a sound source that is generating sound among the second subject based on the sound information acquired by the microphone.

  The trimming unit 25 cuts out a plurality of images from the photographed image based on the subject information specified by the face detection unit 23 and other information. In the present embodiment, a subject located near the center of the image (referred to as a first subject) and a subject located around the subject (referred to as a second subject) are photographed simultaneously. The acquired image is acquired as a captured image. The trimming unit 25 cuts out an image including the first subject (referred to as the first image) and an image including the second subject (referred to as the second image).

  The distortion correction unit 26 performs distortion correction for correcting distortion for the images (first image and second image) cut out by the trimming unit 25. Since the imaging apparatus 1 uses the fisheye optical system 41, it is necessary to perform distortion correction on the captured image in order to obtain an image without distortion.

  The direction determination unit 27 uses the posture of the imaging device 1 acquired from the state determination unit 22 for the images (first image and second image) cut out by the trimming unit 25 and subjected to distortion correction by the distortion correction unit 26. Based on the information and the information on the subject acquired from the face detection unit 23, the image direction such as the vertical direction of the image is specified, and the vertical information is added to the image.

  The image synthesis unit 28 creates a new edited image by synthesizing the cut out first image and second image while referring to the vertical information.

  A usage example of the imaging apparatus 1 will be described with reference to FIGS. 2A to 2C. The user U can freely determine the direction of the optical axis of the fish-eye optical system 41 of the imaging device 1. FIG. 2A illustrates an example of a case where the user U performs shooting while holding the imaging device 1. In the case illustrated in FIG. 2A, the user U holds the imaging device 1 so that the optical axis of the fisheye optical system 41 is in the horizontal direction. Thus, the imaging device 1 can be used in the same manner as a general camera provided with a fisheye lens.

  FIG. 2B shows another example in which the user U performs shooting while holding the imaging device 1. In this case, the user U is shooting in a state where the direction of the optical axis of the fish-eye optical system 41 of the imaging device 1 is directed to the zenith. At this time, an image over the entire circumference in the horizontal direction can be acquired by the imaging device 1.

  FIG. 2C shows an example when the imaging apparatus 1 is used as a document camera. In FIG. 2C, a stand S2 is attached to the desk S1. The imaging device 1 is attached to the stand S2 with a magnet or a hook. The imaging device 1 is fixed to the stand S2 such that the fisheye optical system 41 is vertically downward. In the situation shown in FIG. 2C, a photograph O1 is placed on the desk S1. In addition, there are a plurality of persons O2, O3, and O4 around the desk S1. When the angle of view of the fish-eye optical system included in the imaging device 1 is larger than 180 °, the height at which the imaging device 1 is arranged is, for example, near the height of the eyes of the persons O2, O3, and O4.

  The imaging device 1 performs imaging with the angle of view of the circumferential fisheye in the above state. An example of the captured image obtained at this time is schematically shown in FIG. 3A. Since it is a circumferential fisheye, the obtained captured image I1 is circular, and an image with an angle of view according to the design of the optical system of the fisheye optical system 41 is obtained. Further, the periphery of the circular captured image I1 is in a so-called discolored state. As shown in FIG. 3A, a photograph O1 placed on the desk S1 is shown near the center of the photographed image I1. Further, a plurality of persons O2, O3, and O4 around the desk S1 are shown around the photograph O1. The images obtained in this way can capture photos and documents (documents) on the desk and the state of the person who appreciates them, using only one camera. Therefore, by acquiring this image at a remote place, it is possible to know the state of a meeting, party, etc., and participate in it by voice or the like. Many TV conference systems focus on capturing the faces of participants, and do not envisage applications where everyone talks about such materials.

  The imaging apparatus 1 according to the present embodiment generates an edited image I2 as shown in FIG. 3B from the captured image I1 as shown in FIG. 3A. The edited image I2 represents a central image C2 that is an image of the photograph O1 placed on the desk S1 and a peripheral image P2 that is an image of the persons O2, O3, and O4 who view the photograph O1. . Here, the center image C2 is an image obtained by cutting out only the portion of the photograph O1 in the center region C1 of the captured image I1 shown in FIG. 3A and correcting the distortion. Further, the peripheral image P2 is such that the annular peripheral region P1 of the captured image I1 shown in FIG. 3A is deformed into a rectangle. Further, the center image C2 and the peripheral image P2 are aligned in the vertical direction of the subject.

  Hereinafter, an example of the operation of the imaging apparatus 1 according to the present embodiment will be described. The operations described here are mainly operations when used as the document camera shown in FIG. 2C. FIG. 4 is a flowchart showing the operation of the imaging apparatus 1 of the present embodiment. For example, when the power is turned on, the control unit 2 reads the program recorded in the recording unit 7 and starts the following camera control process.

  In step S101, the control unit 2 determines whether the shooting mode is selected. If it is determined in step S101 that the shooting mode is selected, the process proceeds to step S102.

  In step S102, the control unit 2 causes the imaging unit 4 to start imaging. Using the imaging unit 4, a captured image I1 having a circumferential fisheye angle of view is generated.

  In step S <b> 103, the control unit 2 acquires acceleration information from the posture detection unit 9. The control unit 2 determines the posture state of the imaging device 1 based on the acceleration information.

  In step S104, the control unit 2 determines whether or not the optical axis of the fisheye optical system 41 is oriented in the vertical direction in order to determine the shooting direction. When it faces in the vertical direction, the process proceeds to step S105. Here, the determination as to whether or not it is oriented in the vertical direction is not limited to the strict vertical direction, but may be a determination as to whether or not it is substantially oriented in the vertical direction, and the same applies to the following.

  In step S105, the control unit 2 determines whether or not there is a person in the peripheral portion of the captured image I1. If there is no person, the process proceeds to step S107. On the other hand, when there is a person, the process proceeds to step S106.

  In step S106, the control unit 2 creates the peripheral image P2 and adds the vertical information. A method for creating the peripheral image P2 will be described. The peripheral image P2 is created based on an image of a region cut out in an annular shape from the captured image I1. Here, what is the value of the width of the ring that is the area to be cut out becomes a problem.

  As an example, in the present embodiment, the width of the ring to be cut out is determined as follows. That is, when the radius of the outer periphery of the cut out ring is R0, the radius R0 is the radius of the image circle obtained by the image sensor 42. On the other hand, when the radius of the inner periphery of the cut out ring is R1, the reference of R1 is half of R0 (R0 = 2 × R1). From this reference, the value of R1 is finely adjusted. Specifically, a person appearing in the peripheral portion of the photographed image I1 is specified using the result of face detection, and R1 is adjusted to a value such that a bust of this person appears in the peripheral image P2.

  5A and 5B are schematic diagrams for explaining an example of setting the width of the ring. FIG. 5A shows a case where the person O5 shown in the peripheral part of the photographed image I1 is small, and FIG. 5B shows a case where the person O5 shown in the peripheral part of the photographed image I1 is large. The image circle is divided into an annular peripheral region P1 and a circular central region C1. As shown in FIG. 5A, when the size of the person O5 is relatively small, the inner diameter R1 of the peripheral region P1 is adjusted to be relatively large. On the other hand, as shown in FIG. 5B, when the size of the person O5 is relatively large, the inner diameter R1 of the peripheral region P1 is adjusted to be relatively small. A peripheral image P2 is created based on the image of the peripheral area P1 adjusted in this way, and a central image C2 is created based on the image of the central area C1. Here, the peripheral area P1 and the central area C1 have been described as not overlapping, but the peripheral area P1 and the central area C1 may overlap in part.

  It should be noted that the fact that the above R1 reference is half of R0 has the following meaning. That is, when the angle of view of the fish-eye optical system 41 is 180 °, the peripheral region P1 is selected in the range of 45 ° downward from the horizontal plane as shown in FIG.

  Next, a method for creating the peripheral image P2 from the image of the peripheral region P1 in the captured image I1 will be briefly described. As shown in FIG. 7A, a cutting line is set in the radial direction at one location in the peripheral region P1. For example, the coordinates of each point in the peripheral area P1 are converted into the coordinates of each point in a rectangle having the inner and outer sides as opposed sides and the cutting line as the other opposed side. Perform conversion and so on.

  The rectangle obtained here is, for example, as shown in FIG. 7B, a rectangle whose side corresponding to the outer periphery and inner periphery of the peripheral region P1 is 2πR0. The length of this side is also expressed as 2πR1 × k using the inner diameter R1 and the variable k. Here, the relationship of R0 = k × R1 holds for the variable k. This variable k will be referred to as the stretching rate. If the stretch rate k is too large, the deformation from the captured image I1 to the peripheral image P2 becomes large, which is not preferable. For this reason, in the present embodiment, as described above, the reference value of the stretching rate k is set to 2, and the stretching rate k is adjusted around 2.

  The cutting line is preferably an area where no person is shown. If the cutting line and the person overlap, the person is cut in the created peripheral image P2. Therefore, the setting of the cutting line may be based on the position of the detected person. Further, for example, a portion where the stand S2 is shown may be set as a cutting line. Further, since a person rarely sits at the corner of the desk, a cutting line may be set according to the corner of the desk.

  Returning to FIG. 4, the description of the camera control process will be continued. In step S106, the control unit 2 creates the peripheral image P2 based on the captured image I1 as described above. Thus, the peripheral image P2 corresponds to a second image including the second subject located around the first subject.

  Further, in step S106, the control unit 2 adds the vertical information to the peripheral image P2. The vertical information is information for specifying the vertical direction of the created rectangular peripheral image P2. The upper and lower sides of the peripheral image P2 are specified as follows, for example. For example, for a person included in the peripheral image P2, since face detection is performed and a human image is specified, the top and bottom of the peripheral image P2 can be specified based on these pieces of information. Further, the upper and lower sides may be specified based on the posture of the imaging device 1. That is, as shown in FIG. 2C, when the shooting direction is vertically downward, the upward direction of the peripheral image P2 is the outer peripheral side of the captured image I1. After step S106, the process proceeds to step S107.

  In step S <b> 107, the control unit 2 determines whether or not the shooting direction specified by the posture of the imaging device 1 is vertically downward. If not vertically downward, the process proceeds to step S113. On the other hand, when it is vertically downward, the process proceeds to step S108. That is, as shown in FIG. 2C, when the imaging device 1 is used as a document camera, the process proceeds to step S108.

  In step S108, the control unit 2 determines whether or not there is an object to be a subject in the central area C1. Since the imaging apparatus 1 is used as a document camera, it is assumed that a document or drawing including characters, figures, symbols, photographs, and the like, and subjects such as photographs and documents are arranged in the central area C1. This is based on the reason that it is more natural to shoot from top to bottom because documents are generally placed on a desk or the like (using gravity). A document camera is usually designed for such an environment. When the lens of the document camera is widened as in the present embodiment, when shooting from the top to the bottom as described above, there is a merit that the object in the peripheral portion can be shot. The peripheral object is an image in which a target object standing in the direction of gravity extends radially from the center of the screen toward the periphery in the same manner as a document or the like. Therefore, by performing the same simple correction process around the optical axis corresponding to the image, the obtained image can be converted into a natural image. Of course, when shooting a slogan or the like affixed to the ceiling, the relationship is reversed. When shooting a poster or a projected document that is held on a wall against gravity, this is different from this way of thinking. However, if only a poster is shot, it may be arranged so that it can be shot. In these cases, the up / down determination can be performed by using the posture determination of the camera. On the other hand, if a technique for reading characters is used, the upper and lower characters of a character on a desk or a ceiling are determined, so that the pattern can be determined by database collation or the like to determine the vertical relationship between documents and characters. Of course, even when taking a photograph or the like as shown in FIG. 2C, it is possible to determine whether the person is in the upper or lower position, where the sky is, or how the shadow appears. When there is no object in the central area C1, the process proceeds to step S113. On the other hand, when there is an object in the central area C1, the process proceeds to step S109.

  In step S109, the control unit 2 extracts an image of the object included in the central region C1 of the captured image I1, and corrects the distortion to create the central image C2. Thus, the central image C2 corresponds to a first image including the first subject. In the central image and the peripheral image, the problem is where to make the boundary. However, in the document camera, the description is simple, and the border of the document or photograph or the desk on which the document is placed may be used as the boundary. In many cases, the edge of the document or photo or the desk edge on which it is placed has a different color tone or brightness from the other parts, and it is a quadrilateral, so it can be separated by assuming it. Further, it is possible to cut out from each part in a cut paper style without strictly setting the boundary.

  In step S110, it is determined whether or not the vertical direction of the center image C2 can be determined. For example, when an image of a person or character is included in the central image C2, the person or character can be detected, and the vertical direction of the central image C2 can be determined based on the direction of the person or character. When the vertical direction of the center image C2 cannot be determined, the process proceeds to step S112. On the other hand, when the vertical direction of the center image C2 can be determined, the process proceeds to step S111.

  In step S111, the control unit 2 adds up / down information to the data of the central image C2. Thereafter, the process proceeds to step S112.

  In such an apparatus, there are cases where it is preferable to perform processing according to necessity for each area and to express it, rather than collectively processing what is photographed. Here, the processing is performed by dividing the central object, which is difficult to determine in the device orientation, up and down, left and right, and the peripheral object, which can be determined in the device orientation, etc. In particular, when the angle of view is wide, various images appear in the screen, resulting in an image that is different from human vision. Therefore, it is preferable for the user interface to display the images in an easy-to-understand manner. In step S112, the control unit 2 combines the peripheral image P2 and the central image C2, and creates a single edited image I2 including them. That is, it is assumed that a photographed image I1 as shown in FIG. 3A is obtained by photographing in the situation shown in FIG. 2C. At this time, the control unit 2 divides the captured image I1 into a peripheral region P1 and a central region C1. The control unit 2 creates a peripheral image P2 as shown in FIG. 8A from the peripheral region P1, and creates a central image C2 as shown in FIG. 8B from the central region C1. Then, the peripheral image P2 and the central image C2 are synthesized with the top and bottom aligned to create an edited image I2 as shown in FIG. 3B. Thereafter, the process proceeds to step S113.

  In step S113, the control unit 2 displays the edited image I2 on the display unit 5 or a display device provided outside the imaging device 1. This display device is a display device provided so that, for example, persons O2, O3, and O4 surrounding the photograph O1 can view the display image. Persons O2, O3, and O4 can view the edited image I2 using a display device. Further, instead of displaying the edited image I2 or displaying the edited image I2, the edited image I2 may be recorded in the recording unit 7.

  In step S114, the control unit 2 determines whether or not a release operation performed by the user has been detected. When the release operation is not detected, the process proceeds to step S116. On the other hand, when the release operation is detected, the process proceeds to step S115.

  In step S115, the control unit 2 records the data of the edited image I2 generated as described above in the recording unit 7. Thereafter, the process proceeds to step S116.

  In step S116, the control unit 2 determines whether or not to end the shooting mode. If the shooting mode is not terminated, the process returns to step S103. On the other hand, when ending the shooting mode, the process proceeds to step S117.

  In step S117, the control unit 2 determines whether or not to turn off the power of the imaging apparatus 1. When the power is not turned off, the process returns to step S101. On the other hand, when turning off, the control unit 2 turns off the power and ends the camera control process.

  If it is determined in step S104 that the optical axis of the imaging device 1 is not in the vertical direction, the process proceeds to step S118. In step S118, the top and bottom of the photographed image I1 obtained by photographing is determined. The top and bottom of the captured image I1 is determined based on the orientation of the imaging device 1, the orientation of a person included in the captured image I1, and the like. In step S119, the control unit 2 displays the captured image I1 on the display unit 5 in consideration of the top and bottom of the captured image I1.

  In step S120, the control unit 2 determines whether or not a release operation performed by the user has been detected. When the release operation is not detected, the process returns to step S103. On the other hand, when the release operation is detected, the process proceeds to step S121. In step S <b> 121, the control unit 2 performs a shooting process and records data of the captured image I <b> 1 in the recording unit 7. Thereafter, the process proceeds to step S116.

  If it is determined in step S101 that the shooting mode is not selected, the process proceeds to step S122. In step S122, the control unit 2 executes processing in the reproduction mode. In the reproduction mode, the image recorded in the recording unit 7 is displayed. Here, the description of the operation in the reproduction mode is omitted. After the operation in the reproduction mode, the process proceeds to step S117, and it is determined whether or not the power is off.

  As described above, according to the present embodiment, the imaging device 1 can be used not only as a camera including a general fisheye lens but also as a document camera. When the imaging device 1 is used as a document camera, in addition to an object placed in front of the imaging device 1, for example, an image of a person surrounding the object is also acquired, and the image of the object and the image of the person are obtained. A combined edited image is created. If such an imaging device 1 is used in a conference, for example, the state of participants in the conference can be displayed on a display device or recorded in a recording device together with conference materials. This is particularly convenient in so-called video conferencing, which takes place via remote image and audio communication. Here, an example has been shown in which it is determined whether the optical axis is in the vertical direction and the upper and lower representations are not known, but there are other image parts that need to be determined from other than the posture of the device. In some cases, other methods may be used. When shooting underwater while shooting from an unreasonable posture by the user, or by moving the shooting unit and optical unit by remote control, the photographer's attitude may differ from what is normally assumed. is there. In the case of a document camera, the document and desk boundaries can be used as a guide for segmentation of the image portion, so the explanation is simple. However, the present invention is not limited to this. For example, a fish that is photographed while diving has a purpose of displaying an image of a fellow who dives in a vertical direction based on the posture of the device while enlarging the image in the shape of a fish. In this case, a method can be considered in which the target area is cut out with a margin so that the target area is not cut from the shape of the fish. In other words, in this embodiment, a first image area where the first subject exists and a second area including a second subject different from the first subject are simultaneously photographed to generate a photographed image. An imaging unit, and a trimming unit that determines and cuts out the first image region and the second image region from the captured image, and determines the first image region and the second image region, respectively, up and down The apparatus which has a direction determination part which performs image processing and determines an image direction is provided. An image composition unit that prepares an edited image by adjusting the orientation of the first image and the second image based on the direction of the image and compositing the first image and the second image However, the vertical orientation of one image is determined by the orientation of the device, and the vertical orientation of one image is determined from the top and bottom of the image of the target object included in the image area. There are also features.

  Note that the imaging device 1 may not include the display unit 5 and the touch panel 6 as long as the imaging device 1 is used only as a document camera as described above. In addition, the fish-eye optical system 41, the imaging unit 4, the posture detection unit 9, and the like, and the calculation unit that performs image processing are provided as separate units, and these communicate with each other wirelessly or by wire, and are the same as the imaging device 1 described above. You may fulfill the function.

  Moreover, the above-mentioned embodiment is an example and various modifications can be considered. For example, although FIG. 3B shows an example in which one central image C2 and one peripheral image P2 are arranged as an example of the edited image I2, the present invention is not limited to this. For example, the peripheral images may be combined so that they are divided into two and displayed side by side, or may be combined so that the images are cut out for each person and displayed side by side. The same applies to the center image. The subject of the peripheral image is not limited to a person.

  In the above-described example, the example in which the optical axis of the optical system of the imaging apparatus 1 is installed vertically is shown, but the present invention is not limited to this. The imaging device 1 may be installed such that the optical axis is inclined. Also in this case, for example, various image recognition techniques are used for recognizing an image of a person in the vicinity, the vertical direction of the image is determined, the direction of the image is adjusted based on these vertical directions, An edited image can be synthesized. In addition, an object such as a photograph may be arranged at the center of the imaging region of the imaging device 1 or may be arranged at a biased position. When the object is placed at a biased position in the imaging area, an image recognition technique or the like is used to cut out the image of the object from the captured image or the image of a person around it. To do.

  Also, an edited image that includes a large amount of the person who is speaking may be created as the peripheral image. In this case, the control unit 2 refers to the sound data obtained from the microphone 12 and determines the position of the sound source. The control unit 2 cuts out an image of an area corresponding to the position of the sound source among the peripheral areas. The control unit 2 synthesizes the cut-out peripheral image into the edited image. When such an imaging device 1 is used in a conference, the edited image includes an image of an object placed in front of the imaging device 1 and an image of a person who has made a speech among conference participants around it. Will be included.

  In the above-described embodiment, the optical system included in the imaging device 1 is a fisheye optical system. However, for example, a general wide-angle lens may be used for the imaging device 1. In this case as well, the processing related to image conversion for taking image distortion is different, but the other operations are the same as those of the imaging apparatus 1 described above.

  In the above-described embodiment, the central image and the peripheral image are combined into one edited image and displayed or recorded. However, the present invention is not limited to this. The central image and the peripheral image may be displayed on different display devices as separate images or may be stored as different images.

  As described above, among the technologies described in the embodiments, the control mainly described in the flowchart can be realized using a program. This program can be stored in various recording media and recording units. There are various methods of recording on this recording medium or recording unit, which may be recorded at the time of product shipment, may be recorded using a distributed recording medium, or may be recorded using download via the Internet. May be.

  Moreover, the above-mentioned embodiment may be applied also to information portable terminals, such as a smart phone and a tablet terminal, for example. The information portable terminal includes a wearable terminal. Further, the present invention can be applied not only to a consumer camera, but also to a surveillance camera, an industrial device for inspection, and various medical observation devices. In particular, it is suitable for applications where images are taken with a wide field of view, and are displayed and recorded by adaptively determining up and down depending on the location of the screen. In particular, when the user takes a picture or when a user operates a remote device, the pose of the device may be different from the top and bottom of the image in a harsh environment. Since various objects are captured in the acquisition, it can be effectively used when it is better to display each object by determining the top and bottom. If the screen is divided in this way, it is possible to perform appropriate image processing such as enlargement, reduction, or distortion correction in addition to changing the top and bottom handling for each area, making it easier for users to view and observe There is a merit that Needless to say, these processes may be manually adjusted. Although it is a commonly used technique to correct distortion for each region, it differs here in that it is handled as a separate image for each region and further combined.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Control part, 4 ... Imaging part, 5 ... Display part, 6 ... Touch panel, 7 ... Recording part, 8 ... Operation part, 9 ... Attitude detection part, 10 ... Communication part, 11 ... Temporary storage part , 12 ... Microphone, 21 ... Image processing unit, 22 ... State determination unit, 23 ... Face detection unit, 24 ... Sound source identification unit, 25 ... Trimming unit, 26 ... Distortion correction unit, 27 ... Direction determination unit, 28 ... Image composition Part 41... Fisheye optical system 42.

Claims (6)

An imaging unit that simultaneously captures a first subject and a second subject located around the first subject to generate a captured image;
A trimming unit that cuts out a first image including the first subject and a second image including the second subject from the captured image;
A direction determining unit that determines an image direction that is a direction of the first image;
An image composition unit that prepares an edited image by adjusting the orientation of the first image and the second image based on the image direction and combining the first image and the second image; equipped with a,
The trimming unit uses a character, a figure, a symbol, or a photograph as the first subject, and a person as the second subject.
Image acquisition device. The imaging unit includes a fisheye optical system,
A distortion correction unit that corrects distortion between the first image and the second image cut out by the trimming unit;
The image composition unit synthesizes the first image and the second image in which the distortion correction unit corrects the distortion,
The image acquisition apparatus according to claim 1.   The image acquisition device according to claim 1, wherein the direction determination unit determines the image direction based on pattern matching. An imaging unit that simultaneously captures a first subject and a second subject located around the first subject to generate a captured image;
A trimming unit that cuts out a first image including the first subject and a second image including the second subject from the captured image;
A direction determining unit that determines an image direction that is a direction of the first image;
An image composition unit that prepares an edited image by adjusting the orientation of the first image and the second image based on the image direction and combining the first image and the second image; ,
A microphone to acquire sound information;
A sound source specifying unit for specifying a sound source generating a sound among the second subject based on the sound information;
Bei to give a,
The trimming unit cuts out an image corresponding to the sound source as the second image;
Image acquisition device. An imaging unit that simultaneously captures a first region where the first subject exists and a second region including a second subject different from the first subject, and generates a captured image;
A trimming unit that determines and cuts out a first image related to the first region and a second image related to the second region from the captured image;
The vertical direction of one of the first image and the second image is determined based on the posture of the imaging unit, and the vertical direction of the other of the first image and the second image is determined. A direction determining unit that determines an image direction by up / down determination, and determining based on an orientation of the first subject or the second subject in the first image or the second image;
An image composition unit that prepares an edited image by adjusting the orientation of the first image and the second image based on the image direction and combining the first image and the second image; An image acquisition apparatus comprising: Capturing simultaneously a first region where the first subject is present and a second region including a second subject different from the first subject by the imaging unit to generate a captured image;
Determining and cutting out a first image related to the first region and a second image related to the second region from the captured image;
The vertical direction of one of the first image and the second image is determined based on the posture of the imaging unit, and the vertical direction of the other of the first image and the second image is determined. Determining the image direction by up / down determination, and determining based on the orientation of the first subject or the second subject in the first image or the second image;
Adjusting the orientation of the first image and the second image based on the image direction and synthesizing the first image and the second image to create an edited image. Image acquisition method.